Message queues & event streams
Decouple producers from consumers so work survives spikes and slow downstreams.
Visual model
Put a buffer between fast and slow work
Why go asynchronous
A queue separates the part that creates work from the part that processes it. Each side can scale and fail independently.
Producers enqueue and return. Consumers process the backlog at their own pace.
The buffer absorbs spikes and keeps slow work, such as email or video encoding, outside the request path.
It also stops a downstream outage from spreading back to the API.
See it
A queue converts a spike into time
Queue vs log, and delivery guarantees
A message queue, such as SQS or RabbitMQ, removes a message after a consumer processes it. Use it to distribute tasks among competing workers.
An event log, such as Kafka or Kinesis, retains messages for a period. Independent consumer groups can read the same stream and replay its history.
- Delivery is usually at-least-once. Make consumers idempotent so processing the same message twice is safe.
- Use a dead-letter queue so a poison message doesn't block or silently drop work.
- Watch backpressure. If producers stay faster than consumers, the buffer fills. Scale consumers or shed load.
- Ordering is typically only guaranteed within a partition, not across the whole topic.
Key takeaways
- Queues decouple and buffer work. They make spiky or failure-prone tasks more resilient.
- Make consumers idempotent. At-least-once delivery means duplicates will happen.
- Use a queue for competing workers, a log (Kafka) when many consumers need the same replayable stream.