Message queues and event streaming
In short: Message brokers let services communicate asynchronously, so a producer can carry on even if a consumer is slow or down. Queues deliver each job to one worker; event streams keep an ordered log many consumers can replay. Most systems deliver at least once, so consumers must be idempotent.
Why use messaging
When service A calls service B directly, both must be up at the same moment, and a slow B slows A. A message broker sits between them. A producer publishes a message and moves on; a consumer processes it when ready. This asynchronous style brings:
- Decoupling: services do not need to know each other's address or be online together.
- Load levelling: bursts are absorbed in the queue instead of overloading workers.
- Scaling: add more consumers to drain a backlog faster.
- Resilience: if a consumer crashes, messages wait safely.
Queues and streams
A message queue (point-to-point) hands each message to one consumer, which acknowledges it, and the broker removes it. This suits background jobs such as sending emails or resizing images. RabbitMQ and Amazon SQS are common examples. Publish/subscribe extends this so one message is copied to several subscribers.
An event stream is an append-only log split into partitions. Events stay for a configured retention period, and each consumer group tracks its own offset, so many teams can read the same events independently and replay history. Apache Kafka and Amazon Kinesis are well-known examples. Streams suit event sourcing, analytics, audit and feeding many downstream systems.
Delivery guarantees
| Guarantee | Meaning | Risk |
|---|---|---|
| At most once | Sent once, never retried | Messages can be lost |
| At least once | Retried until acknowledged | Duplicates are possible |
| Exactly once | Effect happens once | Only within limits, such as a single platform's transactions |
At least once is the usual default. A consumer may process a message, crash before acknowledging, and receive it again. So consumers must be idempotent: processing the same message twice has the same effect as once. A simple way is to store each message ID once it has been handled and skip any repeats.
Ordering is usually guaranteed only within a partition or queue. Use a key, such as order ID, so all events for one entity land in the same partition.
Operating it well
- Send messages that fail repeatedly to a dead-letter queue (DLQ) and alert on it, so one bad message does not block others.
- Monitor consumer lag or queue depth; a growing backlog means consumers cannot keep up.
- Version message schemas so producers and consumers can change independently.
- Secure the broker with TLS, authentication and per-topic access control.
- Use the outbox pattern to write a database change and its event in one local transaction, then publish from the outbox.
Tip: brokers add latency and moving parts. Use them where decoupling or buffering matters, not for every simple request that needs an immediate answer.
Practice
Why must consumers be idempotent with at-least-once delivery?
Three teams need to read the same order events independently. Queue or stream?
What is a dead-letter queue for?
Key takeaways
- Brokers decouple producers and consumers in time and space.
- Queues give each message to one worker; streams keep a replayable log.
- At least once is the common guarantee, so consumers must be idempotent.
- Ordering usually holds only within a partition or key.
- Watch consumer lag and dead-letter queues.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.