Designing IoT for scale and reliability: queues, retries and offline buffering
In short: Large IoT systems must expect failures: links drop, servers restart and thousands of devices reconnect at once. Queues absorb spikes, retries with exponential backoff and jitter avoid overload, idempotent processing makes duplicates harmless, and local buffering keeps data through outages. Redundancy removes single points of failure.
Assume things will fail
At small scale, failures are rare and can be fixed by hand. At large scale, something is always failing: a mobile cell is congested, a gateway reboots, a cloud region has an incident. Reliable IoT design starts from the assumption that every link and server will sometimes be unavailable, and makes sure data is not lost and the system recovers on its own.
Buffer, then forward
Store and forward means writing each message to local storage first and deleting it only after the receiver confirms it. MQTT supports this with QoS 1 (at least once), where the broker acknowledges each message. Size the buffer for the longest outage you expect, and decide what happens when it fills: drop the oldest readings, or keep a summary.
Retries, backoff and jitter
When a send fails, the device should retry, but not immediately and not forever. Exponential backoff waits longer after each failure, for example 1, 2, 4, 8 seconds up to a cap. Jitter adds a random amount to each wait. Without jitter, when a server comes back after an outage, every device retries at the same moment and knocks it over again, a problem called the thundering herd.
delay = min(cap, base * 2 ** attempt)
sleep(random.uniform(0, delay)) # "full jitter"Queues and idempotency
On the server side, a durable message queue or stream sits between ingestion and processing, so a burst of messages waits in line rather than overwhelming the database. Consumers can be scaled out to drain the queue faster.
Retries mean duplicates are inevitable. Design processing to be idempotent: handling the same message twice has the same effect as handling it once. Give each message a unique ID made from device ID plus sequence number, and ignore IDs already seen. For commands, idempotency is critical: "set valve to 40 percent" is safe to repeat; "open valve 10 percent more" is not.
| Problem | Pattern |
|---|---|
| Network drops for an hour | Local buffering, store and forward |
| Server overloaded by burst | Queue, autoscaled consumers, rate limits |
| All devices reconnect at once | Exponential backoff with jitter |
| Duplicate messages | Idempotent processing, message IDs |
| Gateway hardware fails | Redundant gateways, health checks |
Reliability and security together
Reliability features can help or hurt security. Rate limits and quotas per device stop one faulty or compromised device from flooding the platform. Buffers on devices should be encrypted if they hold sensitive data. Monitoring for retry storms and queue growth helps spot both outages and denial-of-service attempts. And availability is part of the CIA triad: a system that loses data during every outage is not secure.
Tip: test failure on purpose. Unplug the gateway's uplink for an hour in a lab and check that every reading arrives afterwards, once, in the right order.
Practice
What is the thundering herd problem?
Why must message processing be idempotent in IoT systems?
Which is safer to retry: 'set temperature to 21' or 'raise temperature by 1'? Why?
Key takeaways
- Design IoT systems assuming links and servers will fail.
- Store and forward keeps data until the receiver acknowledges it.
- Retry with exponential backoff and jitter to avoid thundering herds.
- Make processing and commands idempotent so duplicates are harmless.
- Queues, rate limits and redundancy protect both reliability and security.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.