IoT data pipelines and time-series databases
In short: An IoT data pipeline ingests messages from devices, buffers them in a durable queue, processes and enriches them, stores them, and serves them to dashboards and analytics. Time-series databases are built for the timestamped readings IoT produces. Retention, downsampling and access control keep the pipeline affordable and safe.
Why a pipeline?
Ten devices can write straight to a database. Ten thousand devices sending a reading every few seconds cannot, at least not reliably. Spikes, outages and slow consumers would lose data. A data pipeline breaks the flow into stages that each do one job and can scale independently.
The stages
- Ingest: an MQTT broker, HTTPS endpoint or platform hub receives messages and authenticates devices.
- Queue or stream: a durable log (for example Apache Kafka or a cloud streaming service) holds messages so that if processing slows or fails, nothing is lost.
- Process: parse the payload, check it, convert units, add context such as the device's location, and drop duplicates.
- Store: write to a time-series database for readings and object storage for raw archives.
- Serve: dashboards, APIs, reports and machine learning read the stored data.
Time-series databases
IoT readings are almost always timestamp, source, values. A time-series database (TSDB) is designed for this shape: very fast appends, compression of similar values, and queries such as "average per hour for the last week". Popular choices include InfluxDB, TimescaleDB (an extension to PostgreSQL) and Prometheus for metrics, plus cloud equivalents.
| Concept | Meaning |
|---|---|
| Measurement / table | A type of reading, such as temperature |
| Tags / labels | Indexed metadata: device ID, site, floor |
| Fields / values | The numbers themselves |
| Retention policy | How long raw data is kept before deletion |
| Downsampling | Replacing old raw data with averages to save space |
A sensible plan keeps raw readings for weeks, hourly summaries for years, and deletes the rest automatically. Avoid tags with millions of unique values (such as a random message ID); this "high cardinality" slows many TSDBs badly.
Securing the pipeline
- Authenticate devices at ingest and reject unknown sources.
- Validate every message against a schema; discard or quarantine malformed data rather than crashing.
- Encrypt traffic between every stage, not just from the device.
- Give each service its own credentials with only the permissions it needs, for example the processor can read the queue and write to the database but not delete.
- Encrypt storage, back it up and test restores.
- Monitor pipeline health: queue lag, rejected messages and sudden volume changes can signal faults or attacks.
Tip: keep the raw, unmodified messages in cheap storage for a while. If a processing bug corrupts results, you can replay the raw data instead of losing it.
Practice
Why put a durable queue between ingestion and processing?
What is downsampling?
Name one sign in pipeline monitoring that might indicate an attack.
Key takeaways
- Pipelines split ingestion, queuing, processing, storage and serving into stages.
- Durable queues prevent data loss during spikes and outages.
- Hot paths serve alerts quickly; cold paths serve history and analytics.
- Time-series databases suit timestamped readings; use retention and downsampling.
- Validate, authenticate and encrypt at every stage.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.