IoT Data Pipelines - Architecture Insights

The IoT Data Challenge

Connected devices produce data continuously, and that data arrives with characteristics that make standard application architecture patterns inadequate. A fleet of 10,000 devices sending telemetry every five seconds generates roughly 172 million messages per day. Scale that to a million devices and you are looking at over 17 billion messages daily, before accounting for bursts, reconnections, or firmware events. The volume is not just large; it is relentless and structurally irregular in ways that matter for every design decision downstream.

Three properties define why IoT data is genuinely different from transactional application data. First, it is time-series by nature: every reading has a timestamp, and the temporal relationship between readings is often as meaningful as the readings themselves. A temperature sensor at 72 degrees means nothing without knowing whether it was 68 degrees ten seconds ago or 95 degrees. Second, reliability varies dramatically at the edge. Devices go offline, reconnect after hours, and send batched backlogs that arrive out of order. A pipeline designed only for orderly real-time delivery will silently lose data when devices behave normally. Third, schema heterogeneity is the rule rather than the exception. A facility management system might have temperature sensors, occupancy sensors, HVAC controllers, and access card readers all feeding the same pipeline with different payload structures and different update frequencies.

These three properties drive the architectural patterns covered in this guide. A well-designed IoT data pipeline does not treat data as a monolith to store and query later; it routes data through different processing paths based on how quickly that data needs to influence decisions, and it builds in tolerance for the messiness that edge devices produce.

Hot, Warm, and Cold Paths

Most IoT systems need to do three distinct things with the same stream of device data: react immediately to critical conditions, answer operational questions about recent trends, and support deep historical analysis over months or years. No single storage or processing technology handles all three well, so production IoT architectures split incoming data across multiple paths, each optimized for its own latency and retention requirements.

The Hot Path

The hot path processes events within milliseconds to seconds of ingestion. Its purpose is immediate reaction: triggering an alert when a machine temperature exceeds a threshold, detecting an anomaly that suggests equipment failure, or updating a live dashboard that an operator watches in real time. The hot path never waits for data to accumulate; it evaluates each event or a very small window of events as they arrive.

Technologies like Azure Stream Analytics, Azure Functions with event triggers, and Apache Flink are designed for this path. They consume from event brokers like Azure IoT Hub or Apache Kafka and produce outputs in sub-second time. The hot path is stateful in a limited sense: it might maintain a five-second rolling average to smooth noise before comparing against a threshold, but it does not join against months of history. Latency is the primary constraint, so hot path storage outputs tend to be lightweight: a notification to an alerting system, a write to a Redis cache for a live dashboard, or a trigger to an automation workflow.

The Warm Path

The warm path handles queries about recent data, typically covering hours to a few days. Operational dashboards showing shift performance, queries asking whether a device has been behaving unusually in the last four hours, or trend analysis comparing today to yesterday all belong here. The warm path can afford slightly higher latency than the hot path, usually in the range of seconds to minutes, but it must still respond quickly enough to support interactive exploration.

Technologies like Azure Data Explorer (ADX) and Apache Druid are optimized for this workload. They ingest streaming data continuously and make it queryable within seconds to minutes of arrival, while supporting fast aggregation queries over time windows. The warm path stores more data than the hot path holds in memory, but less than the cold path retains on disk. Retention windows of 7 to 30 days are common, with data rolling off to cold storage as it ages.

The Cold Path

The cold path handles historical data at scale: months or years of telemetry used for machine learning model training, compliance audits, root cause analysis of past incidents, and long-term trend analysis. Query latency is measured in seconds to minutes rather than milliseconds, because the cold path typically runs batch queries against very large datasets rather than interactive queries against a small window.

Storage technologies like Azure Data Lake Storage, raw Azure Blob Storage, and processing frameworks like Apache Spark dominate the cold path. Raw events are typically stored in Parquet or Avro format, partitioned by time and device, to support efficient batch scanning. The cold path prioritizes storage cost and query completeness over response time.

Comparing the Three Paths

Most IoT systems need all three paths operating simultaneously. A single device event might be evaluated by the hot path for anomaly detection, written to a warm store for the operations dashboard, and archived to cold storage for compliance, all as part of the same ingestion flow.

Dimension	Hot Path	Warm Path	Cold Path
Latency target	Milliseconds to seconds	Seconds to minutes	Minutes to hours
Retention window	Seconds to minutes (in memory)	Days to weeks	Months to years
Primary purpose	Alerting, live dashboards, automation	Operational queries, recent trend analysis	ML training, compliance, historical analytics
Query pattern	Event evaluation, small windows	Time-range aggregations, interactive exploration	Full scans, batch jobs
Storage technology	Redis, in-memory state, alert queues	Azure Data Explorer, Apache Druid, TimescaleDB	Azure Data Lake, Blob Storage, S3
Processing technology	Azure Stream Analytics, Azure Functions, Apache Flink	ADX continuous ingestion, Druid streaming	Apache Spark, Azure Data Factory, Databricks
Failure tolerance	Low: must process immediately	Medium: can lag by minutes	High: batch retries are routine

Event-Driven Architecture for IoT

The natural model for IoT data is events rather than state. A device does not send “my current temperature.” It sends “my temperature at 14:32:07 was 74.3 degrees.” That event happened, it cannot un-happen, and the pipeline’s job is to route, process, and store a record of what happened. This distinction shapes the entire architectural approach.

Events as the Fundamental Unit

An IoT event is an immutable record of something that occurred at a specific device at a specific time. Well-structured events include a device identifier, a precise timestamp, a payload containing the measurement or state change, and often metadata like firmware version or signal strength. Treating events as immutable records rather than mutable state snapshots makes the system easier to reason about: if something went wrong, the event log tells you exactly what happened in what order.

The immutability of events also enables replay. If a processing bug corrupts aggregated results, you can go back to the raw event log and reprocess from scratch. This capability is central to both Lambda and Kappa architectures discussed later.

Event Ingestion: Brokers and Hubs

High-throughput event brokers sit between devices and the processing infrastructure. They absorb incoming events at the rate devices produce them, buffer them durably, and allow multiple downstream consumers to read at their own pace.

Azure IoT Hub provides device-specific capabilities beyond raw event ingestion: per-device authentication, device twins for managing configuration and reported state, direct methods for sending commands back to devices, and built-in routing rules. It surfaces as an Event Hub-compatible endpoint, so stream processors can consume from it using standard Event Hubs SDKs.

Azure Event Hubs is a pure event broker without device management features, suited for scenarios where devices authenticate through an application gateway or where the source is not literally a device (such as a mobile app or a third-party system feeding IoT-style telemetry).

Apache Kafka is the most widely deployed open-source event broker and provides the richest ecosystem for stream processing integration. Kafka’s consumer group model allows multiple independent consumers to read the same stream simultaneously at different offsets, which enables fan-out patterns discussed below.

Event Routing

Not every event needs the same treatment. A critical alert from a pressure sensor demands immediate hot-path processing. A routine temperature reading from a non-critical sensor might go directly to warm storage. An administrative event reporting a firmware update might route only to a device management system.

Event routing applies rules at the broker or immediately after ingestion to direct events to different downstream processors based on content, source device, message type, or any combination of properties. Azure IoT Hub has built-in message routing that can filter on message properties and send matching events to different endpoints. Kafka achieves similar results through topic partitioning and consumer group configuration, or through stream processing frameworks that read from one topic and write to multiple output topics based on filter conditions.

Routing decisions at the ingestion layer reduce unnecessary processing downstream. A pipeline that sends every event to every processor and then filters later wastes resources and increases complexity.

Event Sourcing

Event sourcing treats the event log as the authoritative source of truth for the system’s state rather than a derived record. Instead of storing the current state of each device and updating it on each reading, the system stores every event in an append-only log and derives current state by replaying the log.

For IoT, event sourcing is a natural fit. The full telemetry history is the source of truth; any aggregated view, such as “average temperature per device per hour last week,” is a projection derived from that log. If the projection logic changes or a bug corrupts a projection, the raw event log remains intact and can be used to rebuild accurate projections.

The practical implication is that the event broker’s retention period matters architecturally, not just operationally. Azure IoT Hub and Event Hubs support configurable retention up to 90 days for standard tiers. For longer-term event sourcing, raw events are typically archived to blob storage or a data lake immediately on ingestion, preserving the complete log indefinitely at low cost.

Fan-Out Patterns

A single incoming event frequently needs to trigger multiple independent downstream processes: alert evaluation, warm store ingestion, cold archive write, and a device-specific state update might all need to happen for the same event. This is the fan-out pattern.

Fan-out is achieved at the broker level by having multiple consumer groups or subscriptions reading the same event stream. Each consumer processes the stream independently at its own pace without coordination. If the alerting consumer falls behind, it does not block the archive consumer. If the warm store consumer needs to be restarted for maintenance, the other consumers continue unaffected. The decoupling is one of the primary reasons event brokers sit at the center of IoT architectures rather than direct point-to-point messaging.

Stream Processing

Stream processing is the computational layer that transforms, aggregates, and evaluates event streams as they flow through the system. It occupies the space between raw ingestion and storage, where business logic gets applied to events before they reach a destination.

Windowing

Time-series data from IoT devices rarely needs to be evaluated one event at a time; most useful computations aggregate over a time window. Windowing defines how events are grouped into sets for processing.

A tumbling window divides time into fixed, non-overlapping intervals. A five-minute tumbling window groups all events from 14:00 to 14:05, then all events from 14:05 to 14:10, and so on. Each event belongs to exactly one window. Tumbling windows work well for periodic reporting: compute average temperature per device for each five-minute block, then store the result.

A sliding window also has a fixed size, but it advances continuously rather than in discrete steps. A five-minute sliding window evaluated every thirty seconds includes all events from the last five minutes at each evaluation point. This creates overlapping windows where a single event can appear in multiple windows. Sliding windows are better suited for anomaly detection where you want continuous evaluation rather than waiting for a tumbling window to close.

A session window groups events by activity, closing the window when a device goes quiet for longer than a defined gap. If a device sends readings every second during active operation and nothing for hours during downtime, session windows naturally group the active periods. Session windows are variable in length and start and end at device-driven boundaries rather than clock-driven ones.

Stateful Processing

Some computations require remembering information across events rather than evaluating each event independently. A running average, a count of errors in the last hour, or a detection of whether a value has been increasing monotonically for ten minutes all require the processor to maintain state across the stream.

Stream processing frameworks like Apache Flink, Azure Stream Analytics, and Apache Spark Structured Streaming manage this state internally, persisting it durably so that a restart does not lose accumulated computations. For IoT workloads with many devices, state is typically keyed by device identifier, so each device maintains its own independent state rather than sharing a global aggregate.

Stateful processing introduces a tradeoff between freshness and cost. Maintaining fine-grained state for millions of devices in memory is expensive. Strategies like pre-aggregating on the device before sending, reducing the frequency of state updates, or offloading state to an external store like Redis can reduce the memory footprint while preserving the ability to answer stateful queries.

Late-Arriving Data

IoT devices go offline. When a device reconnects after hours of disconnection, it may send a batch of events with timestamps from the past. The stream processor’s window for those timestamps has already closed and its results may have already been written downstream. Late-arriving data forces a choice: ignore late events (simple but lossy), reopen and update closed windows (accurate but complex), or accept late events up to a configurable deadline and then drop anything that arrives after.

Watermarks are the standard mechanism for handling this tradeoff. A watermark is the processor’s estimate of how far behind real time the stream has fallen, expressed as a maximum expected latency. If the watermark is set to ten minutes, the processor will wait ten minutes past a window’s close time before finalizing results, accepting any events that arrive within that window. Events arriving after the watermark threshold is exceeded are treated as late and handled according to a configured policy: drop, route to a side output for separate handling, or update the already-emitted result.

Choosing a watermark threshold requires understanding the device population’s typical connectivity patterns. A fleet of industrial sensors on a reliable wired network might need a watermark of only a few seconds. A fleet of battery-powered field sensors that upload over intermittent cellular connections might need watermarks measured in hours. Getting this wrong leads to either silently dropped data or perpetually open windows that consume memory indefinitely.

Processing Semantics

Stream processing systems make guarantees about how many times each event is processed, and these guarantees have direct consequences for the correctness of aggregated results in IoT systems.

At-most-once processing delivers each event zero or one times. Events can be lost if a processor crashes before acknowledging them. For non-critical telemetry where occasional data loss is acceptable, this is the simplest and lowest-overhead option.

At-least-once processing guarantees every event is processed, but allows duplicates. If a processor crashes after processing an event but before acknowledging it to the broker, the broker will redeliver the event and the processor will see it twice. For IoT scenarios where duplicates are harmless (such as archiving raw events to a data lake where duplicates can be deduplicated later), at-least-once is a practical choice.

Exactly-once processing guarantees each event is processed exactly one time even in the presence of failures. It requires coordination between the processor and its output destinations to ensure idempotent writes or transactional commits. Exactly-once is the most expensive guarantee to maintain and is most important when the output is a running count or sum where duplicates would corrupt the result. In practice, many IoT systems use at-least-once semantics with idempotent writes to achieve the effect of exactly-once without the full coordination overhead.

Lambda and Kappa Architecture

Two architectural patterns have emerged as organizing principles for IoT data systems that need to balance real-time responsiveness against historical completeness. Both address the same problem: how do you maintain accurate aggregated results when the stream of events is continuous and potentially infinite?

Lambda Architecture

Lambda architecture separates processing into two independent layers: a batch layer that processes the complete historical dataset periodically and a speed layer that handles real-time data since the last batch run. Query results are served by merging outputs from both layers.

The batch layer runs on a schedule, such as every hour or every day, reprocessing all historical events with correct, complete logic. It produces accurate results for any time window in the past. The speed layer processes events as they arrive, producing approximate or preliminary results for the recent period not yet covered by the latest batch run. When a user queries for results, the system combines the batch layer’s historical output with the speed layer’s recent output to produce a complete answer.

Lambda architecture handles reprocessing well. If a bug is discovered in the processing logic, the batch layer can be corrected and re-run over the full historical dataset, producing accurate results that supersede the original outputs. The speed layer’s results for the recent period will be replaced by the next batch run.

The cost of Lambda is complexity. Two separate codebases implement the same business logic: one for the batch layer using frameworks like Apache Spark, and one for the speed layer using frameworks like Apache Flink or Kafka Streams. Keeping these two implementations synchronized as business requirements evolve is a persistent operational burden. When the batch and speed layers produce different results for the same time period (which happens due to subtle differences in implementation), reconciling those differences becomes a debugging challenge.

Kappa Architecture

Kappa architecture simplifies Lambda by eliminating the batch layer entirely. All processing, both real-time and historical, runs through the same stream processing infrastructure. Reprocessing is achieved not by a separate batch layer but by replaying events from the beginning of the event log through the same stream processors.

This approach requires a durable, replayable event log as the foundation. If the raw event log is stored in an append-only store like Kafka (with long retention) or a data lake, reprocessing means reading from the beginning of the log and running it through the current version of the processing logic. A second consumer group can run the reprocessing in parallel with the live stream consumer, writing results to a new output while the live pipeline continues operating. Once the reprocessed results are validated, traffic is cut over to the new output.

Kappa is simpler operationally than Lambda because there is only one codebase to maintain. For IoT, it is particularly attractive when the event schema is stable enough to support reliable replay and when the event broker has sufficient retention capacity to store the full replayable history.

The limitation of Kappa is that reprocessing at very large scale through a stream framework can be slower and more expensive than the same job run as an optimized batch query in Spark. For some IoT systems with years of history and terabytes of daily data, a pure stream reprocessing approach may be impractical for full historical reprocessing runs. In practice, many IoT platforms adopt a pragmatic hybrid: Kappa-style stream-first processing for most needs, with batch processing reserved for large-scale historical backfills.

For most IoT use cases, Kappa is the better starting point. The operational simplicity of a single processing codebase outweighs the reprocessing speed limitation unless the system has already demonstrated a need for frequent large-scale historical reprocessing.

Data Quality and Schema Management

Raw device telemetry is rarely clean. Sensors drift over time, produce occasional outlier spikes from interference, drop readings entirely when batteries fail, and send payloads with missing or null fields when firmware bugs surface. A pipeline that assumes clean input will silently propagate bad data into aggregations, dashboards, and ML training sets.

Validation at Ingestion

The best place to catch bad data is at the ingestion boundary, before it enters the processing pipeline. Validation at ingestion applies a set of rules to each incoming event and decides whether to accept it, reject it, or quarantine it for review.

Common validation checks include confirming that required fields are present and non-null, verifying that numeric values fall within plausible physical ranges (a temperature sensor reading of 5,000 degrees Celsius should be rejected, not stored), checking that timestamps are within an acceptable window of the current time, and confirming that the device identifier maps to a known registered device.

Events that fail validation should not be silently dropped. A dead-letter queue or quarantine store preserves rejected events with metadata indicating why they were rejected. This creates a feedback loop for identifying misbehaving firmware, degraded sensors, or routing errors.

Schema Registry

When a device fleet includes many device types with different payload structures, or when device firmware evolves over time and changes the structure of its messages, schema management becomes a significant operational concern. A schema registry provides a centralized store for the expected structure of each message type, enabling producers and consumers to agree on format without hard-coding that agreement in application code.

Confluent Schema Registry is the most widely adopted implementation for Kafka-based pipelines. It stores Avro, JSON Schema, or Protobuf schemas by subject (typically topic name) and version, and enforces compatibility rules when new schema versions are registered. A backward-compatible schema change allows existing consumers to read messages produced by the new schema without modification. A breaking change requires coordinated producer and consumer updates.

For IoT, schema evolution is a real operational concern because firmware updates to millions of devices happen gradually. At any given moment, devices on three different firmware versions might be sending events with three different payload structures. The pipeline must handle all three correctly without manual per-version branches in the code.

Data Enrichment

Raw device telemetry often lacks the context that makes it useful for analysis. A temperature reading from device ID A7F3B2 is hard to interpret without knowing that device A7F3B2 is a sensor in the HVAC system on the third floor of Building 4, owned by tenant X, running firmware version 2.3.1, and installed six months ago.

Data enrichment joins raw telemetry events with reference data about devices to add this context before storage or downstream processing. Enrichment typically happens in the hot or warm path using a lookup against a device metadata store. The enriched event carries fields like device location, owning tenant, physical asset it is attached to, and configuration state.

The device metadata store needs to be fast enough to support high-throughput lookups without becoming a bottleneck. A distributed cache like Redis holding device metadata with infrequent invalidation when metadata changes is a common pattern. The raw event archive in cold storage typically stores unenriched events so that enrichment logic can be updated and reapplied if the reference data changes.

Storage Strategies

IoT data needs multiple storage technologies because different query patterns have very different performance characteristics. A technology optimized for high-throughput append writes and time-range scans will perform poorly for point lookups by device identifier, and vice versa.

Time-Series Databases

Time-series databases are purpose-built for storing and querying sequences of measurements indexed by time. They are optimized for the patterns most common in IoT analytics: retrieve all readings for a specific device over a time range, compute aggregations over time windows, and downsample high-frequency data to lower resolution for long-term storage.

Azure Data Explorer is Microsoft’s hosted time-series analytics engine, capable of ingesting millions of events per second and querying over terabytes of data with low latency. It uses the Kusto Query Language (KQL) and is particularly strong for the warm path use case. TimescaleDB is an extension of PostgreSQL that adds time-series optimizations including automatic time-based partitioning and continuous aggregations, suitable for teams who want SQL familiarity. InfluxDB is a purpose-built time-series database with its own query language and strong support for high write throughput and time-based retention policies.

Common time-series optimizations include columnar storage (which enables efficient aggregation of single fields across many rows), automatic data compression (sequential numeric values compress extremely well), and downsampling policies that automatically replace high-resolution old data with lower-resolution summaries to manage storage cost over time.

Document Databases for Device Metadata

Sensor readings are time-series data, but device metadata is not. A device’s registered location, owner, model, firmware version, and configuration state are properties of an entity that change infrequently and need efficient point lookups by device identifier. Document databases like Azure Cosmos DB or MongoDB handle this pattern well, providing flexible schemas for varied device types and fast single-document reads.

The device registry often also stores the device twin concept: a cloud-side representation of the device’s desired configuration and last reported state. This enables the platform to know what configuration a device should have and compare it against what the device last reported, even when the device is offline.

Blob Storage for Raw Archives

Raw event archives in blob storage serve as the foundation of the cold path and the enabler of event sourcing. Every incoming event, in its original form before enrichment or processing, gets written to blob storage in an append pattern partitioned by time and device. This archive is the raw record of what happened.

Blob storage is priced for long-term retention at low cost. Immutability policies and lifecycle management can automatically tier data from hot to cool to archive access tiers as it ages. Raw archives in formats like Parquet or Avro enable efficient batch scanning with frameworks like Apache Spark and integrate directly with data lake query engines like Azure Synapse Analytics.

Partitioning Strategies

Partitioning determines how data is physically organized within storage and has a large impact on query performance. The optimal partitioning strategy depends on which queries run most frequently.

Time-based partitioning organizes data by date or hour, such as storing all events from a given hour in a single partition. This is optimal for time-range queries spanning many devices (retrieve all events between 14:00 and 16:00 yesterday) but performs poorly for device-specific queries over long periods (retrieve all events for device A7F3B2 over the last three months).

Device-based partitioning organizes data by device identifier. This is optimal for device-specific queries but makes cross-device time-range queries expensive because they must scan many partitions.

Composite partitioning uses both dimensions: partition first by time period (day or week), then sub-partition by device or device group within each time partition. This balances the two query patterns without perfectly optimizing for either.

Many IoT platforms partition differently across storage tiers. The warm store, which handles both time-range and device-specific queries interactively, uses composite partitioning or a purpose-built time-series database that handles both patterns natively. The cold store, which primarily runs batch jobs over time ranges, uses time-based partitioning to minimize scan costs.

Visualization and Alerting

Data that reaches storage but never drives a decision or action delivers no value. The final layer of an IoT pipeline connects processed data to the humans and systems that need to act on it.

Real-Time Operational Dashboards

Operational dashboards display the current state of the device fleet for the humans responsible for managing it. A facilities manager watching energy consumption across a building portfolio, a plant engineer monitoring machine health on a factory floor, and a fleet manager tracking vehicle locations all need the same fundamental capability: a live view of current conditions with enough context to recognize when something needs attention.

Dashboard design for IoT differs from business intelligence dashboards because the data changes continuously and the volume of devices often exceeds what can be displayed individually. Effective IoT dashboards aggregate to a useful level (plant, floor, zone, or device group) with the ability to drill down to individual devices when an anomaly draws attention. Grafana is widely used for real-time IoT dashboards because its plugin ecosystem includes native connectors for InfluxDB, TimescaleDB, and Azure Data Explorer, and its alerting system can be driven by the same queries used to power visualizations.

Alerting Strategies

Threshold-based alerting triggers when a measured value crosses a fixed limit: temperature above 80 degrees, battery below 10 percent, pressure outside a safe operating range. This is the simplest form of alerting and is appropriate for conditions with well-known danger thresholds. The risk with pure threshold alerting is alert fatigue: noisy sensors produce spurious threshold crossings that generate alerts no one believes or acts on, so over time operators start ignoring the alerts.

Anomaly-based alerting triggers when a device’s behavior deviates significantly from its own historical baseline or from the baseline of similar devices in the fleet. A motor that normally runs at 1,450 RPM and suddenly runs at 1,380 RPM might not cross a static threshold but represents a meaningful change. Anomaly detection requires historical data and more sophisticated logic than threshold comparison, but produces alerts that are more likely to represent genuine conditions worth investigating.

Pattern-based alerting triggers on sequences of events rather than single events. A device that has crossed a minor threshold three times in thirty minutes might warrant attention even if no single crossing was severe. An anomalous reading immediately following a device reconnection after a long offline period might indicate a sensor calibration issue rather than a real physical change. Pattern recognition connects events over time to identify situations that threshold-based or anomaly-based rules would miss.

In practice, production IoT platforms layer all three approaches. Static thresholds catch obvious, critical conditions immediately. Anomaly detection catches gradual degradation. Pattern rules catch operational edge cases specific to the device types and use cases in the system.

Integration Patterns

Alerts and visualizations integrate with the broader operational toolset through standard integration patterns rather than bespoke connections. Power BI connects to Azure Data Explorer and Synapse Analytics for management reporting and compliance dashboards where the audience is business stakeholders rather than technical operators. Custom dashboards built with frameworks like Blazor or React can connect to streaming APIs to display device state with low latency.

Alert notifications route through messaging infrastructure rather than direct connections from the IoT platform to notification channels. An alert condition triggers an event to a message queue, and a notification service reads from that queue and delivers the alert via email, SMS, Teams message, or PagerDuty, depending on severity and routing rules configured for the alert type. This decoupling allows notification channels to change without modifying the alert detection logic and ensures that a notification service outage does not block alert evaluation.

Putting It Together: Pipeline Design Decisions

Designing an IoT data pipeline involves a sequence of decisions that build on each other. Each decision creates constraints and opportunities for the decisions that follow.

The first design decision is retention: how long must raw events be preserved, and at what granularity? Compliance requirements often dictate minimum retention periods. ML use cases often have minimum data volume requirements. Long retention with full granularity at cold storage tiers is cheap enough that erring toward longer retention is usually the right default.

The second decision is latency: which conditions require immediate response, and how immediate is immediate? A gas leak detection system might need sub-second alerting. An energy consumption optimization system might tolerate hour-long warm-path latency. Latency requirements determine which parts of the pipeline need the hot path and how much investment in low-latency infrastructure is justified.

The third decision is device heterogeneity: how varied are the device types and their data formats? A homogeneous fleet of identical sensors with stable firmware is a simpler schema management problem than a diverse ecosystem of third-party devices from multiple manufacturers. High heterogeneity pushes toward a schema registry, flexible storage formats, and enrichment pipelines that normalize across device types.

The fourth decision is processing architecture, specifically whether to use Lambda or Kappa and which stream processing framework to build on. For most greenfield IoT projects starting today, Kappa with a durable event log is the cleaner choice. Lambda becomes worth its complexity when the organization already has mature batch processing infrastructure or when the scale of historical data makes stream-based reprocessing impractical.

The fifth decision is the query access pattern: who queries the data and how? Operations teams using dashboards have different needs than data scientists running ML training jobs, which differ from compliance auditors running period-end reports. Each audience drives requirements for the warm and cold path storage technologies and the interface layers built on top of them.

These decisions are not made once. IoT systems grow, device fleets expand, and use cases that were not anticipated at design time emerge. An architecture that is designed with explicit trade-offs documented for each decision is much easier to evolve than one where the decisions are implicit in the code.