Offline Capabilities

Store and Forward

Edge Hub stores outgoing messages in a local SQLite database when the cloud connection is unavailable. Once connectivity is restored, it forwards buffered messages to IoT Hub in the order they were received. The buffer is durable across module restarts and device reboots, so messages are not lost if the device loses power while offline.

The timeToLiveSecs property in the Edge Hub desired properties controls how long messages are retained before being discarded. Setting this to 7200 means messages are held for up to two hours. Choosing the right TTL is a balance between storage capacity on the edge device and acceptable data loss. For high-frequency telemetry where old readings have little value, a short TTL saves disk space. For critical alarm events where every message matters, a longer TTL ensures nothing is dropped.

Priority-Based Sending

When connectivity returns, Edge Hub can send messages with higher priority first. Routes can be assigned a priority value so that critical alerts reach the cloud before bulk telemetry, even if the telemetry was buffered earlier. This is useful when the reconnection window is short and the full buffer cannot be flushed before the link drops again.

Priority is configured in the route definition within the Edge Hub desired properties. A route carrying alarm messages might have priority 5 while the bulk telemetry route has priority 1, ensuring alarms are transmitted first regardless of buffer order.

Local Device-to-Device Communication

Modules communicate through Edge Hub using MQTT or AMQP regardless of cloud connectivity. A downstream leaf device, such as a sensor that does not itself run IoT Edge, can connect to Edge Hub using MQTT and send messages that are routed to local modules for processing. This means local analytics and control loops continue running even when the cloud is completely unreachable.

Edge Hub also supports local module-to-device messaging, where a module can send messages back to a connected leaf device through a reverse route. This enables closed-loop control scenarios where the edge device reads sensor data, processes it locally, and sends actuation commands back to downstream hardware without requiring a cloud round-trip.

The Gateway pattern, described in the Common Edge Patterns section, formalizes this topology.

Estimating Storage Requirements

Planning the store-and-forward buffer requires estimating how much data accumulates during the longest expected outage. Consider a device receiving telemetry from ten sensors at one-second intervals, with each JSON message averaging 200 bytes. That produces about 2 KB per second, or roughly 120 MB per hour. A two-hour TTL with that message rate requires about 240 MB of available disk space for the Edge Hub database, plus headroom for the operating system and module images.

Devices with very high message rates and long TTL requirements need SSDs rather than SD cards both for capacity and for the write endurance that continuous database writes demand.

Common Edge Patterns

Data Filtering and Aggregation

The simplest and most common use of IoT Edge is reducing the volume of data sent to the cloud. A raw sensor array might emit temperature, vibration, and pressure readings ten times per second. Sending all of this to IoT Hub is expensive and usually unnecessary. A filter module discards readings that fall within normal ranges and forwards only anomalies, while an aggregation module computes rolling averages and sends one summary message per minute instead of six hundred individual readings.

This pattern significantly reduces IoT Hub messaging costs and downstream storage costs in time-series databases. IoT Hub pricing is based on message count and message size, so a fleet of 500 devices each sending raw readings at 10 Hz generates 5,000 messages per second. Aggregating to one message per device per minute reduces that to about 8 messages per second, a reduction of more than 99% in message volume with no loss of the trends and anomalies that operations teams actually act on.

Separating the filter module from the aggregation module also makes it easier to tune them independently. Thresholds for filtering anomalies can be updated through module twin desired properties without touching the aggregation logic, and the aggregation window can be changed without redeploying the filter.

Protocol Translation

Many industrial devices speak protocols like Modbus, OPC-UA, or BLE rather than MQTT or AMQP. A protocol translation module acts as a bridge, reading from the device using its native protocol and publishing messages to Edge Hub using MQTT. The rest of the module pipeline and the cloud never need to know what protocol the underlying device uses.

Azure IoT Edge OPC Publisher is a Microsoft-provided module that handles OPC-UA translation, turning OPC-UA servers into MQTT publishers with minimal custom code.

Local Analytics with Azure Stream Analytics

Azure Stream Analytics can be packaged as an IoT Edge module. You write a Stream Analytics query in the Azure portal, associate it with an IoT Edge job, and the runtime compiles it into a container that runs on the device. This lets you run time-windowed aggregations, anomaly detection with the built-in ML functions, and joins between streams locally without sending data to the cloud.

ML Inference at the Edge

Deploying trained machine learning models as IoT Edge modules enables real-time inference without cloud round-trips. A model trained in Azure Machine Learning can be exported in ONNX format and wrapped in a container that exposes an HTTP or MQTT interface. Other modules send sensor data to the inference module and receive predictions in milliseconds rather than the hundreds of milliseconds required for a cloud round-trip.

This pattern is common for visual inspection on manufacturing lines, predictive maintenance on rotating equipment, and voice command processing on constrained devices.

The Gateway Pattern

IoT Edge supports three gateway configurations for connecting devices that cannot connect directly to IoT Hub.

In the transparent gateway pattern, downstream leaf devices authenticate directly with IoT Hub through the edge device, which forwards their messages. The leaf devices have their own device identities in IoT Hub and the gateway is invisible to them. This pattern requires no changes to leaf device firmware beyond pointing at the gateway’s address instead of IoT Hub.

In the protocol translation gateway pattern, the edge device translates between a device-native protocol and the IoT Hub protocol. The gateway has a single device identity in IoT Hub and speaks on behalf of the leaf devices, which have no IoT Hub identity. This is appropriate for legacy devices using Modbus or BACnet.

In the identity translation gateway pattern, the edge device creates virtual device identities for leaf devices that lack IoT Hub support. The gateway maps messages from leaf devices to their corresponding virtual identities and reports them as if they were independent devices in IoT Hub.

Deployment and Device Management

Automatic Deployments

Automatic deployments target devices based on device twin tags. When a new device with the matching tag connects to IoT Hub, the deployment is applied automatically without any manual intervention. This is how large fleets are managed without per-device configuration.

A device twin tag might look like {"location": "plant-a", "type": "assembly-line"}. A deployment targeting tags.type = 'assembly-line' applies to every device with that tag, regardless of when it was provisioned.

Updating Modules

Updating a module means publishing a new container image and updating the deployment manifest to reference the new image tag. The Edge Agent on each targeted device detects the manifest change, pulls the new image, stops the old container, and starts the new one. Rolling updates across a fleet happen as each device checks in and receives the updated manifest.

Pinning image tags to specific versions (like 1.2.0 rather than latest) makes deployments deterministic and ensures you can reproduce the exact running state of any device at any point in time.

Monitoring Module Health

IoT Hub surfaces module runtime status through the Edge Agent module twin’s reported properties. The runtimeStatus field for each module reports values like running, stopped, failed, or backoff, and the exitCode and statusDescription fields explain why a module exited. Operations teams can query these properties across thousands of devices to identify modules that are cycling or failing.

Restart counts are also visible in the reported properties. A module with a high restart count indicates a crash loop, which warrants investigation through the module’s container logs.

Reading Module Logs

The most direct way to inspect a module is to read its container logs from the edge device directly. Azure IoT Edge 1.2 and later added a log management feature that lets you pull logs from modules through IoT Hub using direct methods, which means you can retrieve logs from a remote device without needing SSH access.

# On the device directly
sudo iotedge logs TemperatureFilter --since 30m

# Module status overview
sudo iotedge list

# Check overall system health
sudo iotedge check

The iotedge check command is particularly useful when first setting up a device. It runs a suite of diagnostic checks covering certificate validity, DNS resolution, container engine connectivity, and port availability, and reports failures with actionable descriptions.

Common Troubleshooting Scenarios

Modules that enter a backoff state are being restarted repeatedly by the Edge Agent because they keep exiting. The Edge Agent applies an exponential backoff between restart attempts to avoid overwhelming the container engine. The first step is always to read the module logs for the last exit, since the cause is almost always visible there: an unhandled exception, a missing environment variable, or a failure to connect to a dependency.

Modules stuck in pulling state indicate that the Edge Agent cannot download the container image. This is typically a registry authentication problem, a network connectivity issue from the device to the registry, or an incorrect image tag. Checking the Edge Agent logs reveals which error the container engine returned.

If messages are not flowing between modules as expected, the route configuration in the deployment manifest should be verified first. A common mistake is referencing a module output name that does not match what the module code actually sends to. Route names and endpoint names are case-sensitive.

Security Architecture

Module Isolation

Each module runs in its own container with its own filesystem namespace and network namespace. Modules cannot access each other’s filesystems or address each other directly over the network. All inter-module communication passes through Edge Hub, which enforces the routing policy defined in the deployment manifest. A compromised module cannot directly inject messages that bypass routing rules.

Certificate-Based Identity

The Security Daemon provisions each module with a short-lived certificate scoped to that module. Modules use this certificate to authenticate to Edge Hub rather than using a shared connection string. This means compromising one module’s credentials does not compromise the device identity or the credentials of other modules.

The root of trust is either a hardware TPM or an HSM, with a software simulation available for development and testing. In production, using physical hardware for key storage significantly raises the bar for credential extraction attacks.

Securing the Container Registry

Module images are pulled from container registries such as Azure Container Registry. The registry credentials are stored in the deployment manifest’s Edge Agent desired properties and are protected by IoT Hub’s access control. Devices never receive plain-text credentials in their environment; Edge Agent retrieves them from the manifest and passes them to Docker only during image pulls.

Network Security Considerations

IoT Edge devices initiate outbound connections to IoT Hub and the container registry; they do not require inbound firewall rules. The only outbound ports required are 443 (AMQP over WebSocket or HTTPS for IoT Hub), 5671 (AMQP), and 8883 (MQTT over TLS). Organizations that restrict outbound access through a proxy can configure IoT Edge to route all outbound traffic through an HTTP proxy, which is common in industrial environments with outbound internet controls.

Leaf devices that connect to Edge Hub over MQTT need inbound access to the edge device on port 8883. In gateway scenarios, the edge device sits on both the device network (accessible to sensors and controllers) and the corporate or cloud network (with outbound internet access), acting as a controlled bridge between those segments.

Module Image Signing and Verification

For security-sensitive deployments, container images can be signed using Notation and verified by the container engine before running. Azure Container Registry supports image signing as part of its supply chain security features. Verifying image signatures before execution ensures that modules running on edge devices have not been tampered with in transit.

Supported Platforms and Hardware

Azure IoT Edge runs on Linux-based operating systems on AMD64, ARM32v7, and ARM64 architectures. The primary and recommended configuration is Linux containers on Linux. This covers a wide range of hardware including industrial PCs running Ubuntu or Debian, Raspberry Pi 4 and Pi 3 running Raspberry Pi OS, and purpose-built edge appliances from vendors like NVIDIA (Jetson series for GPU-accelerated inference) and Advantech.

For Windows environments, Microsoft provides EFLOW (IoT Edge for Linux on Windows), which runs Linux containers inside a Hyper-V virtual machine on Windows 10 and Windows Server. EFLOW allows organizations with Windows-based operational technology infrastructure to run IoT Edge without switching operating systems.

Hardware requirements scale with the number and complexity of modules. A device running two or three simple filtering modules needs modest resources (1 GHz processor, 512 MB RAM, 10 GB storage). A device running ML inference or Stream Analytics jobs benefits from at least a quad-core processor, 4 GB RAM, and SSD storage for the message buffer database.

Key Decision Points

When Edge Processing Is the Right Choice

Bringing compute to the edge makes sense when latency matters, connectivity is unreliable, data volumes make cloud-only processing expensive, or regulations restrict data from leaving a facility. It makes less sense for workloads where cloud services offer algorithms or data that cannot realistically be replicated on constrained hardware, or when the device fleet is small enough that bandwidth costs are negligible.

Latency-sensitive control loops are a particularly strong case for edge processing. A cloud round-trip adds hundreds of milliseconds of latency at minimum. A safety interlock that must react to a sensor reading within 50 milliseconds has no choice but to run locally, and IoT Edge modules can deliver sub-millisecond latency for module-to-module communication over the local Edge Hub.

Data sovereignty regulations are another strong forcing function. In sectors like healthcare, financial services, and defense contracting in some jurisdictions, raw sensor data may not legally leave a physical facility or geographic region. Edge processing allows compliance-relevant data to be processed locally and only anonymized or aggregated summaries sent to the cloud.

Edge vs. Embedded

IoT Edge is not a good fit for deeply embedded microcontrollers with kilobytes of RAM. The runtime itself requires a container engine and enough memory to run multiple containers. For constrained microcontrollers, libraries like .NET nanoFramework or bare-metal firmware are more appropriate. IoT Edge targets gateway-class hardware: devices with at least a few hundred megabytes of RAM running a full Linux or Windows operating system.

A common architecture combines both tiers: microcontrollers measure physical signals and transmit raw readings over a short-range protocol, while an IoT Edge gateway nearby performs protocol translation, filtering, and cloud forwarding. Each layer handles what it is suited for, and neither is being asked to operate outside its design envelope.

Module Granularity

Designing module boundaries follows the same tradeoffs as microservice design. Very fine-grained modules offer independent deployability and fault isolation but add routing and serialization overhead for each message hop. Very coarse-grained modules simplify deployment but reduce the ability to update individual functions independently. A practical starting point is one module per distinct responsibility such as protocol translation, filtering, aggregation, and inference, and consolidating where the communication overhead outweighs the deployment flexibility benefit.

Message Format Conventions

Modules that communicate through Edge Hub serialize and deserialize messages at every hop, so agreeing on a message format across a module pipeline matters early. JSON with UTF-8 encoding is the standard choice because it is human-readable during debugging and supported natively by the Azure IoT tools. Setting ContentType = "application/json" and ContentEncoding = "utf-8" as system properties on outgoing messages makes this explicit and enables content-based routing on those properties.

For high-throughput scenarios where JSON serialization overhead is measurable, binary formats like MessagePack or Protobuf can reduce message size and CPU cost. The trade-off is that binary messages are harder to inspect during troubleshooting and require both the sending and receiving module to share the schema definition.

Development Workflow

Local Development and Testing

The Azure IoT EdgeHub Dev Tool simulates the Edge Hub routing behavior locally without requiring a physical edge device. Modules connect to the local simulator the same way they would connect to a real Edge Hub, so integration tests can run in a CI pipeline against simulated inputs and routes.

The typical local workflow is to develop and unit test module logic as ordinary .NET code, then test routing and integration using the EdgeHub Dev Tool with Docker Compose, and finally deploy to a real device or a virtual machine for hardware validation before pushing to production devices.

The Azure IoT Edge extension for Visual Studio Code generates a standard module project scaffold, including the Dockerfile, deployment manifest template, and .env file for local registry credentials. It also adds commands to the VS Code command palette for building and pushing module images, which simplifies the inner development loop before setting up a full pipeline. Using this scaffold as a starting point is faster than assembling the project structure by hand, and it ensures the generated Dockerfile follows the official multi-stage build pattern that keeps production images small.

CI/CD for Edge Modules

The Azure DevOps IoT Edge extension provides pipeline tasks for building module images, pushing them to a container registry, and generating deployment manifests with updated image tags. A complete pipeline builds all module images in parallel, pushes them to Azure Container Registry, substitutes the new tags into the deployment manifest template, and applies the manifest to a staging device group for validation before rolling out to production.

Using a deployment manifest template with variable substitution for image tags means the manifest file in source control always reflects the current intended state, and the pipeline fills in the exact image digest at deploy time for reproducibility.

Unit Testing Module Logic

Because module business logic runs in ordinary .NET code before the ModuleClient is involved, it is straightforward to test filtering and transformation logic without any IoT-specific dependencies. The cleanest approach is to extract the core processing logic into a class that takes plain data objects as input and returns plain data objects as output, then test that class directly with xUnit or NUnit.

public class TemperatureFilter
{
    private readonly double _threshold;
    private readonly double _criticalThreshold;

    public TemperatureFilter(double threshold, double criticalThreshold)
    {
        _threshold = threshold;
        _criticalThreshold = criticalThreshold;
    }

    public AnomalyAlert? Evaluate(SensorTelemetry telemetry)
    {
        if (telemetry.Temperature <= _threshold)
            return null;

        return new AnomalyAlert
        {
            DeviceId = telemetry.DeviceId,
            Temperature = telemetry.Temperature,
            Timestamp = telemetry.Timestamp,
            Severity = telemetry.Temperature > _criticalThreshold ? "Critical" : "Warning"
        };
    }
}

The message handler in the IoT Edge module then delegates to this class, keeping the IoT SDK plumbing separate from the business logic. This separation means the filtering algorithm can have full unit test coverage without mocking ModuleClient or simulating the Edge Hub transport layer.

Comparing IoT Edge to Alternatives

Understanding where IoT Edge fits relative to other approaches helps in choosing the right tool for a given deployment scenario.

IoT Edge is the right choice when the deployment needs cloud-managed configuration, offline tolerance, inter-module routing, and a container-based module model, and the hardware can support a container runtime. For devices with less than 256 MB of RAM or without a Linux-capable processor, lighter alternatives are more appropriate.

Provisioning IoT Edge Devices at Scale

Device Provisioning Service Integration

Manually registering devices in IoT Hub is practical for small pilots but does not scale to hundreds or thousands of devices. Azure Device Provisioning Service (DPS) automates this by allowing devices to self-register when they first connect. The device presents a certificate or symmetric key to DPS, which verifies the credentials against an enrollment group and registers the device in the appropriate IoT Hub.

For IoT Edge devices, DPS also delivers the initial configuration that tells the device which IoT Hub to use and which provisioning mechanism to rely on. Combined with a base deployment applied automatically to matching device tags, a new device can go from unboxed to fully configured without a technician touching it beyond powering it on and connecting it to the network.

Enrollment Groups and Device Grouping

DPS enrollment groups allow all devices sharing a common root certificate or symmetric key to be registered under the same policy. A manufacturing line might use an X.509 certificate chain where each device certificate is signed by a line-specific intermediate CA. DPS validates the chain and applies the enrollment group’s IoT Hub assignment to all devices with that lineage, automatically placing them in the right IoT Hub and applying the right tags for deployment targeting.

This approach is particularly valuable for high-volume manufacturing runs where devices are provisioned in batches. Each device boots, contacts DPS, presents its factory-installed certificate, receives its IoT Hub assignment, and begins receiving its deployment manifest, all without an operator logging into any portal. The only manual step is ensuring the certificate authority chain is loaded into the DPS enrollment group before production begins.

Observability and Metrics

Built-in Metrics

IoT Edge 1.0.10 and later exposes a Prometheus-compatible metrics endpoint from both the Edge Agent and Edge Hub. These built-in metrics cover message counts by route, queue depths in the store-and-forward buffer, Edge Hub connection counts, and Edge Agent module restart statistics. Scraping these metrics with Azure Monitor managed Prometheus and visualizing them in Grafana provides a complete view of fleet health without any custom instrumentation.

The metrics endpoint is exposed on port 9600 by default. In constrained environments, the metrics scraper module from the IoT Edge metrics-collector project can push metrics to Azure Monitor or Log Analytics instead of waiting for a Prometheus scrape.

Custom Module Metrics

Custom modules can expose their own Prometheus metrics using the prometheus-net library for .NET. This allows modules to expose business-level metrics such as messages filtered per minute, inference latency percentiles, or protocol translation error rates, alongside the system-level metrics from the IoT Edge runtime.

// Register metrics at startup
private static readonly Counter MessagesProcessed =
    Metrics.CreateCounter("filter_messages_processed_total", "Total messages processed");

private static readonly Counter MessagesForwarded =
    Metrics.CreateCounter("filter_messages_forwarded_total", "Messages forwarded as anomalies");

private static readonly Histogram ProcessingDuration =
    Metrics.CreateHistogram("filter_processing_duration_seconds", "Message processing duration");

// In the message handler
using (ProcessingDuration.NewTimer())
{
    MessagesProcessed.Inc();
    var alert = _filter.Evaluate(telemetry);
    if (alert is not null)
    {
        await SendAlertAsync(moduleClient, alert);
        MessagesForwarded.Inc();
    }
}

Tracking the ratio of forwarded to processed messages over time reveals whether filtering thresholds are calibrated correctly. A ratio that suddenly spikes may indicate a sensor failure or an environmental change rather than a threshold misconfiguration, and that distinction is visible in the metrics before it shows up in incident reports.

Combining built-in Edge runtime metrics with custom module metrics in a single Grafana dashboard creates a complete operational picture from infrastructure health at the container level down to business-level indicators like anomaly rates and protocol translation error counts.