Resilient IIoT gateway store-and-forward

A store-and-forward gateway is useful when telemetry must cross an intermittent WAN without making the cloud part of machine control. Its purpose is to preserve a bounded set of records until an upstream path returns. It is not a zero-loss promise: disks fail, power can interrupt a write, capacity is finite, and an accepted MQTT publish does not prove a downstream application stored the record.

Define the record contract first

A queue cannot recover meaning that was absent at intake. Each accepted record needs an event_id, source, sequence within a source epoch where available, measured_at, received_at, quality, and schema version. Measured time describes the source observation; received time describes local gateway intake. Keep them separate so a delayed delivery is visible rather than rewritten as a fresh measurement.

Define which messages may be queued. Sampled telemetry, alarm transitions, configuration snapshots, and diagnostic events have different retention, order, and loss rules. A command is not merely another message type. Commands and safety decisions remain with the controller, approved local interlocks, and authorized personnel.

Build the durable path

The normal path is source to durable queue to publisher to broker acknowledgement to delete or mark sent. Persist before treating a record as accepted. The publisher reads only complete records, sends a bounded batch, and advances the sent marker only after the configured MQTT acknowledgement. A crash between acknowledgement and marking can create a duplicate; design consumers to accept that possibility.

Do not rely on an in-memory buffer for an outage requirement that includes restart or power loss. Disk persistence introduces permissions, encryption-at-rest, wear, file-system reserve, and corruption recovery. The design must say which failure it covers instead of calling every buffer durable.

Size the queue with an outage budget

Start with capacity greater than or equal to rate times average payload times outage window times safety factor. For 20 records per second, 600 average encoded bytes, four hours, and a 1.5 factor, the payload budget is 20 × 600 × 14,400 × 1.5 = 259,200,000 bytes, about 247 MiB.

That number is not disk allocation. Add queue indexes, record headers, encryption or database overhead, log rotation, retained diagnostics, and a reserved free-space margin for the operating system. Recalculate from peak rather than average rate, and include the drain rate needed after an outage. A queue that can only publish as fast as it receives never catches up.

Persist safely through restart

Use a record format with length, checksum or integrity validation appropriate to the store, and an atomic commit boundary. On startup, scan records, accept only complete committed entries, quarantine corrupt segments, and report the resulting gap. Never guess that a partially written record is valid because its payload happens to parse.

Expose queue states such as online, buffering, draining, degraded, and stopped. Draining must continue to accept new data within limits while preventing old replay traffic from monopolizing CPU, disk, or broker bandwidth. A restart should recover the documented state, not silently discard unknown work.

Replay in order without assuming exactly once

Preserve per-source order when the consuming contract requires it. Global ordering across independent sources is usually neither available nor useful. Publish bounded batches, rate-limit replay beside current traffic, and use exponential backoff with a ceiling when the broker rejects or cannot accept work.

The consumer should deduplicate with source plus event_id, or with a documented sequence epoch and idempotent update rule. Arrival order is not a substitute for source order. MQTT QoS can change transport delivery behavior, but it does not eliminate duplicates across restart, reconnect, or application processing.

Make capacity exhaustion visible

Every class needs a visible policy for TTL expiry, disk-full conditions, publisher backpressure, and malformed records. Telemetry may drop oldest samples and create an explicit gap. High-value events may stop intake and raise a health alarm. Rejected records can enter a bounded dead-letter location with reason, source, and count; that location also needs a retention limit.

Never silently evict records simply because the volume is inconvenient. Report the policy outcome in metrics and operator documentation. A full disk can affect services beyond the gateway, so reserve space and alarm before the operating-system threshold.

Observe health, not just connectivity

Track queue depth and bytes, oldest record age, free disk, accepted and rejected counts, TTL expiry, duplicate detections, retries, publish latency, dead-letter depth, and recovery failures. Report a health state that identifies whether the gateway is online, buffering, draining, or degraded.

A green TCP session is not sufficient health. It says nothing about stale source data, a blocked queue, failing persistence, rejected authorization, or downstream consumer behavior. Use bounded diagnostics that do not expose credentials, private keys, or unrestricted operational payloads.

Test the failure matrix

Test a planned upstream outage, restart while buffering, restart while draining, full disk, write failure, corrupt segment, duplicate publish, reordered arrival, expired TTL, and broker backpressure. For each case, state the expected queue state, accepted or rejected record behavior, health signal, and recovery action.

These tests begin offline or in an isolated integration environment. They do not validate PLC scan timing, plant dynamics, electrical installation, or a safety function. Commissioning on the configured system needs its own authorization and acceptance record.

Deploy and roll back deliberately

Before deployment, baseline queue format, retention policy, broker endpoint, TLS identity, ACLs, disk location, reserve threshold, and source contract. Define who can stop intake, preserve the old store, restore the previous version, and confirm that recovery did not replay commands.

Use this checklist: verify free-space reserve; verify one read-only source contract; simulate an outage; confirm age and depth alarms; confirm bounded drain; confirm duplicate handling; confirm TTL and disk-full actions; and record remaining gaps. Continue with MQTT payload design, IT/OT convergence, and IIoT cloud services.