A site with no SCADA: store-and-forward from an edge that costs nothing

The estate in the previous lab had a SCADA gateway at every site, which made the architecture simple and the licensing expensive. Real estates are mixed. Some sites have operators, screens and control logic and genuinely need a SCADA product. Others have a sensor, a panel and a network drop — nothing to look at, nothing to control, nobody there.

Putting the same licensed gateway at the second kind of site buys a SCADA system to move bytes. This lab is the alternative, held to the same standard as the licensed one: the same link cut, the same heartbeat, the same bucket count.

The rule this rig is built on#

The obvious way to get durability at a site is to historise locally and let the database sync. That is what the previous lab did, and it worked — but it points a database connection across the link, and that is a bad transport when the link is unreliable. A connection pool is stateful and thrashes on reconnect. Every sample is a row and every write is a round trip. When the link returns, the spool flushes as a burst of bulk inserts across the very link that just proved it cannot be trusted.

Messaging was designed for the opposite conditions: one session that expects to drop, small payloads, delivery guarantees, and buffering in the publisher.

Carry events across the link. Write the database at the centre, on a local network.

	database over the link	events over the link
what crosses the link	SQL statements, one per sample	small events, batched
connection style	stateful pool, thrashes on reconnect	one session that expects to drop
on reconnect	burst of bulk inserts	ordered replay from a spool
where the buffer lives	a database spool at the site	the publisher
database written	across the link	at the centre, on the LAN

The rig#

Edge	A publisher process at the site — no SCADA product, no licence
Spool	SQLite on local disk, oldest-first, deleted only after the broker acknowledges
Transport	MQTT to a central broker, QoS 1, persistent session, short keepalive
Broker	EMQX at the centre
Ingest	Redpanda Connect, MQTT input into a SQL insert
Store	The same TimescaleDB hypertable the licensed path writes to

·sampleread the signal, write a row to the local spool

·publishdrain loop sends the oldest rows whenever the broker is reachable

·acknowledgebroker confirms receipt

✓deleteonly now is the row removed from the spool

The agent samples once a second, writes each reading to the spool, and a separate drain loop publishes from the spool whenever the broker is reachable. Nothing is deleted until the publish is acknowledged, which is the whole mechanism. The site is called Seville and it produces the same one-per-second signal the licensed sites do, so the two paths can be compared bucket for bucket.

Anything that speaks the protocol works here. This one is about eighty lines of Python because that made the buffering explicit for the write-up; in production the same job is a Telegraf configuration, a Node-RED flow with a disk-backed queue, or a small Go binary, and the choice mostly comes down to how many protocols the site speaks.

The cut#

Same shape as before: isolate the site from the centre for about two and a half minutes, restore it, then count rows per ten-second bucket for the heartbeat tag.

The spool went from empty to two hundred and sixty queued messages during the outage, and back to empty within seconds of reconnection. Every bucket across the window came back at nine or ten rows, matching the pre-cut baseline — the nines are an artefact of a one-hertz sample against a ten-second boundary, not loss.

link uplink down — spooling to diskdrained

That is the same result the licensed path produced, from a site with nothing licensed on it.

The run that proved nothing#

The first time I ran this cut, it produced a perfect backfill and I nearly wrote it up. It was perfect because there was no outage.

The agent runs in a container with bridge networking. I cut the link with a rule on the OUTPUT chain, which is where locally generated traffic goes — and a bridge container's traffic is not locally generated. It traverses the FORWARD and DOCKER chains and is translated on the way out, so the rule matched nothing and the connection stayed up for the entire "outage". The earlier gateway cuts were valid only because those containers use host networking, where OUTPUT does apply.

The correct rule for a bridge container is on DOCKER-USER. The rule is not the lesson. The lesson is that two signals would have caught it in fifteen seconds, and both were sitting there unread.

signal	first run	valid run
iptables chain used	OUTPUT	DOCKER-USER
container networking	bridge	bridge
did traffic actually stop?	no	yes
client logged a disconnect	never	within 15 s
spool depth during outage	0	260
what the data showed	perfect backfill	perfect backfill

The client never logged a disconnect, and the spool never grew. Both are direct evidence about whether the fault happened, and neither has anything to do with whether the output looks healthy. A zero-loss result from a test that did not inject the fault is not a weak result, it is a meaningless one, and it reads exactly like a strong one.

What it costs to run#

Worth stating plainly, because cost is the argument.

protocol driverin the licencein the agent

store-and-forwardin the licence≈ 80 lines + SQLite

local HMIin the licencenot provided

alarming with acknowledgementin the licencenot provided

licence cost per collect-only siteper gatewaynone

The centre is unchanged either way — the same broker, the same ingest, the same database. The difference is entirely at the sites, and it scales with the number of sites that have nobody standing in front of a screen.

Where the licensed product still wins#

This is not an argument that SCADA is unnecessary. It is an argument about which sites need it.

A site with operators wants alarming with shelving and acknowledgement, an HMI somebody is trained on, control logic with an audit trail, and a support contract that somebody else is accountable for. Reimplementing that on top of a message broker is a project, not a saving. At those sites the licence is buying something.

At a site with no operators, the same licence buys a protocol driver and a store-and-forward queue, and both of those are available in the open stack for nothing. The useful architecture is not one or the other. It is one backbone that accepts both: licensed gateways publish to the same broker as the open agents, the centre ingests one stream, and the question at each site becomes what that site actually needs rather than what the estate standardised on.

site with operatorslicensed gatewayscreens, alarms, control──▶

site with operatorslicensed gatewayscreens, alarms, control──▶

collect-only siteopen agent + spoolno licence──▶

collect-only siteopen agent + spoolno licence──▶

the centreone broker · one ingest · one database

What transfers#

• Carry events over the unreliable link and write the database on a local network. Buffer in the publisher, not in a database connection.
• Delete from the spool on acknowledgement, never on send. That single rule is what makes the backfill exact.
• Decide the edge product per site, on whether anyone is standing in front of it.
• When a fault-injection test reports zero loss, check that the client saw the fault and the spool grew. If neither happened, you measured nothing.

The commercial argument built on these numbers is in the companion post, and the invalid run has a post of its own. The licensed estate this was measured against, including the failover finding that reshaped it, is in the estate lab. Why the cut costs anything at all on a live subscription — the hole this spool exists to close — is in its own post.