I never typed a single tag name: the SCADA built its own database

If you have ever commissioned a SCADA system you know where the week actually goes. Not the driver. Not the screens. It goes into the tag database, with somebody sitting next to a device list and a spreadsheet, typing names, choosing data types, setting scan classes, and getting a handful of them quietly wrong in a way that surfaces three months later when a production report refuses to add up.

MQTT Sparkplug B claims to delete that job outright. The argument is simple enough that it sounds like a trick: the edge device already knows its own data model, so it should say so, and the SCADA system should build its tags from what it hears. Bundled with that claim are three more. That you find out immediately when a device dies instead of waiting for a scan to time out. That you only send values which actually changed. That any number of consumers can join without the publisher being touched.

Four separate claims, and in my experience they get asserted far more often than they get measured. So I built the thing over an afternoon: four independent edge nodes publishing simulated plant data, a broker in the middle, Ignition 8.3 as the consumer, all on the single-node k3s cluster I keep around for this sort of question, which I wrote about in running a SCADA system on Kubernetes. Then I broke it on purpose and wrote down what happened.

Some of what happened was me being wrong, repeatedly, and I have left that in.

What I stood up#

Four sites, deliberately unlike each other, because a fleet where every node is a copy of every other node proves nothing about a protocol.

Twelve devices, eighty-two metrics, one Sparkplug Group named Northwind. Each edge node is a Node-RED instance running node-red-contrib-mqtt-sparkplug-plus, which turned out to be a properly complete client: it handles NBIRTH, DBIRTH, NDEATH and rebirth commands on its own, and it buffers when the primary consumer is away. The broker is EMQX. Ignition consumes through the Cirrus Link MQTT Engine module.

I generated all of it. Each site is a YAML file listing devices, metrics, engineering units, a simulation kind per metric and a deadband, and a Python renderer turns those into Node-RED flows and Kubernetes manifests. That decision paid for itself three times over, because every one of the bugs below turned out to need the same setting changed on all four sites at once, and each fix was one line of Python and a sixty-second redeploy rather than twelve trips through a flow editor.

The simulation is not random noise. Compressor suction pressure does a bounded random walk that reflects off its limits. Chill room air temperature is a sine with a slow period. Chemical tanks drain and refill. Bottle counters only ever increase. Inverter state sits on MPPT and occasionally drops to Derating or Fault. It behaves enough like a plant that report-by-exception has something real to chew on, which matters when the whole point is to measure what report-by-exception is worth.

The birth: eighty-two tags nobody typed#

Nothing in Ignition was configured to expect any of this. No device definition, no tag import, no UDT, no CSV. MQTT Engine subscribes to the Sparkplug namespace, and when an edge node connects it publishes a birth certificate: an NBIRTH for the node itself, then a DBIRTH per device listing every metric with its name, data type and current value.

Ignition built the entire tree from that alone.

The folder structure comes free as well. A metric named Pressure/Suction becomes a Pressure folder with a Suction tag inside it, so whatever naming convention the edge node uses becomes the SCADA hierarchy without a mapping table sitting between them. Engineering units ride along as tag properties, because the Node-RED node lets you attach metric properties to the birth message.

This is the part that lives up to the pitch, and it is worth being precise about why, because the usual explanation is wrong. It is not that MQTT is fast, and it is not that the payload is compact. It is that the payload is self-describing. The schema travels with the data instead of being agreed in advance by two teams on a call and then diverging.

The death: 250 milliseconds#

Every Sparkplug edge node registers a Last Will and Testament with the broker when it connects. The broker holds that message and publishes it on the node's behalf if the client vanishes without saying goodbye. That part is plain MQTT rather than Sparkplug, but Sparkplug is what gives it meaning, because the will is an NDEATH certificate carrying the node's birth-death sequence number.

I scaled the Rotterdam deployment to zero and watched the clock.

07:07:20.300  kubectl scale deploy rot-cold-01 --replicas=0
07:07:20.547  Handling LWT message for Edge Node Northwind/ROT-COLD-01

Two hundred and forty-seven milliseconds from the process dying to Ignition knowing about it. A polled system finds out when a scan class times out, which depending on tuning is somewhere between a few seconds and a minute, and it finds out one tag at a time rather than one site at a time.

The stale overlays are the detail I would point at in a design review. Ignition did not blank those readings and it did not silently freeze them. It kept the last known value and marked its quality bad, which is exactly what an operator needs to see: the number in front of you is real, it is simply not current.

Why pub/sub is the part that scales#

The bandwidth argument gets all the attention and it is the least interesting half of the story. The architectural half matters more.

In a polled system the SCADA server holds a connection to every site and asks each one, over and over, for values that mostly have not changed. Add a second consumer, say a reporting server or a data science team, and you either add a second set of polls on top of the first or you build an export. Every consumer is a new integration and every one of them adds load at the plant end.

With pub/sub the edge node publishes once, to one place, and does not know or care who is listening. During this lab I attached a mosquitto_sub client to the broker perhaps twenty times to see what was on the wire. Not one of those attachments required any change at any of the four sites, and none of them added a single byte to what the edge nodes transmitted. That is the same property that lets a historian, a dashboard and an analytics pipeline all consume the same stream without the plant ever being reconfigured.

Layer report-by-exception on top and the edge is publishing only deltas, only outbound, only once, to any number of listeners. That combination is the actual argument for this architecture at fleet scale, and it holds regardless of what the byte counts say.

Report by exception, measured honestly#

Now the byte counts, because the claim deserves a number rather than a promise. I measured the same fleet twice for sixty seconds: once normally, then again with every deadband stripped out so that all eighty-two metrics publish on every scan.

2.6 times less traffic. But look at where the saving comes from, because the naive reading of that number is misleading. Message count only fell by 23%, from 717 to 552. With twelve lively devices on a one-second scan, almost every device has something worth reporting most seconds, so you rarely skip a publish altogether. The saving lives in payload size: each message carries the two or three metrics that moved rather than all seven.

That has a practical consequence for anybody sizing a cellular link. Report-by-exception does not reduce how often you transmit anywhere near as much as it reduces how much you transmit, so your per-message overhead, meaning TCP, TLS records and the radio waking up, does not fall by 2.6 times. It falls by about a quarter. Size for that, not for the headline ratio. If your process is genuinely quiet, with tank levels that sit still for minutes at a time, the message count saving will be far larger than mine. My simulation was deliberately busy.

Store and forward, and the silence that gave it away#

Sparkplug's buffering is built around the idea of a primary host. The consumer declares an identity and publishes a retained STATE message saying it is ONLINE. Edge nodes subscribe to it. If it goes OFFLINE, or was never there at all, a well-behaved edge node stops publishing and buffers instead, then drains the buffer on recovery with every metric flagged historical so the consumer writes it into history rather than mistaking two-minute-old readings for current ones.

I discovered this worked before Ignition was even installed, and entirely by accident. With four edge nodes running happily and no consumer anywhere on the broker, I subscribed to the Sparkplug namespace expecting a flood and got complete silence. My first assumption was that I had misconfigured the flows. I spent a few minutes reading Node-RED logs that cheerfully said Connected to broker before the penny dropped. It was not a bug. It was the protocol doing precisely what it promises: with no primary host there is nobody to publish to, so the edge holds everything.

Publishing one retained message by hand started all twelve devices at once:

mosquitto_pub -h emqx -t STATE/IgnitionPrimary -m ONLINE -q 1 -r

Which is a satisfying demonstration and also a trap I then walked straight into. That message is retained, so it outlives the client that published it, and until you clear it your edge nodes will never buffer again. Every subsequent store-and-forward test I ran was quietly invalid until I noticed and cleared it with mosquitto_pub -r -n -t STATE/IgnitionPrimary. If you are testing buffering, check for a stale retained STATE before you trust a single result.

Then I did it properly and killed the broker for two minutes. The edges dropped, buffered, and reconnected twenty-eight seconds after the broker came back, which is MQTT reconnect backoff rather than anything Sparkplug controls. Data resumed without anyone touching Ignition.

Security: what Sparkplug gives you, and what it very much does not#

This is the section I would want if I were reading somebody else's write-up, so let me be blunt about it, including about my own lab.

Sparkplug itself specifies no security model. It defines a topic namespace, a payload format and a session state machine, and then points at MQTT for everything else. That is a reasonable division of labour, but it means every security property you want has to be configured on the broker and the clients. Nothing in a Sparkplug payload is signed, and nothing in it is encrypted at rest.

What you actually get, and what you have to switch on:

• TLS on the transport. Standard MQTT over 8883. This is the one everybody remembers, and it protects the data in flight and lets the client verify the broker.
• Client authentication. Either username and password, or X.509 client certificates. Certificates are the right answer for unattended edge nodes, because a password in a Node-RED credentials file on a box in a cabinet is a password you will never rotate.
• Broker ACLs on the topic namespace. This is the important one and it is the one people skip. Sparkplug's topic structure is spBv1.0/<group>/<type>/<edge_node>/<device>, which is exactly the shape you need for meaningful access control. A site's edge node should be allowed to publish only under its own group and node, and nothing else. Your historian should be allowed to subscribe and never to publish. Without ACLs, any client with valid credentials can impersonate any edge node, publish a forged NDEATH for a site it has nothing to do with, or issue a write command to a device.

That last point is worth sitting with. Sparkplug is bidirectional. NCMD and DCMD messages travel from the consumer back to the edge and can write setpoints. The protocol has no opinion whatsoever about who is allowed to send them. Broker ACLs are the only thing standing between a compromised credential and a remote write to a plant.

Cirrus Link have clearly thought about this, and MQTT Engine ships two controls I had to touch during the lab. Block Node Commands and Block Device Commands are both enabled by default, so outbound writes are refused until you deliberately allow them. I had to turn one off before my rebirth button would do anything, and being forced to make that decision explicitly is good design. There is also an Include Security Context option that attaches a hashed context to write commands for the edge node to validate, with a choice of SHA algorithms, which is a defence against a forged command arriving from a client that has the topic but not the shared secret.

There is one more architectural property that matters more than any of the above, and it is the reason security teams tend to like this pattern: every connection is outbound. The edge node dials the broker. The SCADA system dials the broker. Neither one listens for inbound connections, so a remote site needs no inbound firewall rule, no port forward, no VPN terminating at the plant, and no public address. For a fleet of small sites behind carrier-grade NAT on cellular connections, that alone often decides the architecture.

What I ran in this lab was none of that. Plaintext on port 1883, anonymous access, no ACLs beyond the EMQX defaults. It is a disposable lab on a private network and I would not pretend otherwise. If you copy the repo, the TLS listener and a per-node ACL are the first two things to add, and they are the difference between a demo and something you would put on a plant network.

One EMQX default did bite me, and it is worth knowing because it looks like an authentication failure and is not. EMQX 5 ships an ACL that denies subscription to a bare #. My first attempt to sniff the broker returned All subscription requests were denied, which sent me looking for a credentials problem that did not exist. Subscribe to spBv1.0/# instead and it works fine.

Where it bleeds#

Two independent implementations of a published specification disagreed in three places. Every one of them either failed silently or failed in a loop, and none of them appears in the documentation of either product. This is the part of the lab I would most want somebody to read before their commissioning week.

The STATE topic moved, and the default is probably wrong for you#

Sparkplug 2.2 used a flat STATE/<host_id> topic with a plain-text payload. Sparkplug 3.0 moved it to spBv1.0/STATE/<host_id> and made the payload JSON. MQTT Engine 5.0.0 publishes the new form by default. The Node-RED client subscribes to the old one.

Nothing errors. Nothing logs. The edge nodes simply never see a primary host and buffer forever, which presents exactly like a broken broker, and I lost a good twenty minutes chasing the broker before I thought to check the client's source and found it subscribing to a topic Ignition was not publishing on. The fix is a checkbox called Enable Legacy STATE messages, buried under the MQTT server's advanced settings and off by default. Turn it on and both forms go out together. Worth noting the Primary Host ID is not where you would expect either: it lives on the Server Set rather than the server, and defaults to IamHost.

Metric aliasing collapses when two devices share a metric name#

Aliasing is a bandwidth optimisation. The birth certificate maps each metric name to a small integer and the data messages then carry the integer instead of the name. I turned it on because saving bytes was one of the things I had come to measure.

The Node-RED client keys that map on the metric name alone, shared across every device on the edge node:

node.metricsAliasMap[metric.name] = ++node.nextMetricAlias;

My devices deliberately reuse names. Status/Running appears on a compressor, a filler and a palletiser, because that is what real plants look like. Ignition resolves aliases per device, so it kept receiving integers that meant nothing in that device's table. It then did the correct thing and asked for a rebirth. Then received another bad alias, and asked again:

Failed to find the metric name from alias 19 for Device=Northwind/LEI-BOTL-01/Palletiser
Requesting Rebirth from Northwind/LEI-BOTL-01, cause: Failed to find metric name from alias

That loop ran for as long as I let it and would never have converged. The spec says aliases must be unique across the edge node, and a metric is identified by device and name, so I read this as a client bug rather than a genuine ambiguity in the standard. Either way the practical answer is to switch aliasing off, and to accept that the 2.6 times I measured above would have been a slightly better number with it on.

Waiting for values before birthing breaks reconnection#

By default the Node-RED device node holds its DBIRTH until it has seen a value for every configured metric. On first start that is sensible. After a broker outage it is not, because with report-by-exception the metrics that have not moved may not be resent, so the post-reconnect birth can be incomplete. Ignition then logs an unknown metric and requests a rebirth on every subsequent message, and you get a second loop that looks nothing like the first but has the same root shape.

I watched that one run for several minutes before I understood it, largely because I had just fixed the alias storm and assumed I was looking at the tail of the same problem. Setting birthImmediately so the node births from its configured metric list straight away makes it disappear.

Look at the birth counters in that screenshot. Twenty-four on Seville is not a healthy number for a link that has been up for half an hour, and before the fixes it was climbing every few seconds. Birth count is the diagnostic that tells you a Sparkplug link is fighting itself. If you take one operational habit from this piece, make it that one: put node birth count on a dashboard and alarm on its rate of change, not its value.

Two mistakes that had nothing to do with Sparkplug#

Since the point of a lab notebook is the notebook, here are the two that cost me the most time and were entirely self-inflicted.

I took the Proxmox host off the network with a file copy. The MQTT Engine module is a 53 MB .modl, and I copied it from my laptop to the homelab over a relayed Tailscale link. Somewhere around 40 MB the host stopped answering, and it stayed unreachable for about fifteen minutes while I sat there wondering what I had broken. Nothing, as it turned out, but the fix is obvious in hindsight: download large files inside the target machine rather than pushing them across a fragile tunnel. There is no curl in that Ubuntu container, so it is wget with a browser user-agent, because the download endpoint rejects anything else.

I guessed at a file format for an hour when I could have read it in one command. The wallboard in this piece is generated rather than drawn in the Designer, because Ignition 8.3 keeps project resources as plain JSON on disk. Getting that right was miserable. A page mapping with the wrong shape gives you "View Not Found". A map transform with an invalid outputType shows a quality-error overlay on the component, which looks exactly like a missing tag and sent me hunting for tags that were fine. A component action missing a scope field takes down the entire project with a null pointer during serialisation. Each wrong guess cost a ninety-second gateway restart.

What finally worked was to stop guessing and read the schema out of the module's own jar, because the compiled classes carry their field names:

PagesConfig       → Map<String, PageConfig>   (keyed by URL, not an array)
PageConfig        → viewPath, title, docks
MapTransformType  → scalar | color | inlineStyle | styleList | range | expression | document

Three failed guesses and two restarts, replaced by one command. If you ever intend to generate Ignition project resources rather than click them, start there rather than where I started.

The module install has the same flavour. Dropping a signed .modl into the external modules folder does nothing on its own; you also need an entry in data/modules.json giving the module id, its path and the SHA-1 fingerprint of the signing certificate, which you can extract from the certificates.p7b inside the module itself. That gets you ninety per cent of an unattended install, and then the gateway parks in commissioning until a human ticks a licence agreement in a browser. It is the same wall I hit generating ten site gateways from a repository. The configuration generates. The trust does not.

How much will it actually carry?#

Four sites proves the mechanism. It says nothing about whether the mechanism scales, and the whole reason anyone reaches for this architecture is fleets. So I tried to break it.

The first constraint is not the protocol, it is Kubernetes. One Node-RED pod per edge node hits the k3s ceiling of 110 pods per node long before it runs out of memory, and a thousand pods at 53 MB each would want 53 GB of RAM I do not have. The way through is that one Node-RED runtime can host many Sparkplug edge nodes. Each one is still a separate EoN with its own MQTT client, its own birth-death sequence and its own lifecycle; they simply share a process. I added a PACK knob to the generator that packs fifty edge nodes into each pod, which turns 1500 edge nodes into thirty pods.

A second Kubernetes gotcha showed up immediately. kubectl apply stores the entire manifest in a last-applied-configuration annotation, and annotations are capped at 256 KB. My generated flow ConfigMaps are around 320 KB each, so every apply failed with "metadata.annotations: Too long". Server-side apply skips that annotation entirely and works fine.

At 500 edge nodes, nothing interesting happened, which is the good outcome. EMQX reported 506 connections, 1025 topics and 2033 subscribers while using 481 MB and 89 millicores. The Node-RED pods sat at 64 MB each, and remember each of those is hosting fifty Sparkplug clients. The gateway took 175 millicores and 1175 MB, and all 500 edge nodes registered with Ignition inside the rollout.

At 1500 edge nodes, the broker still did not care: 1469 connections, 2953 topics, 6330 subscribers, 543 MB, 237 millicores. I want to be clear about that because it is the least surprising and most reassuring number in this whole piece. The broker is not your problem.

Ignition was, and here I got it wrong in an instructive way. I brought the fleet up, looked at the wallboard during the rollout, saw two of my four real sites showing OFFLINE with data latency of 1148 and 5470 milliseconds, and concluded I had found the ceiling.

It was not the ceiling. It was the birth storm. Fifteen hundred edge nodes connecting in the same few seconds means MQTT Engine creating roughly thirty thousand info and control tags at once, because it builds around twenty status and command tags per edge node on top of your actual metrics. Left alone for a couple of minutes the same fleet settled to all-green at 0 to 1 millisecond latency. Mass birth is expensive; steady state is cheap. Those are two different capacity questions and I had conflated them.

What the settled state did reveal was the real constraint, and it was not hardware. The gateway pod was pinned against limits I had set months earlier for a two-gateway lab:

args:      ["-n","lab-gateway","-m","896"]     # 896 MB JVM heap
limits:    cpu: 2,  memory: 1600Mi

It was sitting at 1510 Mi, which is 94% of its memory limit, and burning 1376 millicores sustained. The host underneath had ten gigabytes and six cores and was idling at 20%. This was a container squeezed into a box, not a machine out of capacity. Raising the heap to 3 GB and the limits to four cores and 4 Gi:

Steady-state CPU fell by roughly seven times. Almost all of the old number was garbage collection thrashing against an undersized heap rather than real work, and the settled memory of 2159 MB is above the old 1600 Mi limit, so the container had genuinely been asking for more than it was allowed. If you are running MQTT Engine against a large fleet, size the heap for the tag count before you conclude anything about throughput.

That is the good news. The honest bad news is that I did not get 1500 edge nodes to a stable resting state. Even with the limits raised and CPU back down at a comfortable 596 millicores, the gateway logged a sustained loop:

1671  Requesting Rebirth from LoadTest/LOAD-N, cause: Received a message for edge node that is offline
1265  Edge Node LoadTest/LOAD-N is offline but we got a DBIRTH from device Skid-01, requesting Rebirth

Around two thousand of those per minute, spread across 1464 distinct edge nodes. Ignition believed almost every node was offline while continuing to receive its data, which is a desynchronised state machine rather than a resource shortage, and it degraded the four real sites along with the synthetic ones. Slowing the load fleet's scan from one second to ten made no difference. I then tried staggering the edge nodes' first publish over five minutes, which also made no difference, and the reason is worth writing down because it is a genuine misunderstanding on my part: a Sparkplug node births when its MQTT client connects, not when it first publishes. Staggering the inject staggered the data and left fifteen hundred simultaneous connections exactly as they were. Testing that hypothesis properly needs manualEoNBirth with delayed connect commands, and I stopped rather than keep drilling.

So the range I can defend is this. Five hundred Sparkplug edge nodes on one modestly sized gateway is comfortable. Fifteen hundred is beyond what this gateway would hold steady, and the failure mode is not a crash or an out-of-memory kill but a self-sustaining rebirth loop that quietly degrades every site on the gateway, including the ones you care about. Whether that ceiling is inherent or just needs staggered connects, I do not know yet, and I would rather say so than publish a number I have not earned.

One last piece of lab-notebook honesty. Partway through the load test the Proxmox host dropped off the network and rebooted, and my first instinct was that I had finally broken something. I had not. There were no out-of-memory kills, no kernel panic and no shutdown sequence in the previous boot's journal, just a log that stops mid-sentence and a machine that comes back four minutes later. That is a power cut, which is a thing that happens here. I nearly wrote up a hardware limit that did not exist, and the only reason I did not is that the journal disagreed with me.

So is Sparkplug the right protocol for edge to central?#

For a fleet of remote sites reporting to a central system, I think it is, and the reasons are not the ones usually given.

Against polled OPC UA, the decisive factor is direction. Classic OPC UA has the client connecting to the server, which for a central SCADA polling remote sites means inbound connections into every plant network. OPC UA Reverse Connect exists precisely to solve this and lets the server dial out to the client instead, but you need both ends to support it, and you are still managing point-to-point sessions per site. Sparkplug gets the same outbound-only property for free and adds one-to-many for nothing. On the other hand OPC UA carries a far richer security and information model natively, with per-node access rights and typed structures, where Sparkplug leans entirely on the broker. If everything lives inside one plant network and there is no firewall in the path, OPC UA is the better-specified choice and I would not migrate a working system.

Against a REST or webhook integration, Sparkplug wins on state. A REST endpoint knows nothing about whether the site it last heard from is alive. Birth, death and sequence numbering are what turn a stream of readings into a system that knows its own health, and building that on top of HTTP means reinventing it badly.

Against OPC UA PubSub over MQTT, which is the genuinely interesting comparison in 2026, the honest answer is that it depends on your vendor support. Part 14 gives you OPC UA's information model and security travelling over the same MQTT transport, which is architecturally the best of both. Sparkplug's advantage today is that the tooling is simply everywhere, from PLC vendors to brokers to SCADA modules, and my Node-RED client and Ignition module found each other without me writing a line of integration code. That ubiquity is worth a great deal, as this lab's three interoperability bugs notwithstanding.

Where I would not use it: anything requiring deterministic timing or sub-100-millisecond control loops, which is not what MQTT is for and never has been; and single-site installations with no firewall between the SCADA and the plant, where you would be adding a broker to solve a problem you do not have.

The honest verdict#

The central claim holds up. Eighty-two tags, twelve devices, four sites, and nobody opened a tag editor. Add a device in Node-RED and it appears in Ignition inside a second, correctly typed, in the right folder, with its units attached. If you run more than a handful of remote sites, that is not a convenience feature. It is the difference between a commissioning process that scales and one that does not.

The state awareness turned out better than I expected, and it is the part I would show an operations manager rather than an architect. A quarter of a second to know that a site is gone, and stale values that announce themselves as stale, is a materially different operating experience from a polled system that goes quiet and leaves an operator wondering whether the number on screen is five seconds or five hours old.

The bandwidth story is real but oversold. 2.6 times on payload is worth having. It is not the order of magnitude the marketing implies, and on a metered link your message count matters as much as your byte count.

What I did not expect was the interoperability. Sparkplug's entire promise is that independently written implementations of a published spec will simply work together, and here two mature ones disagreed three times: on a topic name the spec changed between versions, on the scope of an alias namespace, and on when a birth certificate is complete. Each failure was silent or self-perpetuating. Each one took me longer to find than it should have, mostly because a system that is buffering correctly and a system that is broken look identical from the outside.

That is not an argument against Sparkplug. It is an argument for putting the exact pair you intend to run on a bench before anybody drives to site. The protocol being open is what let me read the client's source and the module's compiled classes and find all three in an afternoon. A closed pair would have left me filing tickets and waiting.

The repo is four YAML files in, four edge nodes and a generated Ignition project out. Which is, in the end, the same argument as the tags: describe the thing once, and let the machines build the rest.