You don't buy a broker for the first consumer

Every talk about modern plant data ends on the same slide: a box labelled Kafka in the middle, with everything else in the plant arranged politely around it. If you run a plant, you have probably looked at that slide and asked the question nobody on stage answers directly. My gateway already writes history to a database. It has done that reliably for years. What exactly is broken?

For a lot of plants the honest answer is nothing, and the slide is selling you a component you would spend the next five years patching for benefits you would never collect. But past a certain estate shape there is a real answer, and it has nothing to do with speed or the phrase "real time". It comes down to two questions. When the database is down, who is holding the reading? And when the next system wants plant data, who takes the load?

What Kafka is, without the vocabulary#

Strip away the terminology and Kafka is a shared logbook. Programs that produce data append lines to it, one after another, and nothing ever edits or removes a line. Programs that read data each keep their own bookmark in the book. The historian's bookmark might be at line nine thousand while a slower reader is still back at line four thousand, and neither knows or cares about the other.

The part most people miss is that reading a line does not remove it. A conventional message queue works like a conveyor: a message is handed to whoever is waiting, and then it is gone. The logbook keeps every line, for as long as you configure it to, whether one program has read it or five have.

WHAT THE LOG ACTUALLY IS

Kafka is the name of both the idea and the original software, which is a Java system that grew up in data centres and traditionally needed its own coordination service and a team who understood it. Redpanda is a newer implementation of the same protocol: a single binary, built to run well on ordinary hardware without the surrounding zoo. Every Kafka client, tool and tutorial works against it unchanged. That is the whole reason it keeps coming up in plant conversations. It is not a feature list, it is fit: the same logbook, sized for the kind of server a plant actually owns.

The first deciding question#

Here is the scenario that separates the architectures. Your historian database goes down for a minute. Maintenance window, disk full, failover hiccup, it does not matter which. The plant keeps producing. Where do those sixty seconds of readings go?

With a direct gateway-to-database connection, holding those readings is the gateway's problem. Store-and-forward exists and it helps, but it turns your control-layer gateway into a buffering appliance, doing retry work with a finite cache on the same box that runs your tag engine.

With a logbook in the middle, the question dissolves. The readings were already appended, so the historian's bookmark simply stops advancing. When the database comes back, the reader picks up from its bookmark and works through the backlog in order. Nothing was lost, because nothing was in flight. It was written down.

WHERE A READING WAITS WHEN THE DATABASE IS DOWN

I have measured this twice, because it sounds too tidy to be true. In one rig I built the same pipeline two ways and cut the database for sixty seconds under load. The version that only acknowledged a reading once the database had confirmed it, with the log holding everything in between, delivered 100.00 percent. The version that took delivery first and inserted second lost 119,500 readings, permanently, in one minute. In a second rig, an Ignition 8.3 gateway published through a Redpanda cluster into a historian while I killed a broker and then took the database down for a full minute: ten out of ten heartbeat ticks arrived in every bucket, straight through both faults. The only loss left anywhere in that system was a gateway restart, which is exactly where a plant should be spending its redundancy money anyway.

The second deciding question#

Durability through outages settles the argument on paper, but the property that changes how an estate grows is a different one: who serves the readers.

Count what is coming. Today the plant data feeds a historian and a dashboard. Next year somebody wants a chatbot for the operators, then a second historian for the new analytics team, then a reporting tool that writes the morning summary, then corporate IT asks for a feed. In a direct-connected estate, every one of those lands on a critical system: another connection configured on the gateway, another query load on the historian, another party touching the control network. Those are exactly the systems the plant cannot afford to slow down, and they end up serving everyone because they are the only place the data lives.

With the logbook in the middle, every new consumer is a new bookmark in a book that already exists. The chatbot reads the log. The reporting AI reads the log. The IT feed reads the log. The SCADA system and the historian never feel any of it, because serving readers is the broker's whole job and it is built for exactly that. The logbook lets everything downstream read everything, while reaching nothing that matters.

WHO CARRIES THE READ LOAD

Two smaller properties come with the log, and both point the same direction.

A slow database cannot reach backwards. Direct coupling means a struggling historian eventually slows the thing feeding it, and a database problem becomes a controls problem. The log absorbs the mismatch, because the producer appends at its own pace no matter how far behind any reader falls.

A SLOW DATABASE CANNOT REACH BACKWARDS

And the past is replayable. Corrupt a historian, or stand up a new one, and you rebuild it by re-reading the book from page one. In my rig a fresh database reached 9,950 rows out of 9,950 in under a minute, and the gateway was never asked for anything. This is also the quiet enabler for the AI work: backfilling a model over last month's production is a replay, not a fresh extraction project against live systems.

THE PAST IS REPLAYABLE

None of this is a fringe position any more. Ignition 8.3 ships Kafka connectivity and Event Streams as first-class parts of the platform. When the most widely deployed SCADA platform treats the logbook as standard plumbing, the argument has left the conference circuit.

Why this is overkill for a single station#

Now the honest half. One PLC, one gateway, one historian, one team: the direct connection is the right architecture, and it is not close.

The built-in path from gateway to database is battle-tested, understood by every integrator you could hire, and fails in ways that are documented and familiar. The broker, in that setting, is a new service to patch, disks to watch, a consumer process somebody has to write and babysit, and one more thing standing between you and your data at two in the morning. Every benefit in this essay is a no-op with one reader. Fan-out with one consumer is a queue with extra steps. Replay of data nothing else will ever read is a backup you already had. Backpressure isolation between two services on the same rack, sized by the same person, rarely earns its keep.

OVERKILL IS A QUESTION OF SCALE

The rule#

Count the independent readers of your plant data, today and honestly over the next two years. The historian is one. A second historian, a corporate analytics push, a data lake, a chatbot, a reporting AI: each is another. At zero or one, keep the direct connection and spend the saved effort on gateway redundancy, because the gateway is the only place left that loses data. At two or more, or the first time someone asks you to rebuild a historian or backfill a model, the logbook stops being an extra component and becomes the architecture.

You don't buy the broker for the first consumer. You buy it because the second one is already on somebody's roadmap, and on a 2026 roadmap it usually is.

One more thing, since it is 2026. Somewhere in your organisation, right now, an internal team or a hired one is building an AI copilot for the plant, and it is going to need every reading with its context attached. Nobody asked you. The second consumer is not on a roadmap any more, it is in the building, asking where the data lives. There is no version of this year where you stay at one reader. Better the copilot finds a logbook than finds your gateway.

Every number in this piece was measured, and both rigs are written up in full: the delivery comparison where one design lost 119,500 readings in a minute, and the switch-off experiments where the broker-fed historian survived everything except the producer itself. For what sits on top of the log once it is there, see a Unified Namespace built from parts.