I switched off every part of a streaming plant pipeline. The data only died once.

Suppose you take the drawing where plant data flows through a broker on its way to the historian, and you start switching things off. The broker. The database. The SCADA gateway itself. Which of those outages loses plant data, and how much?

I had a guess before running this, and it was wrong in a useful way. So this week the pipeline from last week's article stopped being a diagram and carried real readings: an Ignition 8.3 gateway publishing every tag change to a three-broker Redpanda cluster, and a small consumer on the far side writing each one into TimescaleDB. Once it had run long enough to trust, I began pulling plugs.

THE PIPELINE, AND WHAT GOT SWITCHED OFF

What the pipeline looks like when nothing is wrong#

The gateway watches a handful of tags and publishes each change as a small record: the tag path in the key, the reading in the value. One tag ticks once a second, which turns out to be the most useful instrument in the whole rig. A steady one-per-second heartbeat means any gap in the stored history is visible by simply counting rows in ten-second buckets. Ten per bucket is health. Anything less is loss, and there is nowhere for loss to hide.

In steady state a reading takes about two thirds of a second to travel from the gateway to a database row, and the consumer runs essentially caught up, three or four messages behind the front of the stream at any moment. Every config shape and measurement behind this piece is written down in the lab notes.

Switching off a broker#

The cluster keeps three copies of every record on three separate brokers. That is a cost you pay up front, roughly triple the storage, and this test is where the payment comes back.

I deleted one of the three brokers while readings were flowing. Leadership for its share of the data moved to the surviving copies in well under a second, the deleted broker rejoined about forty five seconds later, and the heartbeat count read ten out of ten in every single bucket straight through the incident.

TEST 1 · A BROKER DIES, THE RECORD DOESN'T

10

10

10

10

10

10

10

10

10

↑ broker deleted here

Nothing downstream noticed either. The consumer kept writing rows as if the afternoon were entirely ordinary. If nobody had been watching the cluster console, the event would have no witnesses at all.

Switching off the historian#

This is the outage I find most plants are actually afraid of, because in the conventional drawing the historian is the only place the data goes. Take the database down and the readings arriving during the outage are simply gone.

Here the database went down for a full minute and the story was different. The consumer tried to write, failed, and stopped advancing its bookmark. The readings it could not deliver did not vanish; they stayed exactly where they already were, in the broker's log, which kept accepting new ones. When the database came back the consumer picked up from its bookmark and worked through the backlog in seconds, stamping each row with the time the reading was actually taken rather than the time it finally arrived.

TEST 2 · THE HISTORIAN GOES DOWN AND LOSES NOTHING

Count the buckets afterwards and the minute of downtime is invisible. Ten out of ten, every bucket, including the ones recorded while the database was off. The historian stopped being the only place the data lives, and that single change turned a data-loss incident into a maintenance non-event.

Switching off the gateway#

Then I restarted the SCADA gateway itself, and here the accounting finally shows a wound. From the moment the process stopped until its tag engine was back on its feet, roughly forty five seconds of plant behaviour was never measured at all.

TEST 3 · THE ONLY OUTAGE THAT COST DATA

No broker, no replication factor, no clever consumer can help with this, because the loss happened before the first hop. A reading that was never taken cannot be forwarded, buffered, or replayed. The producer is the one link in the chain that redundancy further downstream cannot cover. If those forty five seconds matter in your plant, the answer lives at the gateway layer, in redundant gateways or buffered edge publishers, not anywhere in the pipeline behind it.

That asymmetry is the finding I carried out of the whole exercise. Everything downstream of the first hop can now fail without losing plant data. The first hop itself is where all the remaining risk lives.

The part where I stopped trusting the software and typed a value in by hand#

To check that the pipeline treats data on its merits rather than by its origin, I opened the cluster's console and published one record by hand: a made-up tag path and the value 42.5. A quarter of a second later it was a row in the historian, indistinguishable in every column from the rows the gateway produces, and filed under the same ordering rules.

NO PRIVILEGED PRODUCER

That sounds like a parlour trick, but it is really the property that makes the whole architecture composable. The pipe has no privileged producer. Anything that can speak the protocol and honour the little contract of key and value is a first-class source, which is precisely how a second site, a test rig, or a colleague's script joins the same historian without anyone rewiring anything.

Rebuilding the historian from nothing#

The last experiment is my favourite. I created a brand-new empty table, pointed a fresh consumer at the broker, and told it to start reading from the beginning of time.

It rebuilt the entire history, nine thousand nine hundred and fifty rows, first reading to last, in under a minute. The gateway was never asked for anything. The original consumer never noticed it had a sibling.

REBUILT FROM NOTHING

THE ORIGINAL

9,950

built over 10 hours

THE REBUILD

9,950

built in under a minute

the gateway was never asked for anything

Sit with that for a moment, because it quietly inverts where the truth lives. The database I have been calling the historian is now just one reader's copy. The durable record is the broker's log, and any number of independent readers can each hold their own bookmark into it. A corrupted historian, a second historian in a different database, a migration you want to trial without risk: all of them are the same operation, a new consumer starting at zero.

What I would tell the person still drawing the old diagram#

The broker in the middle is not a message courier, it is a short-term memory that everything downstream can be rebuilt from. Once it is in place, outages of brokers and databases stop costing data and start costing only attention.

What it cannot do is remember something that was never measured. Spend your redundancy budget at the producer, because after this week I can show you, bucket by bucket, that everywhere else already forgives you.

If you arrived here without the background, the plain-language version of this whole argument, including what Kafka and Redpanda actually are and when a broker is overkill, is You don't buy a broker for the first consumer.