I killed the master gateway. The screens came back in eight seconds and the data never did.

If you pay for a redundant pair of SCADA servers, you are buying an answer to one question: what happens when the primary dies. The answer you are sold is that the backup takes over and nobody notices. Mine took over in eight seconds, which sounds like the story ending well.

It was not until I looked at the historian that I found the estate had stopped recording, and stayed stopped for six and a half minutes, until I intervened by hand.

What the test looked like#

Three sites feeding a central pair, and the central pair publishing everything it sees into a stream that lands in a historian. One tag at each site ticks once a second, so I can count rows in ten-second buckets afterwards and see exactly what was recorded and what was not. Ten per bucket is health. Anything less is a hole, and the hole has a start and an end.

With that running, I killed the master server outright — not a graceful shutdown, the equivalent of pulling its power.

The parts you can see all worked. The backup promoted itself and started answering. The proxy in front of the pair noticed within a couple of seconds and sent the operator traffic to the survivor. The screens went blank for a moment and came back on the same address, which is precisely what the drawing promises.

Underneath, the historian stopped receiving anything at all. Not a reduced rate, not a delayed batch. Zero rows from all three sites, for as long as the master stayed down.

The part that makes this worth writing down#

My first assumption was that the backup was stale. It was reporting itself as out of date, which is a satisfying thing to find because it suggests a synchronisation problem and those are somebody else's fault. I re-synced it, confirmed it was healthy, and killed the master again.

Same outcome. Screens in eight seconds, data never.

So it was not a sync problem, and at that point it was worth reading the documentation properly instead of trusting a system I had built myself. The relevant sentence is short: a connection between two gateways does not automatically move to the backup when the primary fails, and you cannot configure one on a backup node at all. Configuration replicates in exactly one direction, from the master outward.

Which means the direction the connection was opened decides everything.

I had each site reach out to the central server. That is the intuitive way round. Data flows from the site to the centre, the site is the thing being added, so the site connects inward — every instinct says build it that way, and it works perfectly. It keeps working perfectly right up until the moment the master dies, at which point the backup discovers it has no relationship with any site at all. It has the tag model, the projects, the alarm configuration and a licence. It has nothing to collect.

Turn the arrows around, so the central pair reaches out to the sites, and the connections are part of the master's configuration. They replicate. The backup holds live links to every site before it is ever needed, which is the entire point of a warm standby.

I reversed it and ran the same kill. Every site held ten rows in every bucket, including the bucket containing the moment I killed the server. The failure cost nothing at all.

Why this class of mistake is worth fearing#

A wrong password fails immediately. A wrong subnet fails immediately. This fails in eight seconds, once, on the day you can least afford it, and until then every indicator you own is green.

That is the property that makes it dangerous rather than merely annoying. The connections are up. The pair reports itself healthy and synchronised. The screens demonstrably survive a failover, so the redundancy looks tested. If you validated your standby by failing over and checking that operators could still see the plant, you validated the half that was never at risk.

The half that was at risk is the collection path, and it is invisible from the control room because its symptom is an absence in a database nobody is watching during a failover.

The general shape of it#

Redundancy protects a component. It does not protect a relationship.

The pair was genuinely redundant. The tag model, the projects, the licences, the alarms all survived. What did not survive was the set of connections between the centre and the sites, because those had been established by the sites, and the sites had no idea a second server existed. Nothing in the estate was configured to tell them.

So the question to ask about any standby is not whether it holds the same configuration. It is whether it holds the same relationships — the sessions, the subscriptions, the connections that the working system depends on and that were negotiated while the primary was alive. Configuration replicates. Live relationships have to have been established in advance, by the thing that will survive, or they do not exist when you need them.

What I would check on Monday#

Three things, in the order they cost me time.

Look at which end initiated every link between your central servers and your remote sites. If a site opened it, your backup almost certainly cannot use it. This is a configuration change, not a purchase.

Then fail over deliberately and, while the primary is still down, count rows in the historian rather than looking at a screen. The screen is the half that works. The number is the half that tells you the truth.

Then keep counting for a minute after you bring the primary back, because a recovery that looks instant on a dashboard and takes forty seconds in the data is a different fact about your plant.

Everything else in this estate survived being switched off — brokers, consumers, the historian itself, and a simultaneous loss of a server and a broker together. Those numbers, the rig, and the configuration detail that makes the reversal work are in the lab notes. The reason I was counting rows in the first place came out of an earlier experiment, where the only failure that lost data was the one at the producer. This one was the same lesson wearing a better disguise: the producer is still where loss lives, and redundancy at the producer only counts if the connections point the right way.

This is one worked example of a much more general problem: everything a plant monitors answers whether something is up, and nothing answers when a row last landed — nothing was wrong with the configuration.