A site lost its link for two minutes. Those readings are never coming back.

Here is a drawing almost every multi-site system arrives at. The sites collect, the centre subscribes to what they collect, and the centre writes the history. It is clean, there is one place to look, and adding a site is a small job.

Cut one site's link for two minutes and you will find out whether your history has a hole in it. Mine did, and I had assumed it would not.

What I expected, and what happened#

I had three sites feeding a central pair, each publishing a value once a second so I could count rows per ten-second bucket afterwards. I dropped the network between one site and the centre for about two and a half minutes, then restored it.

The site went to zero immediately. The other two sat at ten rows a bucket throughout, which was reassuring — the blast radius was exactly one site.

Then the link came back, the site resumed, and the two and a half minutes stayed empty.

Not delayed. Not thinned. Simply absent, and permanently so. Whatever that site's process was doing during those two minutes is not recorded anywhere and never will be.

Why there was nothing to recover#

I had been thinking of the link as a pipe that had been blocked, with the readings waiting at the other end. That is not what a live tag subscription is.

A subscription is a standing request for updates. The centre asks to be told when a value changes; the site tells it. When the link drops, the site is not holding undelivered messages, because it was never asked to produce any — it was answering a question that nobody is currently asking. On reconnect the subscription is re-established and the centre is told the current value, which is correct and also useless for the two minutes you missed.

There was a buffer in the path, which is what misled me. The publishing stage at the centre batches before it sends, and a batch queue looks like insurance. It is not: it is a smoothing buffer that holds a second or two of events and drops the oldest when full. It exists to make bursts efficient, not to survive an outage, and it sits on the wrong side of the break anyway.

That is the general trap. Almost every one of these paths has a buffer somewhere, and its presence is reassuring long before anybody checks what it is for.

The fix is a decision, not a product#

Once the shape is clear the answer is forced. If nothing durable is written at the site, an outage between the site and the centre cannot be recovered, because recovery requires that somebody kept a copy. So somebody has to keep a copy, and the only place that helps is the site itself.

That is the whole design rule: the first durable write happens where the data is produced.

I turned on the site's own historian, pointed at the same central database, and ran the identical cut. It buffered locally while the link was down and forwarded the backlog on reconnect with the original timestamps. Same outage, same tag, every bucket full.

The live path stayed a hole in exactly the same window, which is the part worth sitting with. Both were running at once, on the same readings.

Two paths, doing two jobs#

That result reads at first like the live path failed. It did not, and replacing it with the durable one would be a mistake.

They are not redundant copies of each other. One is a real-time fan-out: it feeds screens, alarms and anything that reacts, and its job is to be current. A gap in it during an outage is not even a defect, because live data from a site you cannot reach is not a thing that exists. The other is the record: it does not have to be fast, it has to be complete, and it is what you query when somebody asks what happened on Tuesday.

Systems that conflate the two end up with a record that is only as complete as the worst network day, and usually nobody notices until an investigation runs into the gap.

What to check#

Take a site, drop its link for two minutes, restore it, and count rows over the window in whatever you consider your history. That is the whole test and it takes five minutes.

If the rows are there, something at that site kept a copy and you should find out what, so you know its capacity and how long it can hold out.

If they are missing, you have a live subscription pretending to be a historian, and the decision in front of you is where the first durable write is going to happen. It is a configuration change at the edge far more often than it is a purchase.

The rig, the bucket counts, and what the store-and-forward path did during the same cut are in the lab notes. A version of the same site that has no SCADA licence on it at all, buffering the same way, is in a separate experiment. And the failure that started this whole line of testing — a redundant pair that failed over its screens but not its data — is here.

A subscription that silently stops queueing looks identical, from every dashboard you own, to one that is working. What to watch instead: nothing was wrong with the configuration.

Keeping a copy decides whether the readings survive a cut. How long the cut lasts is a different fix, made in the network rather than at the site: when did your backup link last carry the plant?