You don't have a backup problem. You have a reproducibility problem.

Here is a question worth being honest about. Pick your most remote site — the one with the pump station and the panel PC that has been running since handover. Tonight, it burns.

Could you rebuild it exactly?

Not "could you get something running by Thursday". Could you produce a gateway that is provably identical to the one you lost — same tags, same names, same scaling, same alarm limits — and could you demonstrate that it matches, rather than assert it?

Almost nobody can. The reason is not negligence, because almost everybody has a backup. The reason is that a backup and a rebuild are different things, and our industry has spent twenty years treating them as the same thing.

Restore is not rebuild#

Engineering has always understood reproducibility. A drawing package is reproducible: hand the P&ID, the loop sheets and the panel schedule to a contractor who has never seen your plant, and they build the same panel. Not a similar panel. The same one. If theirs differs from yours, one of you is wrong, and the drawings settle which.

That is what reproducibility means, and it has nothing to do with IT. The definition exists separately from the instance, and the instance can be regenerated from it.

A gateway backup is not that. It is a photocopy of one instance at one moment. It cannot tell you what the site was supposed to be, only what it happened to look like at 2am on Sunday — including the change somebody made on Friday afternoon and never wrote down, and excluding the one that went through change control properly on Monday morning.

So when you restore it you do not get a correct site. You get a plausible one, and then you spend the afternoon working out the difference between plausible and correct while the plant waits.

The right-hand column is not aspirational — I measured it, and I will come back to how. What matters first is why the left-hand column is the normal state of affairs, because it is not laziness. It is a structural consequence of where the definition lives.

Everything that annoys you is the same disease#

Once you go looking for the missing definition, the symptoms stop looking like separate problems.

Your sites no longer agree with each other.

Site 1 was commissioned by the original contractor to their house style. Site 2 went to a different integrator because procurement went to tender. Site 3 was upgraded in-house after somebody finally wrote a naming standard, which the first two predate. Every decision was reasonable on the day it was made. The result is an estate that cannot be queried without a translation layer nobody controls.

That is what happens when "what a pump station is" lives in ten places with no mechanism for disagreeing out loud. It is not a discipline problem and you cannot train it away.

A trivial change costs three days.

Almost none of that is engineering. It is transcription — one decision, typed carefully into ten places by someone senior enough to be trusted with a Designer session on a live plant. And the line that turns one day into three sits in the middle: find the two sites where someone renamed things. You are paying for the drift, in hours, every time you touch the fleet.

Handover is an act of faith.

Documentation rots because it is a parallel description of the truth maintained by goodwill, and goodwill loses to a busy quarter every time. Nothing forces it to stay true. When the definition is the thing sites are built from, it cannot drift from them — if it were wrong, the sites would be wrong, and somebody would be shouting by Wednesday.

Three complaints, one cause. None of them get fixed by better backups, more process, or a stricter naming standard, because none of them are caused by the absence of those things.

The definition is smaller than you think#

The objection I would raise, in your position, is that writing down a definition for an entire estate sounds like a project nobody will fund.

It is much smaller than it looks, and this is the number that convinced me.

Ten sites: pump stations, a reservoir, a treatment works, two district cooling plants. Between them, 364 tags. Every one real, every one has to be right, every one will page somebody at 3am if it is wrong.

The number of distinct decisions in that estate is twenty-six.

Seven points describe a pump. Six describe a chiller. Four describe an LV feeder. That is not a criticism of anyone's work — pumps genuinely are alike, which is why they are pumps. But it means the thing you would have to write down is twenty-six points, not three hundred and sixty-four objects. That is an afternoon, not a project.

The reason it currently feels like three hundred and sixty-four objects is that the tooling makes you express the similarity by repeating yourself, and repetition performed by humans across years and contractors does not stay identical. It stays approximately identical. Which is the same disease again.

So I built it, to find out what breaks#

Arguing this from a whiteboard is easy, so I didn't. I built a small water utility for real — one head-office gateway and ten site gateways, running on a cluster, every one generated from a definition.

Ignition 8.3 is what makes this possible. Gateway configuration moved out of an internal database and onto the filesystem as plain JSON: not only project resources, but device connections, tag providers, tags, OPC connections, security settings. If all of that is text, a site can have a definition again. I argued the case for this in the abstract before building anything; this is where it meets a keyboard.

The entire description of a site is this:

site: edge-04
name: Manchester Treatment Works
code: MAN
role: edge
region: UK North
equipment:
  - { type: pump, count: 6, prefix: P }
  - { type: feeder, count: 2, prefix: FDR }
  - { type: flowmeter, count: 3, prefix: FM }

Eleven of those, plus six equipment definitions, generate 630 Ignition configuration resources: every tag, every folder, each site's device with its full point list, head office's connection to all ten.

Two properties do the work. The arrow only points one way — a gateway is an output, not a place where decisions are stored, exactly as the panel was an output of the drawings. And the generator never touches a gateway. It reads files and writes files, so the same input always produces the same bytes, and that output can be committed and reviewed. A tag rename across ten sites becomes a diff a second engineer reads over coffee, rather than a thing Dave is careful about.

The test that matters#

I deleted a site's gateway and its storage — the panel PC dying with an unreadable disk, nothing local to recover.

Then I applied the definition again.

The site was fully back in 60 seconds, with all 31 of its tag resources present and its device running. Nothing was restored, because no backup was involved at any point. It was not an approximation of the site. It was the site, generated from the same definition every other station is running from.

The maintenance numbers fall out of the same property rather than being a separate benefit. Adding bearing temperature to every pump — the three-day job from earlier — took 98 seconds and landed 36 tags across nine sites, with the arithmetic exactly right per site. London gained four because London has four pumps. Head office gained none because it has no pumps. Nobody visited a site, and nobody could have missed one, because missing one is not a thing that can happen when there is a single definition.

The first bar is the one I did not expect to care about. Kill a site gateway outright and it is back on its own in 37 seconds, with nobody involved. That is the standby-server dance OT has done by hand, badly, for twenty years.

What it costs to run#

Before somebody tells you a fleet of gateways needs a server room:

Ten sites and a head office ran inside about 5 GB of memory and 11 GB of disk on one modest machine. The constraint here is not hardware, and has not been for years.

Where reproducibility stops#

Head office should aggregate the ten sites, and this is the part that did not work.

The ten connections generated perfectly: present, enabled, correctly described, pointed at the right addresses, and head office could discover each site's server on demand. Every one sat in FAULTED, because each gateway signs its own certificate the first time it starts, and head office cannot be told in advance to trust ten certificates that do not yet exist. Out of the box, clearing that is a person answering a yes/no dialog, once per site.

I am not going to claim this is unautomatable, because I have not proven it. A two-phase rollout — stand the sites up, collect their certificates, distribute them, connect — is the obvious approach and is how the problem gets solved elsewhere. What I can say is that it does not fall out of the same pass as everything else, and that anybody promising a fully hands-off multi-site rollout has probably not tried one.

On reflection that is the right place for friction to be. Everything describing the plant reproduced cleanly. What refused was the part that decides who is allowed to talk to whom. I would be more worried if handing one machine authority over ten plants had been a single line of configuration.

The test to run on Monday#

You are not going to rebuild your estate this quarter, and you shouldn't.

But there is a cheap and slightly uncomfortable experiment available. Take your most recent backup of one site, restore it onto a spare machine, and compare it with the live one. Not a spot check — a real comparison of tags, scaling and alarm limits.

Whatever you find is your actual recovery position. Most people find somewhere between three and a dozen differences they cannot account for, and that number is the honest measure of how reproducible your estate is today. It is also a far better thing to take into a management conversation than any argument about tooling.

Then pick the thing you own the most copies of — a pump, a booster set, an ETS skid — and write down what it is. Once. Not as a Word document; as the thing the next site's tags get generated from. You will find out very quickly whether your sites agree about what a pump is, and that answer is worth having even if you never generate anything from it.

The rebuild was always the goal. The backup was only ever a substitute for one.