Ten remote SCADA sites from one repository

Commissioning a remote site is the least glamorous job in this industry. Someone drives out, or someone remotes in, and works through a list: install the gateway, name it, set the admin password, add the device connection, build the tag tree, point it at head office. Then does it again at the next site, from memory, slightly differently. Three years later nobody can say why the pump tags at one station are named P1_Run and at another Pump1_Running, and the report that joins them has a hand-written mapping table in the middle of it.

The interesting thing about Ignition 8.3 is that this is now, in principle, a solved problem. Gateway configuration moved out of an internal database and onto the filesystem as JSON — not just project resources, but device connections, tag providers, tags, OPC connections, security settings. If all of that is text, a fleet of sites is a code generation problem, and code generation problems have known answers.

I wanted to find out whether that holds up when you actually build it. So I built a small water utility: one head-office gateway and ten site gateways, every one of them defined by a short YAML file, every one running for real on a Kubernetes cluster in my home lab. Then I broke things and timed the repairs.

This is written for people who know what a tag group is and do not particularly care about Kubernetes. Where the IT machinery matters I will say it in plant terms. The cluster is the same one I stood up for an earlier experiment with a single gateway; this time it holds eleven.

If you want the argument rather than the build log — why a site should be reproducible instead of merely restorable, and what that changes about backups and maintenance — that is a separate piece. What follows is what actually happened on the bench.

The interface is a site file#

The whole thing an engineer touches looks like this:

site: edge-04
name: Manchester Treatment Works
code: MAN
role: edge
region: UK North
equipment:
  - { type: pump, count: 6, prefix: P }
  - { type: feeder, count: 2, prefix: FDR }
  - { type: flowmeter, count: 3, prefix: FM }

Six pumps at seven points each, two feeders at four, three flow meters at two: fifty-six tags, a simulated PLC producing all fifty-six, and a tag group scanning them. What a pump is lives once, in a template:

type: pump
description: Centrifugal pump with VFD
points:
  - name: Running
    dataType: Boolean
    sim: random(0.0, 1.0, true)
  - name: SpeedHz
    dataType: Double
    sim: sine(20.0, 50.0, 240.0, true)
    engUnit: Hz
  - name: DischargePressureBar
    dataType: Double
    sim: sine(3.0, 9.0, 330.0, true)
    engUnit: bar

One Python script turns eleven site files and six equipment templates into 630 Ignition configuration resources and a complete set of Kubernetes manifests. It never talks to a gateway. Rendering is pure, so the generated tree gets committed and a change arrives as a reviewable diff — which is the actual point. A tag change across ten sites should be something a second engineer can read before it reaches a plant.

Each site ends up with a genuinely different tag set, because each site has genuinely different equipment:

Head office runs no field equipment. Its configuration is derived entirely from the others: one OPC connection per site, plus a headline tag set pulled from each. That rule is code, not configuration — the first unit of each equipment type contributes its most operationally interesting point, so a site with six pumps does not send head office thirty-six pump tags.

Eleven gateways in 134 seconds#

From an empty namespace to all eleven gateways answering {"state":"RUNNING"} took 134 seconds. Not eleven commissioning wizards, not eleven admin passwords typed into eleven browsers.

The tags are not merely files on disk. Each gateway's tag provider comes up running, with a tag count that matches its own equipment and nobody else's:

What it cost to run#

The honest resource numbers matter more than the demo, because "can you run ten gateways" is usually a budget question.

An Ignition 8.3 gateway carrying a few dozen tags, given a 512 MB heap, uses about 390 MiB of RAM and 240 MB of persistent state — of which 235 MB is jar cache. Eleven of them, head office included, sit in roughly 5 GB of RAM and 11 GB of disk on a single node.

I did not arrive at those numbers gracefully. My first attempt gave each gateway a 640 MB heap, which the JVM duly filled to about 800 MiB, and eleven of those on a 15 GB host pushed it into swap hard enough that I lost SSH to the Proxmox box entirely for several minutes. Two lessons, both boring and both real: size the heap deliberately rather than letting it default upward, and stagger a fleet rollout instead of starting every StatefulSet at once.

The stranger cost was disk. Early on, eleven gateways consumed 88 GB of container storage and triggered a cascade of evictions that killed the whole fleet. The gateways were not the problem. This k3s had been installed with --snapshotter=native, a workaround from an earlier lab where overlayfs was not available inside the container. The native snapshotter makes a complete copy of the image filesystem for every container, so eleven gateways with an init container each meant twenty-two full copies of a 4 GB image. Switching to overlayfs reclaimed 103 GB and dropped the fleet's total footprint to 11 GB.

It is worth naming because of how it presented. The symptom looked exactly like "Ignition is too heavy to run ten of." It was one flag.

Five things that cost an afternoon#

None of these are in the documentation, and each one silently produces a gateway that looks fine and is not.

Kubernetes hides the image's seed data. Ignition's container image ships a populated data directory. Mount a fresh volume over it and the seed files disappear, and the gateway crash-loops looking for gateway.xml_clean. Docker's named volumes copy image content in on first use; Kubernetes does not. An init container has to do it by hand.

A pre-populated core collection stops the gateway booting. Drop generated config into an empty volume and Ignition faults on startup:

com.inductiveautomation.ignition.gateway.config.ConfigurationSetupException:
  Unable to create 'core' resource collection
Caused by: java.nio.file.FileAlreadyExistsException:
  Resource collection path '.../resources/core' exists but is not empty

It wants to create that collection itself and refuses to adopt one. The fix is to ship the collection markers — an empty .resources/.meta and three small config-mode.json files — so the collections already exist as far as the gateway is concerned, and it takes what it finds.

Generated config belongs in external, not core. This is the one that cost the most. Ignition rebuilds core from its own authoritative store during startup, which quietly deletes anything you put there that it did not put there itself. My tags vanished on every boot, with nothing in the log. Ignition ships a collection called external, documented as externally managed configuration, and that is exactly what a repository is. Moving the generated tree there fixed tags, devices and OPC server settings in one change.

The split is not total. Resources Ignition generates itself — security-properties, the first-run wizard state — have to stay in core, because a core copy shadows the external one. Roughly: config the gateway owns goes in core, config you own goes in external.

A resource without a UUID is written, listed, and never instantiated. My default tag provider appeared in the UI, enabled, described correctly, with status UNKNOWN and no tag count. No error anywhere. Several resource types are keyed by the uuid in their resource.json rather than by folder name; without one, the tag system never builds the provider, and every tag written into it is dropped on the next startup. Adding a deterministic UUID derived from the resource name fixed it instantly.

This one deserves emphasis because a gateway built from generated config never runs the first-launch wizard, and the wizard is what would normally create the default provider. If you generate configuration, you must generate the provider too.

Kubernetes reads manifests as YAML 1.1. A bare Y is a boolean there, so ACCEPT_IGNITION_EULA: Y reaches the API server as true and the StatefulSet is rejected outright. The renderer now quotes anything ambiguous.

Breaking it on purpose#

Three drills, each timed against the live fleet.

A site gateway is killed. Bristol's pod deleted outright. Down within a second, back to RUNNING 37 seconds later, with its tags and device intact because they live on a persistent volume.

A site is destroyed entirely. Hamburg's gateway and its volume deleted — the equivalent of the panel PC dying with no backup. Reapplied from the repository, it was fully back in 60 seconds, with all 31 of its tag resources restored. Nothing was recovered from a backup, because there was no backup; the site was rebuilt from its definition. That is the part I actually care about. A site is no longer a snowflake you restore, it is an output you regenerate.

A change is made to every pump in the fleet. I added a bearing temperature point to the pump template — four lines — re-rendered, and rolled it out. Thirty-six new tags appeared across nine sites in 98 seconds. The arithmetic came out exactly right per site: London gained four, Manchester six, Singapore six, and head office gained nothing because head office has no pumps.

site        before   after
edge-01        41      45     4 pumps
edge-04        65      71     6 pumps
edge-10        91      97     6 pumps
hq             51      51     no pumps

That is the maintenance story in one line: the shared definition changed once, and every site that owns that equipment agreed with it. Nobody visited a site.

Where it stops#

Head office should aggregate the sites, and this is where the lab hit a wall worth reporting honestly.

The ten OPC UA connections generate cleanly, one per site, correctly described and pointed at the right cluster address. The edge gateways expose their endpoints properly. Discovery works — head office finds each site's server on demand. And every connection sits in FAULTED.

Walking the connection wizard by hand shows why. Step four is not a setting:

Each gateway generates its own self-signed certificate on first boot. Head office cannot trust ten certificates that do not exist until the sites are running, and the keystore protecting them is encrypted with a key belonging to that specific gateway, so the trust relationship cannot be rendered from a template the way everything else can.

The same shape shows up in Ignition's API tokens. A token resource stores a tokenHash, and I could not reproduce that hash offline: SHA-256, truncated SHA-512, name-salted, UUID-salted, timestamp-salted, UTF-16, HMAC, double SHA-256 and BLAKE2b all failed to authenticate against a live gateway, with the probe resources confirmed present on disk and listed as enabled in the UI. It appears to be salted with something belonging to the gateway. Which means the one credential CI would use to tell a gateway "re-read your configuration" is the one artifact in the entire configuration that cannot be generated.

I want to be careful about the claim here. I have not proven certificate trust cannot be automated — a two-phase deployment that stands the sites up, harvests their certificates and redistributes them is an obvious approach, and it is how this problem is solved elsewhere. What I have shown is that it is not expressible in the same declarative pass as everything else, and that the out-of-the-box path puts a human in front of a yes/no dialog once per site.

The honest verdict#

The configuration genuinely generates. That is not a small finding. Six hundred and thirty resources across eleven gateways, from eleven short YAML files and six equipment templates, with every site's tag tree matching its own equipment and a fleet-wide change landing in ninety-eight seconds. A destroyed site rebuilt from its definition in sixty. For anyone running more than about three sites, that arithmetic is compelling, and Ignition 8.3 is the first mainstream SCADA platform where it is available without fighting the product.

What is not solved is bootstrapping trust. Certificates and tokens are the part every infrastructure-as-code write-up skips, and in industrial systems they are precisely the part that matters, because the whole reason head office talks to a remote site is that somebody decided it should be allowed to. It is faintly reassuring that the thing which resisted automation was the security boundary rather than the tag tree.

If you take one practical thing from this: generate your tag provider, put your generated configuration in external, and give every resource a UUID. Those three cost me most of a day between them, and all three fail silently.