Three sites into a redundant pair: a kill battery, and the one connection that pointed the wrong way

A redundant pair, three sites feeding it, and a historian with a hot standby. The drawing is unremarkable — most estates look like this. The question was which of its failures actually cost plant data, and the honest answer took two runs of the same test, because the first run measured a rig I had wired backwards.

The rig#

The instrument is one tag per site. Each site's Ramp0 advances once a second, so ten rows per ten-second bucket is health and anything less is loss, per site, with nowhere for a gap to hide. Every number below is that count.

The finding: a connection's direction decides whether data survives#

The headline test is killing the master while the estate is running. The first time I ran it, the control plane behaved beautifully and the data plane died. central-b went active in about five seconds, the VIP re-pointed, the HMI stayed reachable on the same URL — and all three sites went to zero rows and stayed there for six and a half minutes, until I restarted the master by hand.

The backup was reporting RedundantState=OutOfDate, so the obvious suspect was sync drift. I re-synced it, confirmed Good, and killed the master again. Same result: control plane in eight seconds, data plane never.

That ruled out drift and pointed at the topology. Checking the vendor documentation rather than trusting the rig: an outgoing Gateway Network connection does not redirect to the backup on failover, and connections cannot be approved on the backup node at all. The supported arrangement is that the redundant master pair initiates the connections, because configuration made on the master replicates to the backup.

I had built it the other way. Each edge opened a connection to the central master, which is the intuitive direction — the data flows that way, so the connection feels like it should too. Nothing about that arrangement warns you. It runs perfectly until the master dies, at which point the backup has no link to any site.

Reversing it means three outgoing connections created on the master, which then appear on every edge twice, once for the master and once for the backup. After the reversal the same kill cost nothing: every site held ten of ten straight through, including the bucket containing the kill.

Two operational notes from doing the reversal. The receiving side is now the edge, so Require Two Way Auth has to come off there or the incoming handshake fails with certificate_unknown. And the old edge-initiated connections must be deleted rather than left in place, because Ignition permits one connection per gateway pair and the leftovers are rejected with Duplicate connection — the resource lives at data/config/resources/core/ignition/gateway-network-outgoing/<ip>_<port>/, and the port in that directory name is not always the one you expect.

The kill battery#

Seven failures, each accounted against the per-site heartbeat.

The six zero-loss results are worth stating plainly because they are the boring, correct outcome of paying for replication. Deleting a broker moved leadership in under a second. Deleting the consumer cost nothing because the replacement resumed from its committed offset and read the backlog out of the log. A sixty-second historian outage was invisible in the final data: the sink logged connection refusals once a second, the log kept accepting writes, and the backlog drained on restore with event-time stamps intact. Killing a gateway and a broker simultaneously — the closest thing here to losing a physical host — also cost nothing, because each layer fails over independently and none of them waits for the others.

The WAN cut is a hole, not a backlog#

The one test that loses data even when everything is configured correctly is cutting a single site's link. Rotterdam went to zero for the duration and stayed at zero after reconnection. The other two sites read ten of ten throughout, so the isolation is exact, but the missing readings never arrive.

This surprised me, and it should not have. A Remote Tag Provider is a live subscription, and a subscription has no memory. During the cut no change events fire, so nothing is queued anywhere; on reconnect the subscription simply resumes at the current value. Event Streams has a batch queue in front of the handler, but that is an in-memory smoothing buffer with a drop policy, not store-and-forward.

The fix is not a bigger buffer. It is moving the first durable write to the site.

Enabling the tag historian on the edge gateway, pointed at the central database, gives you store-and-forward for free: during the same cut it buffered locally, and on reconnect it forwarded the whole backlog with original timestamps. Same outage, same tag, ten of ten in every bucket. The live path stayed a hole in the same window, which is the useful shape of the answer — the two paths are not redundant with each other, they are doing different jobs. One is a real-time fan-out that may miss a window. The other is the audit record that must not.

The practitioner version of this one result — why a subscription has nothing to replay, and how to tell which of your own links is a record rather than a convenience — is in its own post.

Making the HMI survive the same kill#

Ignition redundancy keeps the tag engine and the historian alive across a failover. It does not keep an HMI endpoint reachable, because the session was being served by the gateway that just died. That is a separate availability problem and it wants a separate answer, which is why the pair sits behind an HAProxy VIP.

The health check is the whole trick, and the obvious probe is useless: both nodes answer /StatusPing with RUNNING regardless of which one is active. The discriminator is /system/gwinfo, where the master reports RedundantNodeActiveStatus=Active and the backup reports Cold.

option httpchk
http-check send meth GET uri /system/gwinfo ver HTTP/1.1 hdr Host ignition-estate
http-check expect string RedundantNodeActiveStatus=Active
default-server inter 2s fall 2 rise 1

Two details cost a while each. Without ver HTTP/1.1 and a Host header the check fails outright with "HTTP content check did not match", because the gateway's Jetty wants a proper HTTP/1.1 request. And timeout tunnel has to be long, or Perspective's websocket upgrade is cut from under the session. With those in place the VIP followed the failover in about five seconds and the operator's URL never changed.

Configuration notes that are not in the manual#

Authoring gateway configuration as files works, with edges.

• A file scan registers new project resources, not new gateway ones. Dropping both an Event Stream and a Kafka Service Connector on disk and scanning registered the stream and silently ignored the connector, which then had to be created through the interface. The stream ran green at 17 events per second while delivering nothing, because the handler referenced a connector that did not exist and the failure strategy was IGNORE.
• A scan run while the licence has lapsed does nothing at all, with no error. Modules pause when the trial expires, so the scan cannot instantiate anything. Two of my scans failed for this reason before I noticed the timer.
• Perspective views load at project start, not on a config scan. A new view dropped on disk stays invisible until the project reloads.
• Perspective has no HTTP binding. Bindings are tag, property, expression and query. Pulling non-Ignition metrics — broker health, proxy state, replication lag — into a native view needs a gateway script writing to tags, so the infrastructure dashboard here is a separate page rather than a contortion.

The run where the test proved nothing#

One result in this lab was invalid and worth writing down, because it looked perfect.

Testing store-and-forward on a container running with bridge networking, I cut the link with iptables -I OUTPUT -d <central> -j DROP and measured a flawless backfill across the outage. It was flawless because there was no outage. A bridge container's traffic traverses the FORWARD and DOCKER chains and is translated on the way out, so an OUTPUT rule never touches it. The earlier gateway cuts were valid only because those containers run with host networking, where locally generated traffic does go through OUTPUT.

The correct rule for a bridge container is iptables -I DOCKER-USER, and the general version of this mistake is a post of its own. The useful part is not the rule, it is the pair of signals that would have caught it immediately: the client never logged a disconnect, and the local queue never grew. A fault-injection test has to prove the fault was injected. A healthy output is not evidence of resilience if the thing you switched off was still on.

What transfers#

• Open Gateway Network connections from the redundant pair toward the sites, never from the sites inward. Redundancy replicates configuration made on the master, and that is the only reason the backup has anywhere to send data.
• Treat a live tag mount as a real-time convenience, not a record. If a window of readings matters, the first durable write belongs at the site.
• Health-check the property you actually care about. RUNNING is not active, and a green tile is not a moving number.
• When a fault-injection test reports zero loss, verify the fault happened before you believe the zero.

The practitioner argument built on the failover number is in the companion post. The producer-side thesis this estate was built to test — that everything downstream of the first durable write is forgiving — came out of the earlier single-gateway rig.

Every site in this estate ran a licensed gateway, which is what made the architecture simple and the licensing expensive. The same link cut, run against a site with no SCADA product on it at all, is in the zero-licence edge lab.

This estate ran flat — one subnet, every host reachable from every other. What its traffic looks like measured over a full day, and the zone-and-conduit policy that falls out of it, is in the segmentation lab.