What drift detection actually catches: three tiers on a small estate

The question this rig was built to answer is narrow: when somebody changes something by hand, what notices?

Not whether infrastructure-as-code works — the earlier fleet experiment covered provisioning. This is about what happens afterwards, on an estate that already exists.

The rig#

The whole fleet is a map in one variable. Adding a site is four lines and no other file changes:

sites = {
  rotterdam = { vmid = 210, address = "…/24", role = "edge",   memory = 1024, scan_ms = 1000 }
  houston   = { vmid = 211, address = "…/24", role = "edge",   memory = 1024, scan_ms = 1000 }
  central   = { vmid = 212, address = "…/24", role = "centre", memory = 1536, scan_ms = 500  }
}

Provisioning: 35 s from nothing to three running gateways. plan against the empty estate: 230 ms.

Plan as a change request#

Worth stating separately because it is the part that maps onto how an industrial change process already works. tofu plan produces the complete list of what will change before anything changes, and it is the same artefact whether the change is one setting or forty sites.

That is a change-board document generated by the system that will execute it, which is a genuinely better position than a written intention and a person at a keyboard.

Tier 1 — declared as infrastructure#

Two changes made by hand through the hypervisor, the way somebody would at 3 a.m.:

• central memory raised 1536 → 2048 MB
• rotterdam tagged donotreboot

~ memory { dedicated = 2048 -> 1536 }
~ tags = ["donotreboot","edge",…] -> ["edge","gateway","ot"]
Plan: 0 to add, 2 to change, 0 to destroy.

Detected in 303 ms, with old and new values.

Tier 2 — declared as a file#

Then the change that matters operationally: houston's scan period edited from 1000 ms to 250 ms — four times the load on the device and the network.

$ tofu plan
No changes. Your infrastructure matches the configuration.

scan_period appears zero times in the plan output. The provisioning layer manages containers; the setting lives inside one.

Putting the config under git and running an agent on each gateway moves the line:

$ drift-check houston
DRIFT     houston
            < scan_period_ms  = 1000
            > scan_period_ms  = 250

652 ms, with the exact differing line.

Same estate, same instant, same change. One layer reports a clean bill of health and the other prints the divergence, and the only difference between them is which one had been told to care.

Tier 3 — declared nowhere#

The interesting one. A new file placed beside the managed config:

/etc/ot-gateway/gateway.conf          managed, in git
/etc/ot-gateway/local-overrides.conf  nobody declared this

Both layers reported clean. Neither is wrong: a file that is not on the list is not a difference from the list.

There is no configuration that fixes this, because it is not a gap in the tools. Any comparison-based check answers "does the declared set match?" and cannot answer "is there anything here I was never told about?" without a separate inventory of what should exist — which is the same discipline problem in a different costume.

The reconciliation run#

Enabling automatic reconciliation and applying:

It reverted both tier-1 changes and left the tier-2 change in place.

Read that ordering carefully. The two things it undid — a memory increase and a label — were harmless to leave. The one it preserved was the change actually altering how the gateway read the plant. Reconciliation restores what is declared, which is not a synonym for what matters.

The donotreboot tag was mine, added deliberately to make the point rather than found in the wild. It stands in for any out-of-band signal a human leaves for other humans, and the system deleted it without comment because it was not in the file.

An aside that is also the argument#

The containers came up unable to resolve public names, inheriting a split-DNS resolver from the host. I fixed it by adding four lines to the repository and applying, rather than by editing three boxes:

dns { servers = ["1.1.1.1", "9.9.9.9"] }

Which is the workflow working exactly as advertised — the fix is in the description, so it is now true of every future gateway as well, and there is a commit explaining why.

What transfers#

• Ask what is on the list, not whether you have the tool. Settings that change plant behaviour usually are not, and the report will look clean regardless.
• A clean result means "the declared set matches" — a narrower statement than it appears, and worth reading as such.
• Tier 3 is a habit, not a product. Periodically compare the real thing against an inventory of what should exist, because no comparison against a list can find what the list omits.
• Keep the detection, disable the reconciliation. In an office, automatic repair is the feature. In a plant it reverts human decisions made with context the tool does not have.
• Plan output is a change-request document produced by the thing that will perform the change, which is a better artefact than a written intention.

Measured afterwards: how far tier 2 actually stretches#

The gateways above are plain containers with a text config, not a SCADA product, so the first version of this note guessed that a real file-based product would push tier 2 much further out. That guess is now unnecessary — I measured it on a running Ignition 8.3 gateway, and it turned out I had been too pessimistic in one direction and too generous in another.

What I expected to find. A file tree for projects, and gateway-level settings — users, device connections, audit — still locked in the internal database, with the boundary running through the middle of the product.

The database is still there: data/db/config.idb, 1.35 MB, 146 tables, including exactly the names you would worry about — AUTHPROFILES, ALARMNOTIFICATIONPROFILES, AUDITPROFILES, device settings. On the table list alone the pessimistic reading looked correct.

What the rows said. Of those 146 tables, 5 contain any rows at all, and every one is bookkeeping — sequences, feature flags, and a table named MIGRATED_TABLES_8_3 whose 146 rows are a log of tables migrated out of the database, timestamped, one per table:

TABLENAME                        MIGRATIONTIMESTAMP   MIGRATIONDURATION
WSCONNECTIONSETTINGS             781307578529941      518960
AUTHPROFILES                     781307644781734      25540207
INTERNALUSERTABLE                781307644781734      25540207

The binary database is a husk. The configuration left.

Where it went. Into data/config/resources, as JSON — 619 resource files among 1,459 config files, covering audit profiles, identity providers, database connections, alarm journals, gateway network settings and device connections. A device's scan rate, the exact thing that was invisible in the container lab above, is now four readable lines:

{
  "profile":  { "type": "ProgrammableSimulatorDevice" },
  "settings": { "legacyMode": true, "repeat": true, "timeIntervalRate": 1000 }
}

The same test, on the real thing. 1,451 files committed as a baseline, then the scan rate changed from 1000 ms to 250 ms inside the running gateway. Exported and diffed:

.../opcua/device/Sample_Device/config.json | 2 +-
-    "timeIntervalRate": 1000
+    "timeIntervalRate": 250

1,720 ms end-to-end, one file and one line located out of 1,451.

Is the tree live, or a one-time export? A fair objection: files the product only reads at startup would make git a snapshot of history rather than a mirror of the plant. So I checked what the gateway writes on its own.

It writes plenty — 84 config files modified since the container started. My first reading of that was pessimistic: 83 of them are the gateway's own material, mostly system tag definitions for redundancy and store-and-forward metrics, plus two certificates auto-trusted when peers connected. One in eighty-four being an operator change is a terrible signal-to-noise ratio for a drift check.

That reading was wrong, and the timestamps say why:

seconds after boot   files written
   8                     1
  11                     2
  15                     3
  16                     5
  20                    72
  29,407                 1   ← the scan-rate edit, eight hours later

Eighty-three of the eighty-four writes land in the first twenty seconds. After that the tree is silent. A baseline committed at 16:46 and re-exported at 00:19 — seven and a half hours later, with no operator change in between — came back byte-identical: zero differing files.

Before believing that zero I proved the instrument could see anything at all, because a comparison pipeline that silently failed to refresh would produce exactly the same answer. Writing a single marker file into the gateway and re-running the export made it appear immediately as an untracked file; removing it returned the tree to clean. Only then is the zero worth quoting.

So the config directory is a live mirror rather than an export, and it is quiet enough to diff — with one operational caveat worth writing on the runbook: do not snapshot a gateway during the first half-minute after a restart, or you will capture a settling burst and call it drift.

So tier 2 in a product that has done this work is far larger than this lab's containers suggested — it reaches gateway configuration, not just project resources. Tier 3 is untouched by any of it. A device connection added through the web interface and never committed is still a thing nobody declared, and nothing compares against a list it is not on.

The practitioner argument built on these numbers is in the companion post.

The same estate's backup and restore path is measured in what a gateway backup actually restores — including what that stray migration log does on the way back.