Timing a recovery to the first good row

The companion experiment to what a gateway backup actually restores, which measured how fast a gateway comes back. This one asks the question that matters more: how long until the plant is recording again, and what does the outage cost.

The rig#

Drill 1 — everything down#

The first drill stopped both the gateway and the historian, then recovered both.

The historian was never the bottleneck: PostgreSQL was accepting connections 1.4 seconds after docker start. Effectively all the recovery time is the gateway, and roughly a third of it happens after the gateway says it is up.

Drill 2 — clean baseline, ungraceful kill#

Drill 1 could not measure the gap in the record, because the baseline period was contaminated: the estate's original gateway had a lapsed historian licence and had been writing nothing for hours, so the "gap" spanned four hours of nothing happening rather than the outage.

Drill 2 fixed that by establishing a continuously-writing baseline first and verifying it before touching anything:

baseline write rate: 63 rows / 20 s      (~3.15 rows/s)
last row before outage: 2026-08-30 12:07:12+00

Then docker kill — no graceful shutdown — and a replacement built from the same backup:

Both drills agree to within a second on both figures, which is the useful part: a gateway that reports RUNNING is not yet a gateway that is recording, and the distance between those two states is consistently about nine seconds.

Anyone timing recovery by watching the process appear is understating it by 36%.

The hole#

With a clean baseline the gap is attributable:

with r as (
  select t_stamp, lag(t_stamp) over (order by t_stamp) prev
  from (select distinct t_stamp from sqlt_data_1_2026_08 order by t_stamp) s
)
select to_timestamp(prev/1000) gap_start,
       to_timestamp(t_stamp/1000) gap_end,
       round((t_stamp-prev)/1000.0,1) gap_seconds
from r where t_stamp - prev > 5000 order by t_stamp desc limit 1;

     gap_start      |      gap_end       | gap_seconds
2026-08-30 12:07:12 | 2026-08-30 12:07:39 |        26.8

26.8 seconds, permanently absent. At 3.15 rows/s that is approximately 84 readings that were measured and no longer exist.

Note the arithmetic: the hole (26.8 s) is shorter than time-to-first-row (32.9 s), because the replacement's first write carries a timestamp from the moment it started polling, not from when the clock started. The hole is bounded below by how long nothing was polling, and no recovery speed drives it to zero.

Why store-and-forward does not help here. The buffer is a store inside the gateway process and its local cache. docker kill destroys both. The replacement is constructed from a .gwbk, which — as measured previously — contains 2,711 configuration entries and zero rows of history. So the replacement has no knowledge that an outage occurred and nothing to backfill from.

Store-and-forward protects against the historian being unavailable, which is a different failure and one it handles well. It offers nothing against the gateway itself dying.

What this drill deliberately did not test#

Stated plainly, because the 24-second figure is misleading without it:

Every item in the right column is measured in minutes or hours and none of them appear in a recovery tool's report.

Verdict#

Time-to-restore is a measurement of the software. Time-to-first-good-row is a measurement of the plant, it is 36% larger, and it is still not the damage. The damage is a hole whose size equals the outage, cannot be backfilled, and will not appear in any report generated by the recovery itself.

Measure the finish line at the historian, not at the process.

The argument for a general audience is in back online is not the same as recording.