Event Streams to TimescaleDB: the config recipe, five silent failures, and a loss-accounted kill battery

Last month's rig proved the modules exist. This one makes them carry data, and the distance between those two statements turned out to be five silent failure modes and a set of config shapes documented nowhere outside the module jars.

The rig#

The heartbeat instrument: [System]Gateway/CurrentDateTime changes once per second, so stored history audits itself. Count rows in 10-second buckets; ten is health, anything less is loss.

The config recipe the docs don't give you#

Event streams are JSON resources at projects/<proj>/com.inductiveautomation.eventstream/event-streams/<name>/config.json. Authoring them by hand works, but the shapes below came from javap on the module jars, not from documentation.

The parts that cost a restart each to learn:

• Handler type ID is the full string com.inductiveautomation.connectors.kafka.eventstream.handler. Two plausible shorter forms both fail with No handler type registered.
• sourceEncoder, batch, filter, transform, onError are all mandatory. Each missing one is a separate NullPointerException at stream creation, surfaced one restart at a time.
• topic, key, value, partition, timestamp are ExpressionValue objects — {"data": "...", "type": "STATIC" | "EXPRESSION"}. A bare string parses without complaint and produces nothing.
• Tag identity lives in metadata, not data. The event's data payload is the qualified value; tagPath, isInitial, previousValue are metadata. So the key is {event.metadata.tagPath} and a readable payload is toStr({event.data.value}) — raw {event.data.value} coerces through a type-keyed encoder registry into binary IEEE-754 doubles.
• Batch timers set to 0 are disabled, not fast. With debounceMs and maxWaitMs at 0, the stage flushes only at maxQueueSize: 1000 events at 1 ev/s is a sixteen-minute pipe that looks completely dead.

Deploying file changes: the two-scan cycle#

8.3 removed the project scan interval, so the gateway never watches its own disk. Changes apply through Platform → Projects → Scan File System, and a changed stream needs the cycle twice: the first scan tears the running stream down, the second one starts the replacement.

Miss the second scan and the stream stays down with nothing in the log, which belongs in the next section.

Five silent failures, all reproduced#

1. Boot-order race. Event Streams starts before the Kafka Connector module registers its handler type, so a file-based stream with a Kafka handler can fault at boot with No handler type registered. Intermittent across boots.
2. No filesystem watching. External edits are invisible until a manual scan or an API-key scan call. kubectl cp alone changes nothing.
3. Shutdown-without-restart. The first scan after a change logs Shutdown event stream and stops; no start attempt follows until another scan with a diff.
4. failureStrategy: IGNORE hides delivery errors completely. The handler reports GOOD with error count 0 while producing nothing.
5. A malformed handler entry unregisters the whole stream. No fault, no log line; the stream simply no longer exists. This is the one that turns config typos into invisible outages.

The theme is the same one every plant integration eventually teaches: these pipelines do not crash, they lie. A green stage status is not evidence; a number that changed is.

The running pipeline, measured#

End-to-end latency, event timestamp to database row, over a fresh two-minute window of 235 rows:

The tail is the historian stream's deliberate 5-second debounce, not jitter. Steady-state consumer lag: 3–4 messages across 9 partitions.

Kill battery, loss-accounted#

Broker kill. kubectl delete pod redpanda-1 mid-flow. Leadership moved sub-second, the pod rejoined in ~45 s, heartbeat read 10/10 in every bucket through the whole event. Zero loss, and zero symptoms downstream.

Historian outage. docker stop tsdb for 60 s. The sink logged connection refusals once per second, its bookmark froze, the log kept absorbing new records. On restart the backlog cleared in seconds with event-time stamps. Every bucket in and around the outage window: 10/10. Zero loss.

Gateway restart. kubectl delete pod gateway-0. Buckets read 23/30, 6/30, then full again: about 45 seconds of plant behaviour never measured. Nothing downstream can repair this class of loss; a reading that was never taken cannot be replayed. On this boot both streams auto-started cleanly; combined with failure mode 1 above, treat producer recovery as the untrusted step and verify it, not assume it.

Replay#

A second consumer group (replay-demo, start_from_oldest: true) pointed at a fresh empty table rebuilt the complete history in under a minute: 9,950 rows in, 9,950 rows out, identical first and last timestamps, no request ever made to the gateway. The original consumer's bookmark never moved.

One hand-produced record through the console (Produce Record, key [Lab]Manual/TestTag, value 42.5) landed as a historian row 250 ms later with its Kafka offset in the seq column — the sink cannot tell a human from a SCADA system, which is exactly the property that makes the log the plant's source of truth rather than any single database.

What transfers#

• Author event streams as files and treat the two-scan cycle as your deploy step; it is Git-reviewable SCADA config once you know the shapes above.
• Never ship failureStrategy: IGNORE without a downstream count check; prefer a heartbeat tag whose bucket counts audit the whole pipe for free.
• Spend redundancy at the producer. Everything after the first hop already forgives you.

The practitioner argument built on these numbers is in the companion post.