Ignition and Redpanda on the same cluster, and the module that deletes the bridge

I built this expecting to write about persistent volumes. The storage behaved, and the genuinely useful finding was sitting in a directory listing.

The rig#

Cluster	k3s v1.31.5, single node
Ignition	inductiveautomation/ignition:8.3.8, StatefulSet, 3 Gi PVC
Redpanda	redpandadata/redpanda:v24.2.18, StatefulSet, 2 Gi PVC, 1 core / 1 GB
Storage	k3s local-path provisioner

Redpanda as a StatefulSet rather than a Deployment on purpose: a broker has identity and storage, and must come back as the same node with the same data.

The finding that changes the drawing#

$ kubectl -n ignition exec gateway-0 -c ignition -- \
    ls /usr/local/bin/ignition/user-lib/modules/ | grep -iE "kafka|event"
EventStream-module.modl
Kafka Connector-module.modl

Checked against Inductive Automation's documentation rather than inferred from a filename: the Kafka Cloud Connector connects to Kafka from the gateway, and Event Streams uses Kafka as both a source and a handler — consuming from topics and publishing to them.

GETTING IGNITION DATA ONTO KAFKA

The usual drawing puts a translator in the middle: gateway publishes MQTT, something subscribes, that something writes to Kafka. Redpanda Connect is a common pick for that box, and since it ships no OPC UA or industrial protocol support it can only ever live downstream of a broker anyway.

On 8.3 the gateway is the Kafka client. If the middle box only translated, it is now a component being maintained for no reason. If it also buffered, fanned out to several destinations, or spoke protocols the gateway does not, it keeps its job.

Pod deletion — Redpanda#

Five hundred known records written, then the pod deleted outright.

	Before	After
Pod name	redpanda-0	redpanda-0
Pod IP	10.42.0.71	10.42.0.72
High-watermark, partition 2	500	500
Log start offset	0	0
Leader epoch	1	2
Container restarts	—	0 (new pod, not a restarted container)

Ready again in 7 seconds, every record intact. Run three times; a clean run with nothing else touching the cluster reproduced it exactly — 500 records, 7 s, 10.42.0.72 → 10.42.0.74, epoch 1 → 2, nothing lost.

AFTER DELETING EACH POD OUTRIGHT

pod nameredpanda-0→redpanda-0

pod IP10.42.0.71→10.42.0.72

records stored500→500

leader epoch1→2

The leader epoch incrementing is the cluster correctly recording that a new leadership term began — the broker knows it died and says so, rather than pretending nothing happened.

Pod deletion — the Ignition gateway#

Terminated in	5 s
Back to Ready	16 s
Name	gateway-0 → gateway-0
IP	10.42.0.59 → 10.42.0.73
GET /StatusPing afterwards	HTTP 200

DELETE THE POD, TIME THE RETURN

Redpanda500 of 500 records intact

7s

Ignition gatewayHTTP 200 on /StatusPing

16s

0s10s20s

Sixteen seconds against Redpanda's seven, and the difference is a JVM starting.

Footprint, both running#

Pod	CPU	Memory
gateway-0 (Ignition)	1873 m	859 Mi
redpanda-0	2 m	250 Mi
db-0 (Postgres)	1 m	22 Mi
node total	1698 m (28%)	2084 Mi (22%)

MEMORY, BOTH RUNNING TOGETHER

859 Mi

250 Mi

22 Mi

Ignition gatewayRedpandaPostgres

Worth keeping for the next time somebody objects to "adding a streaming platform to the plant". Redpanda idles at 250 MiB. The gateway already in the rack uses three and a half times that.

THE FOUR NUMBERS

7s

REDPANDA BACK, DATA INTACT

16s

IGNITION GATEWAY BACK

500/500

RECORDS THROUGH A POD DELETION

2

MODULES THAT REMOVE THE BRIDGE

Verdict#

The Kubernetes half was uneventful, which is the right result. Delete either pod and it returns with its name and its data in under twenty seconds with nobody watching. The StatefulSet contract held for both.

Two things to carry out of it.

Address by name, never by IP. Both pods came back on different addresses with unchanged names. The plant-floor habit of writing an address into a config screen breaks the first time a pod moves, and pods move for entirely routine reasons.

Check the gateway's module list before commissioning a bridge. On 8.3 the protocol-translation box in the middle of your drawing may already be redundant.

A check that hung instead of failing. My first version of the kill test counted records with rpk topic consume --num 5000 against 500 records, which blocks forever waiting for records that will never arrive. It did not return a wrong number — it returned nothing, and looked exactly like a slow cluster. Counting from the partition high-watermark is instant and cannot hang. A check that can block is worse than one that can fail: a failure announces itself, a hang looks like patience.

And the limit of this test. It measures processes returning, not a plant resuming. A gateway answering HTTP in sixteen seconds still has device connections to rebuild, and whatever was mid-transaction when the pod died is a question this does not touch. Kubernetes restarts software reliably; it has no opinion about your process.

The argument, without the measurements, is in the article.