Cluster Mode
Cluster mode runs BoilStream as three or more replicas. Catalog, identity, topic, tenant, and grant metadata is committed by a quorum, while S3-compatible object storage holds bulk data and complete disaster-recovery snapshots.
Use the Kubernetes deployment for the supported production topology. It configures stable node identities, private node-to-node authentication, public TLS, quorum metadata, and S3 recovery as one profile.
Availability and recovery model
Solid arrows show normal request and snapshot flow; the dashed arrow is disaster recovery. A replacement recovers metadata from healthy peers. Complete local-state loss uses a completed object-storage recovery generation. Unpublished hot rows have a separate durability boundary and must not be treated as replaceable cache data.
Metadata design
- A three-replica cluster continues serving quorum-backed metadata while any two replicas are healthy.
- A request may reach any healthy replica. Metadata changes are ordered consistently and become visible across the cluster.
- A lost or replaced replica rebuilds metadata from healthy peers. This does not replicate every admitted input batch or restore unpublished local hot rows.
- If every replica loses local state, the cluster restores identity, catalog, and metadata state from the latest completed S3 backup.
- Tenant and catalog authorization remains fail closed during leader changes, quorum loss, and recovery.
Catalog availability and row durability have different requirements. A healthy cluster does not imply that every recently accepted row has reached durable storage.
Check the deployment and recovery requirements for your release; see Reliability and completion.
Availability behavior
| Event | Expected behavior |
|---|---|
| One replica stops | Healthy replicas elect or retain leadership and continue serving. Clients connected to the stopped replica reconnect. |
| One replica is replaced | It rejoins and catches up from peers without a persistent volume. |
| Quorum is unavailable | Metadata-dependent operations reject rather than use stale or node-local authority. |
| All local node state is lost | Follow the recovery procedure to restore the latest completed backup. |
| Object storage is unavailable | Operations that require durable object storage or recovery publication fail or pause; S3 is not substituted with local-only state. |
Failover does not promise that an existing TCP connection survives the loss of its serving pod. Applications should reconnect using the vended endpoint or normal load-balanced hostname and retry only operations their client contract defines as retryable.
Required production settings
The Helm chart validates the supported profile:
- at least three replicas;
- both identity/catalog and DuckLake metadata on the quorum-backed store;
- node-to-node mutual TLS with a private trust root;
- strict, encrypted S3 disaster recovery with a unique cluster identity and fenced recovery incarnation;
- one stable externally managed storage-encryption key and a distinct stable token-signing key shared by every replica;
- no persistent-volume requirement.
Do not create a multi-replica deployment with local metadata. The only supported local-file profile is an explicit single-replica development installation.
Recovery identities
Choose clusterId once for a logical cluster and never reuse it for an unrelated cluster. For the first installation, use recovery epoch 1 and a unique recovery ID. After the first durable generation exists, switch the bootstrap mode to restore.
For a deliberate all-node recovery:
- Fence the previous incarnation so it cannot publish new state.
- Increment the recovery epoch exactly once.
- Assign a new unique recovery ID to all replicas.
- Restore with the original cluster ID, object-storage prefix, and encryption key.
- Verify catalog access and writes after leadership changes before reopening traffic.
The latest completed recovery backup is the recovery point. Data acknowledged after that generation but before a sudden all-node loss is not claimed to be in the restored generation unless its interface provides a stronger durability acknowledgement.
Operating the cluster
Check health and membership with:
boilstream-admin cluster statusFor rolling updates, keep enough replicas available for quorum, wait for each replacement to become ready, and only then continue to the next replica. The chart's disruption budget prevents more than one voluntary disruption at a time in the default three-replica profile.
Monitor these conditions:
- all expected replicas are healthy;
- a leader is known and quorum is available;
- recovery generations are being published to the intended S3 prefix;
- replacement replicas catch up before old replicas are removed;
- public authentication, catalog access, and ingestion probes remain green.