Skip to content
BoilStream

Operate a cluster

operations
TL;DR

A cluster coordinates catalog and access metadata across replicas and uses object storage for published data and completed recovery backups. Three replicas need a healthy majority for metadata-dependent operations. Clients reconnect after a serving node fails. Metadata recovery and recent-row durability are separate; unpublished rows are not disposable historical cache.

ArchitectureReplicas coordinate metadata and share published object storage
Replicas coordinate metadata and share published object storageClients connect to a server replica. The replicas participate in metadata consensus, while published files live in shared object storage. Recovery also depends on local state, completed backups and original encryption keys.Cluster: separate server processesSQL clientSupported entry pointReplica ALocal state and serverReplica BLocal state and serverReplica CLocal state and serverObject storagePublished files andbackupsConnectMetadata consensusConsensusConsensusPublish / readReplicas coordinate metadata and share published object storageClients connect to a server replica. The replicas participate in metadata consensus, while published files live in shared object storage. Recovery also depends on local state, completed backups and original encryption keys.Cluster: separate server processesSQL clientSupported entry pointReplica ALocal state andserverReplica BLocal state andserverReplica CLocal state andserverObject storagePublished files andbackupsConnectMetadata consensusConsensusConsensusPublish / read
Each replica uses shared storage; one storage path is shown. Metadata quorum alone does not prove that every acknowledged recent row is recoverable.

Cluster mode runs BoilStream as three or more replicas. Catalog, identity, topic, tenant, and grant metadata is committed by a quorum, while S3-compatible object storage holds bulk data and complete disaster-recovery snapshots.

Use the Kubernetes deployment for the supported production topology. It configures stable node identities, private node-to-node authentication, public TLS, quorum metadata, and S3 recovery as one profile.

Availability and recovery model

Section titled “Availability and recovery model”

Metadata design

Section titled “Metadata design”
  • A three-replica cluster continues serving quorum-backed metadata while any two replicas are healthy.
  • A request may reach any healthy replica. Metadata changes are ordered consistently and become visible across the cluster.
  • A lost or replaced replica rebuilds metadata from healthy peers. This does not replicate every admitted input batch or restore unpublished local hot rows.
  • If every replica loses local state, the cluster restores identity, catalog, and metadata state from the latest completed S3 backup.
  • Tenant and catalog authorization remains fail closed during leader changes, quorum loss, and recovery.

Catalog availability and row durability have different requirements. A healthy cluster does not imply that every recently accepted row has reached durable storage.

Check the deployment and recovery requirements for your release; see Reliability and completion.

Availability behavior

Section titled “Availability behavior”
Event Expected behavior
One replica stops Healthy replicas elect or retain leadership and continue serving. Clients connected to the stopped replica reconnect.
One replica is replaced It rejoins and catches up from peers without a persistent volume.
Quorum is unavailable Metadata-dependent operations reject rather than use stale or node-local authority.
All local node state is lost Follow the recovery procedure to restore the latest completed backup.
Object storage is unavailable Operations that require durable object storage or recovery publication fail or pause; S3 is not substituted with local-only state.

Failover does not promise that an existing TCP connection survives the loss of its serving pod. Applications should reconnect using the returned endpoint or normal load-balanced hostname and retry only operations their client contract defines as retryable.

Required production settings

Section titled “Required production settings”

The Helm chart validates the supported profile:

  • at least three replicas;
  • both identity/catalog and DuckLake metadata on the quorum-backed store;
  • node-to-node mutual TLS with a private trust root;
  • strict, encrypted S3 disaster recovery with a unique cluster identity and fenced recovery incarnation;
  • one stable externally managed storage-encryption key and a distinct stable token-signing key shared by every replica;
  • no persistent-volume requirement.

Do not create a multi-replica deployment with local metadata. The only supported local-file profile is an explicit single-replica development installation.

Recovery

Section titled “Recovery”
Decision flowChoose recovery by what trustworthy state remains
Choose recovery by what trustworthy state remainsDetermine whether a healthy peer remains. With a healthy peer, repair or replace the failed node through the documented recovery path. If all local state is lost, fence old nodes and restore from a completed backup with the original encryption keys, then verify before reopening traffic.Recovery incidentStop unsafe writesHealthy peer?Repair failed nodeUse documented peer pathRestore backupFence nodes; retain keysInspect clusterYesNo local stateChoose recovery by what trustworthy state remainsDetermine whether a healthy peer remains. With a healthy peer, repair or replace the failed node through the documented recovery path. If all local state is lost, fence old nodes and restore from a completed backup with the original encryption keys, then verify before reopening traffic.Recovery incidentStop unsafe writesHealthy peer?Repair failed nodeUse documented peerpathRestore backupFence nodes; retainkeysInspect clusterYesNo local state
After either path, verify data and application behavior before resuming traffic. A backup cannot restore data that never reached it.

Use the recovery procedure after complete local-state loss. Restore the latest completed backup with the original identity and keys.

Operating the cluster

Section titled “Operating the cluster”

Check health and membership with:

Shell
boilstream-admin cluster status

For rolling updates, keep enough replicas available for quorum, wait for each replacement to become ready, and only then continue to the next replica. The chart’s disruption budget prevents more than one voluntary disruption at a time in the default three-replica profile.

Monitor these conditions:

  • all expected replicas are healthy;
  • a leader is known and quorum is available;
  • recovery generations are being published to the intended S3 prefix;
  • replacement replicas catch up before old replicas are removed;
  • public authentication, catalog access, and ingestion probes remain green.
Section titled “Related guides”

Metadata status

Section titled “Metadata status”
Shell
boilstream-admin cp-metadata status

Investigate unhealthy status or a prolonged lack of leadership before reopening traffic. See rolling updates.