Skip to content
BoilStream

Update a cluster

operations
TL;DR

Update one replica at a time while preserving a healthy metadata majority. Pin the intended release artifacts, keep stable keys and recovery identity, and wait for each replacement to become ready. Clients may need to reconnect. Stop the rollout if access, data verification or recovery publication fails.

Message sequenceUpgrade one replica, verify it, then move to the next
Upgrade one replica, verify it, then move to the nextAn operator replaces a single replica, waits for its readiness checks and verifies application reads and writes before advancing. Failed checks stop the rollout.OperatorOne replicaTest clientReplace one replicaReadiness and health statusRun application checksVerify reads and writesObserved resultsPass: continue; fail: stopTime moves downward · dashed arrows are responsesUpgrade one replica, verify it, then move to the nextAn operator replaces a single replica, waits for its readiness checks and verifies application reads and writes before advancing. Failed checks stop the rollout.OperatorOne replicaTest clientReplace one replicaReadiness and healthstatusRun application checksVerify reads and writesObserved resultsPass: continue; fail:stopTime moves downward · dashed arrows are responses
Do not replace all replicas together. A ready process alone is not an application-level upgrade check.

Upgrades and failures

Section titled “Upgrades and failures”

Upgrade one replica at a time and wait for readiness before continuing:

Shell
helm upgrade boilstream oci://ghcr.io/boilingdata/charts/boilstream \
  --version '<chart-version>' \
  --namespace boilstream \
  --values my-values.yaml \
  --set-string image.digest='sha256:<release-image-digest>' \
  --set-string configRenderer.image.digest='sha256:<python-image-digest>'
kubectl -n boilstream rollout status statefulset/boilstream

The default disruption budget keeps two of three replicas available during voluntary maintenance. Spread replicas across nodes or failure domains when the platform permits.

A client connected to a failed pod must reconnect; existing TCP sessions are not promised to migrate. With quorum, metadata operations continue through healthy replicas. Without quorum, metadata-dependent admission fails closed. A replacement pod catches up from peers without a PVC.

For complete local-state loss, fence the old incarnation, increment the recovery epoch once, set a new recovery ID on every replica, and restore with the original cluster ID, S3 prefix, and encryption key. The recovery point is the latest completed generation.

Failure and rollback

Section titled “Failure and rollback”

Stop the rollout if quorum or verification fails. Keep the stable encryption/signing keys and cluster identity. Restore the previous deployment values and image only when the release supports that downgrade; otherwise follow its recovery procedure. Do not assume a Helm rollback reverses stored data or format changes.

Verify

Section titled “Verify”
Shell
kubectl -n boilstream rollout status statefulset/boilstream
kubectl -n boilstream get pods,gateway,certificate
boilstream-admin cluster status

Before admitting production traffic, exercise authentication, catalog vending, every enabled ingestion/query interface, a leader failover, a rolling restart, and an S3-only restore in a staging cluster built from the same release artifacts.