Skip to content
BoilStream

Recover cluster state

operations
TL;DR

Recover a replacement node from healthy peers when a metadata quorum remains. After complete local-state loss, restore the latest completed object-storage backup using the original cluster identity and encryption material. Fence the old incarnation before starting a new recovery. Metadata restoration does not recover every unpublished recent row.

Decision flowChoose recovery by what trustworthy state remains
Choose recovery by what trustworthy state remainsDetermine whether a healthy peer remains. With a healthy peer, repair or replace the failed node through the documented recovery path. If all local state is lost, fence old nodes and restore from a completed backup with the original encryption keys, then verify before reopening traffic.Recovery incidentStop unsafe writesHealthy peer?Repair failed nodeUse documented peer pathRestore backupFence nodes; retain keysInspect clusterYesNo local stateChoose recovery by what trustworthy state remainsDetermine whether a healthy peer remains. With a healthy peer, repair or replace the failed node through the documented recovery path. If all local state is lost, fence old nodes and restore from a completed backup with the original encryption keys, then verify before reopening traffic.Recovery incidentStop unsafe writesHealthy peer?Repair failed nodeUse documented peerpathRestore backupFence nodes; retainkeysInspect clusterYesNo local state
After either path, verify data and application behavior before resuming traffic. A backup cannot restore data that never reached it.

Recovery identities

Section titled “Recovery identities”

Choose clusterId once for a logical cluster and never reuse it for an unrelated cluster. For the first installation, use recovery epoch 1 and a unique recovery ID. After the first durable generation exists, switch the bootstrap mode to restore.

For a deliberate all-node recovery:

  1. Fence the previous incarnation so it cannot publish new state.
  2. Increment the recovery epoch exactly once.
  3. Assign a new unique recovery ID to all replicas.
  4. Restore with the original cluster ID, object-storage prefix, and encryption key.
  5. Verify catalog access and writes after leadership changes before reopening traffic.

The latest completed recovery backup is the recovery point. Data acknowledged after that generation but before a sudden all-node loss is not claimed to be in the restored generation unless its interface provides a stronger durability acknowledgement.

Verify before reopening traffic

Section titled “Verify before reopening traffic”

Check all expected replicas, metadata leadership, authenticated catalog access, a small write/read workflow and recovery publication. Keep traffic fenced if any required check fails. Preserve the failed state and restore settings for diagnosis; do not initialize a fresh cluster to mask a failed restore.