Kubernetes on CloudFleet + Hetzner

Step-by-step install of BoilStream on a CloudFleet Kubernetes Engine (CFKE) cluster running Hetzner ARM64 nodes with local NVMe, using Hetzner Object Storage for S3.

What you get:

• CloudFleet — managed Kubernetes control plane + autoscaler (Karpenter). You bring a Hetzner account and CFKE provisions the cluster for you.
• Hetzner Cloud — provides the nodes. We pin to the cax* (ARM64 Ampere Altra) line with local NVMe, so DuckDB hot-tier and Tantivy index writes hit local disk, not network block storage.
• Hetzner Object Storage — S3-compatible bucket for cluster state (leader election, broker registry) and per-user catalog backups.

The chart ships a values-hetzner-example.yaml overlay that matches this layout; defaults pull the N1-safe image from Docker Hub so nothing else needs to change.

1. Create the Hetzner Object Storage bucket

Create one bucket in your chosen Hetzner region (e.g. hel1 / Helsinki). Note the S3 endpoint — Hetzner Object Storage requires path-style addressing, which the chart handles via s3.forcePathStyle: true.

Generate an S3 access key / secret key scoped to the bucket. Keep them handy for step 4.

Endpoint:    https://hel1.your-objectstorage.com
Bucket:      boilstream-state
Access key:  …
Secret key:  …

2. Provision the CFKE cluster on Hetzner ARM64 + local NVMe

In CloudFleet, create a cluster backed by Hetzner Cloud and define a NodePool pinned to ARM64 instances with local NVMe. Recommended minimum for a 2-pod BoilStream cluster:

• Instance type: cax21 (Ampere Altra, 4 vCPU, 8 GB RAM, 80 GB NVMe) or larger (cax31, cax41)
• Architecture: arm64
• Region: matches the object-storage region (e.g. hel1)
• Count: 2 (one node per BoilStream pod — anti-affinity keeps them separated)

Why cax* and not cx*

The cax* ARM64 line ships with local NVMe on every size tier. The cx* Intel line uses network block storage on the smaller tiers. BoilStream's hot tier benefits from local NVMe (lower latency writes, no network I/O contention).

Tag your CFKE NodePool so the chart's nodeSelector matches — e.g. boilstream.com/nodepool: arm64-hel (the chart's values-hetzner-example.yaml selects this label by default).

Point kubectl at the CFKE cluster (CFKE gives you a kubeconfig) and verify:

kubectl get nodes
# NAME                          STATUS   ROLES    AGE   VERSION
# abc-1234…                    Ready    <none>   2m    v1.33.x
# def-5678…                    Ready    <none>   2m    v1.33.x

3. Install the prerequisites

BoilStream needs two cluster-wide components before you install the chart:

# cert-manager (issues the public wildcard TLS cert + internal cluster mTLS)
helm install cert-manager jetstack/cert-manager \
  --version v1.16.2 \
  -n cert-manager --create-namespace \
  --set crds.enabled=true

# Envoy Gateway (TLS passthrough + SNI routing for pgwire/kafka/flight/flightsql/auth)
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.1 \
  -n envoy-gateway-system --create-namespace

Then create a ClusterIssuer for Let's Encrypt — the chart references it by name.

DNS-01 is required

Envoy Gateway uses TLS passthrough for :443, so cert-manager's http01 solver can't intercept the ACME challenge. Use dns-01 with a provider cert-manager supports. Example below uses Hetzner DNS via the community cert-manager-webhook-hetzner; any supported DNS provider works.

# cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt-prod }
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@your-domain.example
    privateKeySecretRef: { name: letsencrypt-prod-key }
    solvers:
      - dns01:
          webhook:
            groupName: acme.your-domain.example
            solverName: hetzner
            config:
              secretName: hetzner-dns-api-token   # pre-create with your Hetzner DNS API token
              zoneName: your-domain.example

kubectl apply -f cluster-issuer.yaml

4. Create BoilStream's secrets

Three secrets in the boilstream namespace, all pre-created so Helm never sees them in values:

kubectl create namespace boilstream

# (a) Hetzner S3 credentials
kubectl -n boilstream create secret generic boilstream-s3-creds \
  --from-literal=access_key=<HETZNER_S3_ACCESS_KEY> \
  --from-literal=secret_key=<HETZNER_S3_SECRET_KEY>

# (b) Superadmin password
kubectl -n boilstream create secret generic boilstream-superadmin \
  --from-literal=password='<STRONG_PASSWORD>'

# (c) Superadmin MFA secret (optional — enables CLI automation via TOTP).
#     TOTP shared secrets must be base32 (RFC 4648); base64 or hex won't
#     enroll. `oathtool` ships on most distros; pyotp works too.
MFA_B32=$(oathtool --totp -b --key "$(openssl rand -hex 20)" -v 2>/dev/null | awk '/Base32 secret/{print $3}')
kubectl -n boilstream create secret generic boilstream-superadmin-mfa \
  --from-literal=mfa_secret="$MFA_B32"
# save MFA_B32 somewhere safe — you'll seed it into your TOTP app (1Password,
# Authy, google-authenticator) to generate codes.

5. Adjust the overlay for your bucket and domain

Copy values-hetzner-example.yaml and change only the Hetzner-specific fields:

# my-values.yaml
domain: data.your-company.example   # wildcard cert will cover *.data.your-company.example

s3:
  endpoint: https://hel1.your-objectstorage.com
  bucket: boilstream-state
  region: hel1
  prefix: cluster/
  existingCredsSecret: boilstream-s3-creds   # from step 4(a)

superadmin:
  existingSecret: boilstream-superadmin       # from step 4(b)
  existingMfaSecret: boilstream-superadmin-mfa # from step 4(c), omit if skipped

tls:
  issuer:
    create: false
    name: letsencrypt-prod                     # ClusterIssuer from step 3
    kind: ClusterIssuer

Everything else in values-hetzner-example.yaml (anti-affinity, Karpenter toleration, nodeSelector: arm64-hel, 2 replicas, 1 vCPU / 2 GiB requests) is already sized for cax21 nodes.

6. Install the chart

git clone https://github.com/boilingdata/boilstream
cd boilstream

helm install boilstream ./charts/boilstream \
  -f ./charts/boilstream/values-hetzner-example.yaml \
  -f my-values.yaml \
  -n boilstream

kubectl -n boilstream rollout status statefulset/boilstream --timeout=5m

The default image (docker.io/boilinginsights/boilstream:aarch64-generic-linux-{VERSION}) is the N1-safe ARM64 build — correct for Hetzner Ampere Altra. Replace {VERSION} with the latest release; see the BoilStream releases page.

7. Point DNS at the Gateway LoadBalancer

CloudFleet provisions a Hetzner Load Balancer in front of Envoy Gateway. Grab its external IP and create two DNS records at your provider:

kubectl -n boilstream get gateway boilstream -o jsonpath='{.status.addresses[0].value}'
# → 65.x.y.z

Record	Type	Target
data.your-company.example	A	65.x.y.z
*.data.your-company.example	A	65.x.y.z

cert-manager issues the wildcard once DNS resolves.

8. Verify

# Pods ready + gateway provisioned
kubectl -n boilstream get pods
kubectl -n boilstream get gateway,tlsroute

# Download the matching boilstream-admin CLI (same release as the image)
VERSION=0.10.19   # set to the appVersion you helm-installed
curl -L -o boilstream-admin \
  https://www.boilstream.com/binaries/$(uname -s | tr A-Z a-z)-$(uname -m)/boilstream-admin-$VERSION
chmod +x boilstream-admin

# Log in as superadmin (reads password from the K8s Secret; TOTP from MFA_B32)
kubectl -n boilstream get secret boilstream-superadmin \
  -o jsonpath='{.data.password}' | base64 -d > /tmp/password.txt
echo "$MFA_B32" > /tmp/mfa.txt
BOILSTREAM_PASSWORD_PATH=/tmp/password.txt \
BOILSTREAM_MFA_SECRET_PATH=/tmp/mfa.txt \
  ./boilstream-admin auth login \
    --server https://data.your-company.example:8443 \
    --as-profile hetzner
rm /tmp/password.txt /tmp/mfa.txt

./boilstream-admin -P hetzner cluster status

PGWire verification

PGWire auth uses vended temporary credentials (superadmin_<random>), not the raw superadmin password. Vend a PG credential via ./boilstream-admin -P hetzner ... or the Web Auth UI, then connect.

The vended host + port from the admin CLI / GUI points at the per-pod TCPRoute on pgwire.publicTcpPortBase + pod_index (default base 15432). That listener is L4-passthrough — Envoy doesn't parse TLS — so any libpq-based client connects with plain sslmode=require:

psql "host=boilstream-0.data.your-company.example port=15432 \
      user=<vended_user> dbname=<vended_db> sslmode=require"

If you'd rather use the bare app_domain:5432 SNI-routed listener, that path requires direct TLS (libpq 17+, psql 17+, pgjdbc 42.7+):

psql "host=data.your-company.example port=5432 user=<vended_user> \
      dbname=memory sslmode=verify-full sslnegotiation=direct"

Both reach the same pgwire backend. Tools that don't expose sslnegotiation (DBeaver / Tableau ODBC / Power BI / DuckDB's stock postgres extension) should use the per-pod port; libpq-17 clients can use either.

Healthy output:

Cluster Status: HEALTHY
Leader:         boilstream-1 (acquired 2m ago)
Brokers:        2 active, 0 stale

Scaling and operations

• Add more nodes: scale the CFKE NodePool up; Karpenter provisions more cax* instances. Bump replicas in your overlay and helm upgrade.
• Rolling upgrades: bump image.tag in your overlay and helm upgrade. The PDB keeps one pod available throughout.
• Failover test: kubectl -n boilstream delete pod boilstream-<leader>. A peer promotes within ~60 s (leader lease on S3 expires at the stale threshold). Non-leader pods keep accepting PGWire — SCRAM hashes are fetched from whoever the current leader is over the internal :8444 mTLS API, so failover doesn't break in-flight clients.
• Hot-tier sizing: each cax* node's local NVMe holds hot DuckDB state. Cold data continuously uploads to Hetzner Object Storage, so pod loss is recoverable — the new pod pulls per-user catalog backups from S3 on start.

Reference

• Chart source: github.com/boilingdata/boilstream/charts/boilstream
• Example overlay: values-hetzner-example.yaml
• Docker Hub image: boilinginsights/boilstream — use aarch64-generic-linux-* for Hetzner ARM64
• Binaries (if you prefer bare VMs over Kubernetes): https://www.boilstream.com/binaries/linux-aarch64/boilstream-{VERSION}-generic