Cold Tier Hydration

Hydration preloads a DuckLake table's S3 Parquet files into a node's disposable DuckDB file cache. It can reduce repeated-read latency without changing the table, copying rows, or making local storage authoritative.

S3 remains the source of truth. Hydration does not change the table's data. Cache contents may be evicted at any time and are rebuilt from S3 after restart or node replacement; no persistent volume is required.

CLI

# Service status
boilstream-admin hydration stats
boilstream-admin hydration queue
boilstream-admin hydration health

# Warm a table. The response includes its table/topic ID.
boilstream-admin hydration table -d CATALOG_ID -t TABLE_NAME
boilstream-admin hydration table -d CATALOG_ID -t TABLE_NAME -p query_driven

# Inspect and manage node-local cache work
boilstream-admin hydration list -d CATALOG_ID
boilstream-admin hydration status -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration pending -d CATALOG_ID
boilstream-admin hydration cancel -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration flush -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration flush-all -d CATALOG_ID --yes

Replace CATALOG_ID, TABLE_NAME and TOPIC_ID with your catalog UUID, table name and the numeric topic ID returned by the service.

Supported priorities are query_driven, scheduled (default), and background. Unknown values are rejected.

REST API

Authenticated catalog owners, tenant members, and direct grantees can use:

Method	Endpoint	Purpose
POST	/auth/api/hydration/table	Warm a table's active files
GET	/auth/api/hydration/tables	List table cache status
GET	/auth/api/hydration/tables/status	Get one table's status
DELETE	/auth/api/hydration/tables	Cancel work and flush one table
DELETE	/auth/api/hydration/tables/all	Cancel work and flush a catalog
GET	/auth/api/hydration/pending	List queued and active work
DELETE	/auth/api/hydration/pending	Cancel pending table work
GET	/auth/api/hydration/health	Check worker availability

Superadmin service endpoints are under /auth/api/admin/hydration: POST /table, GET /stats, GET /queue, and GET /health.

Catalog permissions apply to hydration requests. Access denial or exhausted capacity can prevent a request from being accepted. Cache operations do not grant access to other catalogs.

Configuration

Hydration is enabled by default with bounded node-local work. Optional limits are:

auth_server:
  enable_hydration: true
  hydration:
    max_admitted_bytes: 2147483648
    max_queue_depth: 1000
    per_ducklake_quota_bytes: 1073741824
    worker_count: 2
    max_retries: 3

max_admitted_bytes covers queued plus active file bytes. A per-DuckLake quota prevents one catalog from consuming the entire admission budget. Retries are bounded; flush and shutdown cancel or drain active work.

See DuckLake and boilstream-admin CLI.