Skip to content
BoilStream

Warm the historical cache

operations
TL;DR

Hydration preloads existing Parquet files from object storage into a node-local cache for repeated reads. Use the administration CLI to request warming and inspect progress. Cache eviction does not delete table data, and warming does not make newly accepted rows durable. Catalog permissions and capacity limits still apply.

ArchitectureHydration copies published files into a disposable local cache
Hydration copies published files into a disposable local cacheA query needs published files from object storage. Hydration workers copy them into a local cache that can be discarded and refilled. The cache is not the recent-data log and is not a recovery backup.SQL clientRequests dataBoilStream queryReads available filesLocal cacheDisposable file copiesHydration workersBounded background loadingObject storagePublished source filesSQLRead filesDownloadFill cacheHydration copies published files into a disposable local cacheA query needs published files from object storage. Hydration workers copy them into a local cache that can be discarded and refilled. The cache is not the recent-data log and is not a recovery backup.SQL clientRequests dataBoilStream queryReads available filesLocal cacheDisposable filecopiesHydration workersBounded backgroundloadingObject storagePublished sourcefilesSQLRead filesDownloadFill cache
Eviction changes local availability and latency. It does not delete the authoritative published object.

Hydration preloads a DuckLake table’s S3 Parquet files into a node’s disposable DuckDB file cache. It can reduce repeated-read latency without changing the table, copying rows, or making local storage authoritative.

S3 remains the source of truth. Hydration does not change the table’s data. Cache contents may be evicted at any time and are rebuilt from S3 after restart or node replacement; no persistent volume is required.

CLI

Section titled “CLI”
Shell
# Service status
boilstream-admin hydration stats
boilstream-admin hydration queue
boilstream-admin hydration health


# Warm a table. The response includes its table/topic ID.
boilstream-admin hydration table -d CATALOG_ID -t TABLE_NAME
boilstream-admin hydration table -d CATALOG_ID -t TABLE_NAME -p query_driven


# Inspect and manage node-local cache work
boilstream-admin hydration list -d CATALOG_ID
boilstream-admin hydration status -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration pending -d CATALOG_ID
boilstream-admin hydration cancel -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration flush -d CATALOG_ID -t TOPIC_ID
boilstream-admin hydration flush-all -d CATALOG_ID --yes

Replace CATALOG_ID, TABLE_NAME and TOPIC_ID with your catalog UUID, table name and the numeric topic ID returned by the service.

Supported priorities are query_driven, scheduled (default), and background. Unknown values are rejected.

REST API

Section titled “REST API”

User endpoints require an authenticated web/API session and the deployment’s MFA enrollment policy. Use the session cookie from sign-in or its session bearer token; an ingest, stream or Schema Registry token is not a session credential. Catalog owners, active tenant members and direct grantees must still have current access. Superadmin endpoints additionally require verified MFA.

The CLI’s hydration table command calls the superadmin endpoint. Applications acting as ordinary catalog users use POST /auth/api/hydration/table instead.

Warm a table

Section titled “Warm a table”

Send JSON to POST /auth/api/hydration/table:

JSON
{
  "ducklake_id": "<catalog-uuid>",
  "schema_name": "main",
  "table_name": "activity",
  "priority": "query_driven"
}

ducklake_id and table_name are required. Omit schema_name only when the table name is unique across active schemas. priority accepts scheduled (default), query_driven or background. An optional snapshot_range object has unsigned integer start (inclusive) and end (exclusive) snapshot bounds.

The JSON response contains success, message, ducklake_id, table_name, nullable topic_id, files_queued, files_skipped and estimated_bytes. Queued work is not completed warming. Keep the returned numeric topic_id for status, cancel and flush requests.

Inspect or cancel work

Section titled “Inspect or cancel work”
Method Endpoint Parameters and response
GET /auth/api/hydration/tables Query: ducklake_id. Returns tables, total_files, total_bytes.
GET /auth/api/hydration/tables/status Query: ducklake_id, topic_id. Returns hydrated_files, hydrated_bytes, pending_files, is_fully_hydrated.
GET /auth/api/hydration/pending Query: ducklake_id. Returns pending_jobs, total_pending, total_pending_bytes.
DELETE /auth/api/hydration/pending JSON: ducklake_id, topic_id. Returns success, message, jobs_cancelled.
DELETE /auth/api/hydration/tables JSON: ducklake_id, topic_id. Returns success, message, files_evicted, pending_cancelled, tables_dropped.
DELETE /auth/api/hydration/tables/all JSON: ducklake_id. Cancels work and flushes the catalog cache.
GET /auth/api/hydration/health Returns worker status and active_workers.

URL-encode query values. tables_dropped is retained for response compatibility; flushing cached files does not drop SQL tables. Cached bytes describe the current resident cache, not a second durable data copy.

Superadmin service endpoints use /auth/api/admin/hydration: POST /table takes the same warm-table body; GET /stats, GET /queue and GET /health report service-wide activity.

Failures

Section titled “Failures”
Status Action
400 Correct the priority, schema/table ambiguity or request values.
401 / 403 Renew the authorized session or resolve MFA/catalog access.
404 Check that the catalog and table exist and are accessible.
429 Hydration capacity is exhausted. Inspect queued work before retrying.
503 Metadata, cache inspection or workers are unavailable; inspect health.

A failed request can have queued some files. Inspect files_queued and the status endpoint; repeating hydration is idempotent for the same existing files. This property applies to cache warming, not to ingestion retries.

Configuration

Section titled “Configuration”

Hydration is enabled by default with bounded node-local work. Optional limits are:

YAML
auth_server:
  enable_hydration: true
  hydration:
    max_admitted_bytes: 2147483648
    max_queue_depth: 1000
    per_ducklake_quota_bytes: 1073741824
    worker_count: 2
    max_retries: 3

max_admitted_bytes covers queued plus active file bytes. A per-DuckLake quota prevents one catalog from consuming the entire admission budget. Retries are bounded; flush and shutdown cancel or drain active work.

See DuckLake and boilstream-admin CLI.