DuckDB hot-tier persistence
BoilStream can keep recently ingested streaming data in node-local DuckDB storage for server-side queries. This hot tier complements durable Parquet data in object storage; it does not replace the managed DuckLake catalog or its recovery path.
Storage roles
Ingested rows -> local DuckDB hot tier -> recent server-side queries
\-> Parquet on object storage -> durable DuckLake queries| State | Purpose | Durability role |
|---|---|---|
| Local DuckDB hot tier | Recently committed streaming data | A local commit; rows may not yet have a durable cold copy |
| Hydrated query cache | Preloaded historical Parquet files | Disposable cache |
| Parquet objects | Analytical table data | Durable data copy |
| Clustered catalog metadata | Tables, files, identities, grants, and placement | Quorum replicated with S3 recovery backups |
Local files are managed by BoilStream. Do not attach or edit them directly: their names and layout are not a public interface, and direct access would bypass tenant and authorization boundaries.
Query behavior
PGWire streaming reads combine committed hot-tier rows with registered cold-tier data. FlightSQL provides the server's Arrow-native SQL query surface. Native DuckDB clients use the BoilStream extension to attach by UUID and run SQL remotely. Managed DuckDB and FlightSQL do not currently promise the same unpublished recent-data visibility as PGWire.
The hot tier is useful for recent operational queries. Historical scans should use the durable DuckLake path; repeated historical reads may use cold-tier hydration to warm a disposable cache.
Failure and recovery
- Loss or replacement of one node does not change catalog identity or object data.
- Local hot-tier and hydration files may start empty after replacement. Unpublished hot rows must not be assumed recoverable just because historical cache files are.
- Durable DuckLake files remain available from object storage.
- Cluster metadata restores from healthy peers or, after complete local-state loss, the latest completed and validated S3 recovery generation.
- Metadata recovery does not restore every unpublished hot row. Follow the input protocol's durability boundary and the recovery instructions for your release.
Applications should not treat the presence of a local hot-tier row as the durability boundary for a particular ingestion interface. Use that interface's documented acknowledgement and retry contract.
Configuration
duckdb_persistence.enabled controls the local hot tier. Its path, writer count, and commit cycle are deployment settings. Start from the generated configuration and use the configuration reference for fields, defaults, and bounds; do not copy a partial storage configuration into production.
Object storage remains independently configured and required for the durable cold path.
Capacity planning
Hot-tier size, writer count, and commit cycle should be chosen for the recent-data window and concurrent query load. Throughput and latency depend on schema, batch shape, durability settings, hardware, and the simultaneous object-storage workload. Validate them with a representative workload.
Related guides
- DuckLake — managed catalogs and hot/cold visibility
- Architecture — end-to-end data and metadata flow
- Reliability and completion — supported failure boundaries
- Ingestion interfaces — public write and query surfaces