Skip to content
BoilStream

Manage recent data

operations
TL;DR

The local DuckDB hot tier stores recent streaming rows for server-side queries. It complements published Parquet history and has its own commit and failure boundaries. Size it for your retention window and query load. Unpublished recent rows may be lost after node replacement; do not treat them as a rebuildable historical cache.

ArchitecturePGWire can read recent rows and published history
PGWire can read recent rows and published historyA SQL client connects to the BoilStream PGWire server. The server plans reads against recent data and published files in object storage. Other interfaces do not promise the same recent-data visibility.BoilStreamSQL clientPostgreSQL wire protocolQuery planningCatalog and permissionsRecent rowsUnpublished dataObject storagePublished historySQL requestRecent dataPublished filesPGWire can read recent rows and published historyA SQL client connects to the BoilStream PGWire server. The server plans reads against recent data and published files in object storage. Other interfaces do not promise the same recent-data visibility.BoilStreamSQL clientPostgreSQL wireprotocolQuery planningCatalog andpermissionsRecent rowsUnpublished dataObject storagePublished historySQL requestRecent dataPublished files
Use the compatibility reference for each client. A successful write and visibility through a chosen reader are different checks.

BoilStream can keep recently ingested streaming data in node-local DuckDB storage for server-side queries. This hot tier complements durable Parquet data in object storage; it does not replace the managed DuckLake catalog or its recovery path.

Storage roles

Section titled “Storage roles”
State Purpose Durability role
Local DuckDB hot tier Recently committed streaming data A local commit; rows may not yet have a durable cold copy
Hydrated query cache Preloaded historical Parquet files Disposable cache
Parquet objects Analytical table data Durable data copy
Clustered catalog metadata Tables, files, identities, grants, and placement Quorum replicated with S3 recovery backups

Local files are managed by BoilStream. Do not attach or edit them directly: their names and layout are not a public interface, and direct access would bypass tenant and authorization boundaries.

Query behavior

Section titled “Query behavior”

PGWire streaming reads combine committed hot-tier rows with registered cold-tier data. FlightSQL provides the server’s Arrow-native SQL query surface. Native DuckDB clients use the BoilStream extension to attach by UUID and run SQL remotely. Managed DuckDB and FlightSQL do not currently promise the same unpublished recent-data visibility as PGWire.

The hot tier is useful for recent operational queries. Historical scans should use the durable DuckLake path; repeated historical reads may use cold-tier hydration to warm a disposable cache.

Failure and recovery

Section titled “Failure and recovery”
  • Loss or replacement of one node does not change catalog identity or object data.
  • Local hot-tier and hydration files may start empty after replacement. Unpublished hot rows must not be assumed recoverable just because historical cache files are.
  • Durable DuckLake files remain available from object storage.
  • Cluster metadata restores from healthy peers or, after complete local-state loss, the latest completed and validated S3 recovery generation.
  • Metadata recovery does not restore every unpublished hot row. Follow the input protocol’s durability boundary and the recovery instructions for your release.

Applications should not treat the presence of a local hot-tier row as the durability boundary for a particular ingestion interface. Use that interface’s documented acknowledgement and retry contract.

Configuration

Section titled “Configuration”

duckdb_persistence.enabled controls the local hot tier. Its path, writer count, and commit cycle are deployment settings. Start from the generated configuration and use the configuration reference for fields, defaults, and bounds; do not copy a partial storage configuration into production.

Object storage remains independently configured and required for the durable cold path.

Capacity planning

Section titled “Capacity planning”

Hot-tier size, writer count, and commit cycle should be chosen for the recent-data window and concurrent query load. Throughput and latency depend on schema, batch shape, durability settings, hardware, and the simultaneous object-storage workload. Validate them with a representative workload.

Section titled “Related guides”