Skip to content
BoilStream

How data moves

concept
TL;DR

Rows move through queue acceptance, local storage and published history at different times. PGWire can query committed recent rows alongside stored history. Views, search indexes and live consumers have separate progress. Use these boundaries to decide when to query, retain source data or reconcile an uncertain write.

ArchitecturePGWire can read recent rows and published history
PGWire can read recent rows and published historyA SQL client connects to the BoilStream PGWire server. The server plans reads against recent data and published files in object storage. Other interfaces do not promise the same recent-data visibility.BoilStreamSQL clientPostgreSQL wire protocolQuery planningCatalog and permissionsRecent rowsUnpublished dataObject storagePublished historySQL requestRecent dataPublished filesPGWire can read recent rows and published historyA SQL client connects to the BoilStream PGWire server. The server plans reads against recent data and published files in object storage. Other interfaces do not promise the same recent-data visibility.BoilStreamSQL clientPostgreSQL wireprotocolQuery planningCatalog andpermissionsRecent rowsUnpublished dataObject storagePublished historySQL requestRecent dataPublished files
Use the compatibility reference for each client. A successful write and visibility through a chosen reader are different checks.

BoilStream brings streaming ingestion, DuckDB SQL, DuckLake storage and optional full-text search into one service. Use it to query recent events, analyze stored history and build continuous transformations.

Data flow

Section titled “Data flow”

This diagram shows the available workflows. Their results become available independently: a producer response does not mean that a view, search index or subscriber has caught up. See Reliability and completion.

Recent data and stored history

Section titled “Recent data and stored history”

Recent streaming data can be queried before its Parquet files are published to object storage. PGWire combines available recent data with published history for streaming tables. Do not assume the same recent-data visibility through every query interface.

Published history uses DuckLake tables and Parquet files. Keep both the data and its catalog available for queries. Hydration preloads existing files to improve repeated reads; it does not make newly accepted input durable.

Query paths

Section titled “Query paths”
Client Use it for
PostgreSQL wire protocol (PGWire) DuckDB SQL from SQL clients, including recent streaming data
Native DuckDB with the BoilStream extension Authenticate, attach a catalog by UUID and run SQL remotely
FlightSQL Preview Arrow query results and metadata discovery

Use the interface matrix to choose a client. Managed DuckDB and FlightSQL do not currently promise the same recent-data visibility as PGWire. Direct managed attachment from DuckDB-WASM is unavailable.

Access control

Section titled “Access control”

Users access catalogs through tenant membership or explicit grants. A catalog UUID identifies the catalog; a display name or local SQL alias does not grant access. Sharing gives another user access to the existing catalog without copying its data.

Managed clients use short-lived credentials and TLS. They do not need your object-storage credentials. See Authentication and Catalog sharing.

Deployment

Section titled “Deployment”

Run one server for development or use the cluster deployment for production planning. Availability depends on healthy replicas, object storage and the configured recovery settings. Recent data that has not reached durable storage can be lost on failure. See Reliability and completion before choosing retention and retry policies.