Skip to content
BoilStream

What is BoilStream?

concept
TL;DR

BoilStream combines streaming ingestion, DuckDB SQL and DuckLake history in one service. Use it to query recent events, transform arriving rows and analyze stored data. Add full-text search where needed. Each interface has its own completion and visibility rules; a producer response does not guarantee durable storage.

ArchitectureClients connect to BoilStream; BoilStream manages the data
Clients connect to BoilStream; BoilStream manages the dataProducers send rows to BoilStream ingestion interfaces. SQL clients query through the server. Inside BoilStream, incoming rows feed recent storage and processing. Durable data is published to object storage. Live applications receive a separate SSE stream, not a durable change log.BoilStream serverProducersServices and collectorsSQL clientspsql · DuckDB · Flight SQLIngestionHTTP Arrow · Kafka ·FlightRecent dataStorage and processingSQL interfacesPermissions and queryexecutionObject storagePublished data filesLive applicationsSSE consumersRowsAccepted inputRead visible dataSQL / resultsPublish filesRead historyLive eventsClients connect to BoilStream; BoilStream manages the dataProducers send rows to BoilStream ingestion interfaces. SQL clients query through the server. Inside BoilStream, incoming rows feed recent storage and processing. Durable data is published to object storage. Live applications receive a separate SSE stream, not a durable change log.BoilStream serverProducersServices andcollectorsSQL clientspsql · DuckDB ·Flight SQLIngestionHTTP Arrow · Kafka ·FlightRecent dataStorage andprocessingSQL interfacesPermissions and queryexecutionObject storagePublished data filesLive applicationsSSE consumersRowsAccepted inputRead visible dataSQL / resultsPublish filesRead historyLive events
These are separate paths: accepting input does not confirm object publication, a query result or live-client processing.

BoilStream ingests streaming data, runs DuckDB SQL and stores analytical history in DuckLake. It also provides optional Tantivy search and live subscriptions in the same service.

Ingest and store

Section titled “Ingest and store”

Create a table in a catalog whose name ends in __stream, then send rows through Kafka, HTTP/2 Arrow, FlightRPC or streaming SQL INSERT. Recent rows enter local DuckDB storage; Parquet files are published to object storage for analytical history.

A producer acknowledgement has a protocol-specific meaning. Acceptance into memory is not a durable commit. Start with the Quick Start, then choose an ingestion interface.

Transform and aggregate

Section titled “Transform and aggregate”

Streaming views filter and transform incoming rows into derived streams. Materialized views run SQL over tumbling or sliding windows. A successful source write does not mean that every view result is already available. See Views and materialized views.

Section titled “Search”

Enable full-text search for tables and fields that need it. New rows can become searchable at a different time from their SQL visibility. See Full-text search.

Query and consume

Section titled “Query and consume”

PGWire serves DuckDB SQL for recent streaming data and published history. Native DuckDB clients use the BoilStream extension to attach a catalog by UUID and run SQL remotely. FlightSQL offers preview Arrow query results.

These clients do not currently promise the same visibility of unpublished recent rows. SSE and Flight cursors provide preview live subscriptions; Kafka replay is limited to retained recent data. See the interface matrix.

Deploy

Section titled “Deploy”

BoilStream can run as one server or a cluster, using object storage for published history and recovery backups. See Architecture and Reliability and completion for the operating limits.

BoilStream fits workloads that combine streaming ingestion, SQL analytics and search. Keep a general-purpose event broker when you need durable broker retention, transactions or a broader messaging and connector ecosystem.