Introduction

BoilStream ingests streaming data, runs DuckDB SQL and stores analytical history in DuckLake. It also provides optional Tantivy search and live subscriptions in the same service.

Ingest and store

Create a table in a catalog whose name ends in __stream, then send rows through Kafka, HTTP/2 Arrow, FlightRPC or streaming SQL INSERT. Recent rows enter local DuckDB storage; Parquet files are published to object storage for analytical history.

A producer acknowledgement has a protocol-specific meaning. Acceptance into memory is not a durable commit. Start with the Quick Start, then choose an ingestion interface.

Transform and aggregate

Streaming views filter and transform incoming rows into derived streams. Materialized views run SQL over tumbling or sliding windows. A successful source write does not mean that every view result is already available. See Views and materialized views.

Search

Enable full-text search for tables and fields that need it. New rows can become searchable at a different time from their SQL visibility. See Full-text search.

Query and consume

PGWire serves DuckDB SQL for recent streaming data and published history. Native DuckDB clients use the BoilStream extension to attach a catalog by UUID and run SQL remotely. FlightSQL offers preview Arrow query results.

These clients do not currently promise the same visibility of unpublished recent rows. SSE and Flight cursors provide preview live subscriptions; Kafka replay is limited to retained recent data. See the interface matrix.

Deploy

BoilStream can run as one server or a cluster, using object storage for published history and recovery backups. See Architecture and Reliability and completion for the operating limits.

BoilStream fits workloads that combine streaming ingestion, SQL analytics and search. Keep a general-purpose event broker when you need durable broker retention, transactions or a broader messaging and connector ecosystem.