Catalogs and data visibility

concept

TL;DR

A catalog groups tables and controls access to their data. Names ending in __stream enable streaming-table provisioning; ordinary catalogs serve warehouse SQL. Use catalog UUIDs for managed DuckDB attachment. PGWire combines committed recent data with published history; other query interfaces do not promise the same unpublished-row visibility.

Decision flowA shared catalog still checks each user’s permissions

Display names identify a catalog in a user’s context; sharing does not create a second copy of the data.

BoilStream manages DuckLake catalogs, their access permissions and streaming-table provisioning. Native DuckDB clients connect through the BoilStream extension.

Catalog types

Display name	Purpose
Ends in `__stream`, such as `events__stream`	Streaming tables with ingestion topics and metadata columns
No streaming suffix, such as `analytics`	Warehouse tables with ordinary DuckLake SQL

The suffix needs a base name; __stream alone is reserved. When default provisioning is enabled, a new tenant receives duckie and boil__stream. Each tenant’s catalogs have different UUIDs even when their display names match.

Connect a client

Use native DuckDB or PGWire with the credentials for that interface.

Create a streaming catalog

PRAGMA boilstream_create_ducklake('events__stream', 'Application events');
FROM boilstream_ducklakes();

-- Use the UUID of the catalog just created, not its display name.
PRAGMA boilstream_attach_ducklake('<catalog-uuid>', 'events');
USE events;

CREATE TABLE main.activity (
    event_id BIGINT,
    region VARCHAR,
    event_time TIMESTAMP
);

Creating the table provisions its ingestion topic. Send rows through streaming SQL INSERT, Kafka, HTTP/2 Arrow or FlightRPC. Ordinary warehouse tables use standard SQL operations rather than streaming ingestion. See Ingestion interfaces.

Partitioning

Streaming tables receive time partitions based on the ingestion metadata timestamp. Add custom partitions without repeating the automatic year/month/day/hour columns:

ALTER TABLE events.main.activity SET PARTITIONED BY (region);

Use SQL to query and manage partitions. Object names and directory layouts are managed by BoilStream; do not use them as application identifiers or edit files directly.

Hot/cold tiered storage

Recent streaming rows may be queryable before their Parquet files are published to object storage. These have different durability and visibility boundaries.

Visibility paths

Client	Visibility
PGWire	Combines committed recent streaming rows with published history
Managed DuckDB	Remote SQL; does not promise the same unpublished recent-data visibility as PGWire
FlightSQL	Preview Arrow queries; does not promise the same unpublished recent-data visibility as PGWire

An admission response or completed upload does not mean that all rows are available in table queries. See Reliability and completion before choosing when to delete source data or retry a request.

Configuration

Local commit and cold finalization intervals are separate settings. This is an excerpt; retain the other required fields in your generated configuration:

duckdb_persistence:
  commit_interval_ms: 1000

processing:
  snapshot_interval_minutes: 10

Intervals schedule work; they are not end-to-end latency guarantees. See the generated configuration reference.

Cold-tier hydration preloads existing object files for repeated reads. It does not turn newly accepted input into durable data.

DuckDB extension commands · Catalog sharing