Full-Text Search (Tantivy)

BoilStream integrates Tantivy, a high-performance full-text search engine, directly into the streaming ingestion pipeline. Enable it for the tables and fields that need search. Incoming rows on those tables feed the index; other tables are not indexed by default.

Search data

Search is enabled per table and selected text fields. Query the generated search table, named {table}__tantivy_idx, using multilake_search() as shown below. New data becomes searchable independently of the source table's SQL visibility.

Enabling Full-Text Search

Enable the search service in the deployment first (see Configuration), then enable indexing on the required streaming table:

The examples assume a catalog named my_catalog__stream and a main.documents table with id BIGINT, title VARCHAR and body VARCHAR columns. Create the table first and enable indexing before sending the rows you want to search. See Quick Start for catalog setup and credentials.

ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = true,
  tantivy_text_fields = 'title,body'
);

Parameter	Description
tantivy_enabled	true or false — toggles indexing for this table
tantivy_text_fields	Comma-separated column names to tokenize for full-text search. If empty, all string columns are stored as exact-match only (no tokenization).
parquet_enabled	true or false — toggles S3 Parquet persistence for this table (default: true)

WARNING

At least one of tantivy_enabled or parquet_enabled must remain true. Disabling both is rejected with an error.

When you enable tantivy on a table, BoilStream:

1. Creates a search table named {table_name}__tantivy_idx
2. Begins indexing subsequent inserts for the configured fields

To disable indexing:

ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = false
);

Tantivy-Only Mode

You can disable Parquet persistence and use tantivy as the sole storage backend for a table:

ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = true,
  tantivy_text_fields = 'title,body',
  parquet_enabled = false
);

In this mode, data is indexed into tantivy's hot tier (local disk) and cold tier (S3 bundles) without producing Parquet files. A producer response alone does not establish durable or query-visible search results. This is useful for workloads where full-text search is the primary query pattern and columnar analytics are not needed.

Querying

multilake_search()

Use the multilake_search() table function to run full-text queries against the shadow tantivy table:

SELECT * FROM multilake_search(
  'my_catalog__stream',           -- catalog name
  'documents__tantivy_idx',       -- shadow table name
  'distributed systems'           -- search query
);

The function returns columns from the search table plus a _score relevance column. Use the supported column types below. An optional fourth parameter limits the number of results; it does not select the highest-scoring matches:

-- Return at most 10 results
SELECT * FROM multilake_search('my_catalog__stream', 'documents__tantivy_idx', 'Rust', 10);

Query Syntax

Queries support field-prefixed syntax to target specific indexed columns:

-- Search the "title" field only
SELECT * FROM multilake_search('my_catalog__stream', 'documents__tantivy_idx', 'title:Rust');

-- Search "body" for a phrase
SELECT * FROM multilake_search('my_catalog__stream', 'documents__tantivy_idx', 'body:"distributed systems"');

-- Unqualified terms search across all indexed string fields
SELECT * FROM multilake_search('my_catalog__stream', 'documents__tantivy_idx', 'performance');

Example: Search with Relevance Ranking

SELECT id, title, body, _score
FROM multilake_search('my_catalog__stream', 'documents__tantivy_idx', 'title:Rust')
ORDER BY _score DESC
LIMIT 10;

To rank the full result set, apply SQL ORDER BY and LIMIT as above without the function's fourth argument.

Column Type Support

BoilStream automatically maps Arrow column types to tantivy field types:

Input data	Search behavior
String columns selected in tantivy_text_fields	Tokenized for full-text queries
Other string columns	Exact-match queries
Integers and floating-point values	Numeric queries
Timestamps	Date queries
Booleans	Boolean filters
Binary, nested values and other unsupported types (including decimals and dates without a time)	Not indexed; selecting these columns can fail

The tantivy_text_fields parameter controls which string columns are tokenized for full-text search versus stored as exact-match strings. For tables with unsupported types, select supported columns explicitly rather than SELECT *. The search table excludes the source table's automatic ingestion metadata columns.

Search visibility

New rows become searchable after the configured index commit and publication steps. Search results can lag behind SQL results. When durable search is required, enable object-storage publication and check the result through the intended query interface.

The default commit interval is 30 seconds; it is a scheduling setting, not a search-latency guarantee. Changing indexed fields does not automatically reindex existing rows.

Configuration

Enable the search service in the deployment, then enable the required tables:

tantivy:
  enabled: true
  commit_interval_secs: 30
  s3_upload_enabled: true
  enable_for_all_topics: false

Merge this excerpt into the generated configuration. Keep table-specific enablement unless you intend to index every provisioned topic. See the configuration reference for supported settings. Do not read or modify index files directly; use the SQL search interface.

Use Cases

• Log search — Index application logs for fast error investigation across cold storage
• Document search — Full-text search over ingested documents, articles, or support tickets
• Product catalog — Search product descriptions and metadata in real-time
• Event analysis — Find specific events across large streaming datasets by content

Limitations

WARNING

• Streaming tables only — Full-text search is only available on tables in __stream DuckLake catalogs.
• No automatic schema migration — Choose tantivy_text_fields before indexing. Changing it does not reindex existing data, and is not a supported way to update an active index. Use a new table with the desired settings and replay the required source data.
• Visibility delay — Search depends on index commit and publication; the interval is not a maximum delivery delay.
• Deployment support — Use a server release that includes full-text search and enable it in the deployment.

Next Steps

• Learn about Materialized Views for real-time streaming transformations
• Explore the DuckLake Integration for catalog management
• Check out Configuration for full server settings