Skip to content
BoilStream

Search text

tutorial
TL;DR

Enable full-text search for selected fields in a streaming table, then query its generated search table with multilake_search. Indexing applies to subsequent inserts and becomes visible independently of SQL rows. Choose fields before ingesting data; changing the field list does not automatically rebuild an existing index.

Message sequenceBuild the index, publish it, then query indexed data
Build the index, publish it, then query indexed dataThe SQL client creates a full-text index and waits for its documented build and publication path. Later search queries use the index; visibility in the source table does not alone prove visibility in search.SQL clientBoilStreamSearch indexCreate full-text indexBuild and commit indexPublished index availableRun search queryFind and rank matchesSearch resultsTime moves downward · dashed arrows are responsesBuild the index, publish it, then query indexed dataThe SQL client creates a full-text index and waits for its documented build and publication path. Later search queries use the index; visibility in the source table does not alone prove visibility in search.SQL clientBoilStreamSearch indexCreate full-text indexBuild and commit indexPublished index availableRun search queryFind and rank matchesSearch resultsTime moves downward · dashed arrows are responses
Check indexing progress separately from source-write success. Preserve full-set ranking before applying the final SQL LIMIT.

BoilStream integrates Tantivy, a high-performance full-text search engine, directly into the streaming ingestion pipeline. Enable it for the tables and fields that need search. Incoming rows on those tables feed the index; other tables are not indexed by default.

Search data

Section titled “Search data”

Search is enabled per table and selected text fields. Query the generated search table, named {table}__tantivy_idx, using multilake_search() as shown below. New data becomes searchable independently of the source table’s SQL visibility.

Enabling full-text search

Section titled “Enabling full-text search”

Enable the search service in the deployment first (see Configuration), then enable indexing on the required streaming table:

The examples assume a catalog named my_catalog__stream and a main.documents table with id BIGINT, title VARCHAR and body VARCHAR columns. Create the table first and enable indexing before sending the rows you want to search. See Quick Start for catalog setup and credentials.

SQL
ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = true,
  tantivy_text_fields = 'title,body'
);
Parameter Description
tantivy_enabled true or false — toggles indexing for this table
tantivy_text_fields Comma-separated column names to tokenize for full-text search. If empty, all string columns are stored as exact-match only (no tokenization).
parquet_enabled true or false — toggles S3 Parquet persistence for this table (default: true)

When you enable tantivy on a table, BoilStream:

  1. Creates a search table named {table_name}__tantivy_idx
  2. Begins indexing subsequent inserts for the configured fields

To disable indexing:

SQL
ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = false
);

Tantivy-Only mode

Section titled “Tantivy-Only mode”

You can disable Parquet persistence and use tantivy as the sole storage backend for a table:

SQL
ALTER TABLE my_catalog__stream.main.documents SET (
  tantivy_enabled = true,
  tantivy_text_fields = 'title,body',
  parquet_enabled = false
);

In this mode, data is indexed into tantivy’s hot tier (local disk) and cold tier (S3 bundles) without producing Parquet files. A producer response alone does not establish durable or query-visible search results. This is useful for workloads where full-text search is the primary query pattern and columnar analytics are not needed.

Query the index

Section titled “Query the index”

See the query and type reference for complete syntax, ranking and unsupported types.

Search visibility

Section titled “Search visibility”

New rows become searchable after the configured index commit and publication steps. Search results can lag behind SQL results. When durable search is required, enable object-storage publication and check the result through the intended query interface.

The default commit interval is 30 seconds; it is a scheduling setting, not a search-latency guarantee. Changing indexed fields does not automatically reindex existing rows.

Configuration

Section titled “Configuration”

Enable the search service in the deployment, then enable the required tables:

YAML
tantivy:
  enabled: true
  commit_interval_secs: 30
  s3_upload_enabled: true
  enable_for_all_topics: false

Merge this excerpt into the generated configuration. Keep table-specific enablement unless you intend to index every provisioned topic. See the configuration reference for supported settings. Do not read or modify index files directly; use the SQL search interface.

Use cases

Section titled “Use cases”
  • Log search — Index application logs for fast error investigation across cold storage
  • Document search — Full-text search over ingested documents, articles, or support tickets
  • Product catalog — Search product descriptions and metadata in real-time
  • Event analysis — Find specific events across large streaming datasets by content

Limitations

Section titled “Limitations”

Next steps

Section titled “Next steps”