Skip to content
BoilStream

Ingest with HTTP Arrow

concept
TL;DR

Send binary Arrow IPC to an existing streaming table using its ingest token. HTTP success reports rows accepted into the processing queue. The current server admits a validated request as one group; compare the returned count with the input count. Preserve input after partial or unknown outcomes, and reconcile before sending it again.

Message sequenceAn HTTP response reports admitted rows, not downstream completion
An HTTP response reports admitted rows, not downstream completionA producer encodes Arrow IPC and posts it to the token-scoped ingestion endpoint. BoilStream validates and admits rows, then returns a response. The current handler admits a validated request as one group; unexpected counts and missing responses require reconciliation.ProducerBoilStreamEncode Arrow IPC streamPOST /ingest/{token}Validate schema and admit rowsStatus and admitted row countCompare sent and reported rowsTime moves downward · dashed arrows are responsesAn HTTP response reports admitted rows, not downstream completionA producer encodes Arrow IPC and posts it to the token-scoped ingestion endpoint. BoilStream validates and admits rows, then returns a response. The current handler admits a validated request as one group; unexpected counts and missing responses require reconciliation.ProducerBoilStreamEncode Arrow IPC streamPOST /ingest/{token}Validate schema and admitrowsStatus and admitted rowcountCompare sent and reportedrowsTime moves downward · dashed arrows are responses
Successful admission should report the full request count. Reconcile unexpected counts or lost responses before replaying.

Choose your workflow

Section titled “Choose your workflow”

Before you send

Section titled “Before you send”

Create the target table in a streaming catalog and obtain a topic-scoped ingest token. The Arrow schema must match the table. Use HTTP/2 over trusted TLS; browser origins must be allowed by the deployment.

Keep the input until its outcome is known

Section titled “Keep the input until its outcome is known”

An HTTP 200 response can admit fewer rows than you sent. The count does not identify which rows succeeded. A connection loss can leave the entire outcome unknown. Preserve the original bytes and reconcile; automatic replay can duplicate rows.

For local commit, published history and derived results, see completion boundaries.