Skip to content
BoilStream

HTTP Arrow request and responses

reference
TL;DR

POST a binary Arrow IPC stream to the token-bearing ingestion URL. The current server admits the validated request as one group. Its response reports admitted rows, rather than committed or published rows. Check both status and body. A count cannot identify accepted positions, and a lost response requires reconciliation before the same bytes are replayed.

Message sequenceAn HTTP response reports admitted rows, not downstream completion
An HTTP response reports admitted rows, not downstream completionA producer encodes Arrow IPC and posts it to the token-scoped ingestion endpoint. BoilStream validates and admits rows, then returns a response. The current handler admits a validated request as one group; unexpected counts and missing responses require reconciliation.ProducerBoilStreamEncode Arrow IPC streamPOST /ingest/{token}Validate schema and admit rowsStatus and admitted row countCompare sent and reported rowsTime moves downward · dashed arrows are responsesAn HTTP response reports admitted rows, not downstream completionA producer encodes Arrow IPC and posts it to the token-scoped ingestion endpoint. BoilStream validates and admits rows, then returns a response. The current handler admits a validated request as one group; unexpected counts and missing responses require reconciliation.ProducerBoilStreamEncode Arrow IPC streamPOST /ingest/{token}Validate schema and admitrowsStatus and admitted rowcountCompare sent and reportedrowsTime moves downward · dashed arrows are responses
Successful admission should report the full request count. Reconcile unexpected counts or lost responses before replaying.
Jump to reference ↓

Request

Section titled “Request”
HTTP
POST https://HOST:PORT/ingest/TOKEN
Content-Type: application/vnd.apache.arrow.stream

Use the HTTP ingestion endpoint from your deployment; its default port is 8443. Generate an ingest token for the table in the dashboard. Pass the returned token unchanged; do not construct or decode it in your application. Keep it out of access logs. The table must already exist and the Arrow schema must match it.

The endpoint uses HTTP/2 over TLS. Configure allowed browser origins in the deployment. Use binary Arrow IPC; sendBeacon text payloads are not supported by the current handler. Neither page-unload delivery nor HTTP/2 itself guarantees ingestion or compression.

With an existing Arrow stream file:

Shell
# Set INGEST_URL to the endpoint including your scoped token.
curl --http2 --silent --show-error \
  --header 'Content-Type: application/vnd.apache.arrow.stream' \
  --data-binary @events.arrow "$INGEST_URL"

Check the status and response body. Curl completing successfully is not evidence that all input rows were accepted.

Response and row counts

Section titled “Response and row counts”

A successful response has this form:

JSON
{"status":"success","rows":100}

Compare rows with the number sent. The current handler validates the complete request and submits its batches as one admission group. Successful admission should report the full count. If you receive an unexpected partial count from another server version or intermediary, keep the input and reconcile it; do not infer an accepted prefix.

Status Meaning and action
200 Check the admitted row count. A partial count needs reconciliation.
400 Invalid content type or schema; correct the request.
401 / 403 Token or authorization/origin failure; refresh credentials or correct access.
404 Target not found; check the provisioned table and endpoint.
429 Rate limit; admission checks may supply retry_after_ms. Keep the batch.
503 Capacity or availability failure. Only a confirmed pre-admission refusal permits safe replay; keep the batch.
500, missing response or unreadable response Outcome may be unknown. Reconcile before replaying.

Authentication and rate checks precede body intake. Whole-request validation and queue admission follow intake. The typed capacity response includes code: ingestion_capacity, accepted: false and retry guidance. Other availability failures, a proxy response or a lost connection may leave the outcome unknown; a 503 status alone does not prove that nothing was admitted.

Completion

Section titled “Completion”

Local DuckDB commits, Parquet publication, search commits and live delivery can occur at different times. HTTP success does not establish any of those later boundaries. See Reliability and completion before defining retention or retry policies for your producer.