Skip to content
BoilStream

Ingest with FlightRPC

conceptpreview
TL;DR

The producer Flight service receives Arrow batches through DoExchange. It is separate from FlightSQL and does not implement producer DoPut. Completion depends on storage settings and does not imply published table data or completed views. Direct clients need the release-specific exchange contract; use HTTP Arrow when you need the documented request workflow.

Message sequenceConsume the complete Flight exchange before classifying a write
Consume the complete Flight exchange before classifying a writeA Flight producer sends Arrow batches through DoExchange and consumes the server response stream through its terminal status. An initial response alone does not establish a successful exchange.Flight producerBoilStreamOpen DoExchangeSend Arrow batchesExchange responsesFinish sendingTerminal stream statusTime moves downward · dashed arrows are responsesConsume the complete Flight exchange before classifying a writeA Flight producer sends Arrow batches through DoExchange and consumes the server response stream through its terminal status. An initial response alone does not establish a successful exchange.FlightproducerBoilStreamOpen DoExchangeSend Arrow batchesExchange responsesFinish sendingTerminal stream statusTime moves downward · dashed arrows are responses
The direct producer contract is incomplete. Completion also depends on the storage path; this is not a new copy-ready handshake recipe.

The producer FlightRPC service accepts Arrow batches through DoExchange. It is separate from the FlightSQL query service. Both use gRPC over HTTP/2, but their operations and completion rules differ.

Before sending data

Section titled “Before sending data”

Create the destination table in a streaming catalog. Obtain the producer endpoint, TLS trust and scoped credentials from your deployment. The typical producer port is 50051, but it is configurable. Use the exact client and extension builds validated with the server release.

For a complete SQL-based setup, use the Quick Start.

Direct Arrow Flight clients must use the producer’s DoExchange contract, including its destination descriptor and response handling. A generic Flight connection or a successful write call alone does not establish ingestion completion. Keep input until the exchange has returned its terminal result and your required completion boundary is established.

Methods

Section titled “Methods”
Operation Purpose or limit
DoExchange Bidirectional streaming ingestion of Arrow batches
Producer-service DoPut Unimplemented; do not use for bulk uploads
FlightSQL DoPut Separate query service; do not use it as a supported bulk-ingestion endpoint

Completion and visibility

Section titled “Completion and visibility”

Recent queries, stored history and live output can become available at different times.

With hot-only storage, exchange completion confirms admission rather than local commit. Do not use its reported count alone to prove that every input row is stored. With cold storage enabled, upload progress is still distinct from catalog registration and complete cold publication. A successful exchange does not certify completion of every derived view, search index or consumer.

Use Reliability and completion to choose the required boundary. An interrupted exchange may have accepted part of its input. Preserve the input and reconcile before replaying; automatic retries can duplicate rows.

Client errors

Section titled “Client errors”

Authentication failures require valid credentials; schema failures require corrected input. Capacity and transport errors may be temporary, but retry safety depends on whether any data was accepted. Record both the transport result and the application-level response.

See HTTP/2 Arrow for a simpler request/row-count API and Interface compatibility for the current interface limits.

Direct-client availability

Section titled “Direct-client availability”

A complete, versioned destination-descriptor and terminal-response contract is not available in this public guide. Do not infer it from generic PyArrow examples. Use the HTTP Arrow workflow for a documented direct upload path; use FlightRPC only with the producer client supplied and qualified for your deployment.