Skip to content
BoilStream

Know when a write is complete

concept
TL;DR

Acceptance, local commit, file upload and table publication are different outcomes. Choose the boundary your application requires before deleting source records or retrying a write. An early empty query does not prove failure. Keep input after a partial or lost response; event identifiers help reconciliation but do not enable automatic deduplication.

Data flowA write crosses several independently observable boundaries
A write crosses several independently observable boundariesAdmission accepts work. Local commit makes a storage mutation available on its storage path. File upload and metadata publication are additional boundaries. Derived views and SSE delivery run separately, so none of these signals proves application processing.ProducerSends rowsAdmissionHTTP / KafkaacknowledgementLocal commitStorage-path completionFile uploadData object storedPublicationMetadata exposes filesDerived resultsViews / live processingSendStorage workPersist filePublish metadataIndependent processingA write crosses several independently observable boundariesAdmission accepts work. Local commit makes a storage mutation available on its storage path. File upload and metadata publication are additional boundaries. Derived views and SSE delivery run separately, so none of these signals proves application processing.ProducerSends rowsAdmissionHTTP / KafkaacknowledgementLocal commitStorage-pathcompletionFile uploadData object storedPublicationMetadata exposesfilesDerived resultsViews / liveprocessingSendStorage workPersist filePublish metadataIndependentprocessing
This is a boundary map, not a universal acknowledgement ladder. Protocol and storage mode determine what a particular response proves.

A write can be accepted before it is durable or queryable. Choose the completion boundary your application needs, and keep the original input until that boundary is established.

When is a write complete?

Section titled “When is a write complete?”
Result What it tells you What it does not tell you
Admission response The service accepted rows for processing The rows are durable or visible to queries
Local commit Recent rows were committed on the receiving server They are published to object storage or visible through every client
Object upload A data file reached object storage The file is available in table queries
Published table data The data is available through the table’s stored history Every view, index or live subscriber has caught up
Live event A batch reached a subscription The receiving application processed or durably stored it

Kafka acks=1 and HTTP success report admission. The current HTTP path admits a validated request as one group; reconcile any unexpected count mismatch instead of replaying it. FlightRPC completion depends on the deployment’s storage settings. See the interface matrix and each API’s response description.

A development deployment using NoOp storage provides no object-storage durability.

Query visibility

Section titled “Query visibility”

PGWire streaming queries combine committed recent data with published history. Managed DuckDB and FlightSQL do not currently promise the same recent-data visibility. An early empty query result is not evidence that an ingestion request failed.

Views, full-text search and live subscriptions can lag behind the source table. Check the result you need instead of treating source-write success as completion of every operation.

Failures and recovery

Section titled “Failures and recovery”

A healthy cluster needs a majority of its replicas for metadata-dependent operations. Clients must reconnect after their serving node fails. Existing connections do not move between servers.

Recovery of catalog metadata does not imply recovery of all recently ingested rows. After complete loss of local state, recovery is limited to the latest completed backup and data that reached its documented durable boundary. Preserve the configured storage, encryption keys and cluster identity needed for restoration.

A cache of published historical files can be rebuilt from object storage. Unpublished recent data cannot be treated as that kind of disposable cache.

Retry decisions

Section titled “Retry decisions”
Decision flowClassify the observed outcome before sending rows again
Classify the observed outcome before sending rows againCompare the response with the number of rows sent. Complete admission proceeds to visibility checks. Known rejection before admission can be corrected and retried. Partial, unreadable or missing results are ambiguous and require reconciliation rather than whole-batch replay.Write responseRead status and row countOutcome known?All rows admittedVerify intended readerRejected before inputFix cause; retry safelyPartial / unknownReconcile with stable IDsInspect resultFull countConfirmed rejectionMissing / partialClassify the observed outcome before sending rows againCompare the response with the number of rows sent. Complete admission proceeds to visibility checks. Known rejection before admission can be corrected and retried. Partial, unreadable or missing results are ambiguous and require reconciliation rather than whole-batch replay.Write responseRead status and rowcountOutcome known?All rows admittedVerify intendedreaderRejected beforeinputFix cause; retrysafelyPartial / unknownReconcile with stableIDsInspect resultFull countConfirmed rejectionMissing / partial
A proxy error or timeout may arrive after the server accepted data. Row counts do not identify an accepted prefix.

A lost response can leave a write outcome unknown. The server may already have accepted some or all of the input. Replaying it can create duplicates.

  • Keep the original input after a partial or unknown result.
  • Reconcile accepted rows before replaying an uncertain request.
  • Retry confirmed pre-admission rejections with bounded backoff.
  • Use application event IDs to support reconciliation; an ID alone does not enable automatic server deduplication.

Automatic reconnection is not an exactly-once delivery guarantee.

Plan for your release

Section titled “Plan for your release”

Use matching server, extension and client versions. Confirm the required write, read, authentication and recovery behavior for that release before production use. The interface matrix identifies preview features and current limits; these pages do not promise that every client or failure scenario is supported.

For operating procedures, see Cluster mode, Kubernetes and Metadata availability.