Skip to content
BoilStream

Kafka protocol and acknowledgements

reference
TL;DR

Kafka ingestion accepts RecordBatch v2 containing Confluent Avro values for a provisioned streaming table. Use acks=1 for an admission response; acks=0 gives none, and acks=all is rejected. Transactions and idempotent-producer semantics are unavailable. Preview consumption is limited by retained recent data rather than an unlimited broker log.

Message sequenceKafka producers receive an admission acknowledgement
Kafka producers receive an admission acknowledgementA producer encodes records with a compatible schema and sends a Produce request. BoilStream handles the request and returns an acknowledgement; this does not establish all Kafka broker durability guarantees.Kafka producerBoilStreamEncode records with schemaProduce request (acks=1)Validate and admit rowsProduce acknowledgementTime moves downward · dashed arrows are responsesKafka producers receive an admission acknowledgementA producer encodes records with a compatible schema and sends a Produce request. BoilStream handles the request and returns an acknowledgement; this does not establish all Kafka broker durability guarantees.KafkaproducerBoilStreamEncode records withschemaProduce request (acks=1)Validate and admit rowsProduce acknowledgementTime moves downward · dashed arrows are responses
An acknowledgement is an admission signal. Check supported protocol features and reconcile ambiguous writes before replaying.
Jump to reference ↓

Protocol and payload

Section titled “Protocol and payload”

Kafka RecordBatch v2 is the wire format. The documented record payload is Confluent Avro: a magic byte, a Schema Registry ID and Avro-encoded values. These are different layers of the request.

Create the table through SQL first. BoilStream derives its schema for the Schema Registry. Do not rely on a producer to create tables or register arbitrary new schemas. Raw JSON, text, Protobuf and arbitrary binary payloads are outside the documented ingestion path.

Acknowledgements

Section titled “Acknowledgements”
Setting Behavior
acks=1 Responds after decoding and memory admission of the partition’s batch group
acks=0 Sends no Produce response; the producer cannot confirm acceptance
acks=all / -1 Rejected; this endpoint does not provide Kafka replication acknowledgements
Transactions Not implemented

Admission is not local DuckDB commit, cold publication or replication of the input rows. Do not use the acknowledgement as an object-storage durability guarantee. See Reliability and completion.

Failures and retries

Section titled “Failures and retries”

Queue pressure can reject admission. Oversized input and invalid Avro are different errors and should not be retried unchanged. Inspect per-partition errors and the client’s delivery report; a successful network send is not an admission receipt.

A lost response can leave the result unknown. Producer retries may create duplicates; BoilStream does not promise Kafka idempotent-producer or transaction semantics. Keep source records until your required completion boundary and reconcile uncertain outcomes.

Consumption limits

Section titled “Consumption limits”

Kafka Fetch has a live path and a fallback over retained local hot chunks. It does not provide replay of cold files from object storage. Once data leaves hot retention, Kafka offsets are not an unlimited historical log.

Treat Kafka consumption as a preview, including recovery after lag or a server failure. Use DuckLake SQL for published analytical history; retain a separate broker if applications require durable broker replay or Kafka transaction semantics.

See Interface compatibility for the other producer and query paths.