Kafka interface

BoilStream accepts Kafka producer requests for tables already provisioned in a streaming catalog. It implements a subset of Kafka, not a general-purpose broker replacement.

Protocol and payload

Kafka RecordBatch v2 is the wire format. The documented record payload is Confluent Avro: a magic byte, a Schema Registry ID and Avro-encoded values. These are different layers of the request.

Create the table through SQL first. BoilStream derives its schema for the Schema Registry. Do not rely on a producer to create tables or register arbitrary new schemas. Raw JSON, text, Protobuf and arbitrary binary payloads are outside the documented ingestion path.

Acknowledgements

Setting	Behavior
acks=1	Responds after decoding and memory admission of the partition's batch group
acks=0	Sends no Produce response; the producer cannot confirm acceptance
acks=all / -1	Rejected; this endpoint does not provide Kafka replication acknowledgements
Transactions	Not implemented

Admission is not local DuckDB commit, cold publication or replication of the input rows. Do not use the acknowledgement as an object-storage durability guarantee. See Reliability and completion.

Producer configuration

Use the host, port, topic name, ingest token and trusted CA provided by your deployment. A Java producer configuration starts with:

bootstrap.servers=HOST:PORT
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="TOPIC" password="INGEST_TOKEN";
acks=1
enable.idempotence=false
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=https://HOST/schema-registry
auto.register.schemas=false

The registry also requires a Schema Registry bearer token; configure it using your serializer client's authentication options. It is separate from the Kafka ingest token. Configure the Kafka client's TLS trust store and the registry's TLS trust separately when using a private CA. Match the producer's Avro schema to the registry schema and check every delivery callback. Client defaults vary, so set acknowledgements and idempotence explicitly.

For Python's confluent-kafka, the corresponding producer settings are:

producer_config = {
    'bootstrap.servers': 'HOST:PORT',
    'security.protocol': 'SASL_SSL',
    'sasl.mechanism': 'PLAIN',
    'sasl.username': 'TOPIC',
    'sasl.password': 'INGEST_TOKEN',
    'acks': '1',
    'enable.idempotence': False,
}

Serialize values with Confluent Avro using the existing registry schema before producing. The dictionary above configures the connection; it does not serialize values. Use the client and serializer versions validated with your server release.

Failures and retries

Queue pressure can reject admission. Oversized input and invalid Avro are different errors and should not be retried unchanged. Inspect per-partition errors and the client's delivery report; a successful network send is not an admission receipt.

A lost response can leave the result unknown. Producer retries may create duplicates; BoilStream does not promise Kafka idempotent-producer or transaction semantics. Keep source records until your required completion boundary and reconcile uncertain outcomes.

Consumption limits

Kafka Fetch has a live path and a fallback over retained local hot chunks. It does not provide replay of cold files from object storage. Once data leaves hot retention, Kafka offsets are not an unlimited historical log.

Treat Kafka consumption as a preview, including recovery after lag or a server failure. Use DuckLake SQL for published analytical history; retain a separate broker if applications require durable broker replay or Kafka transaction semantics.

See Interface compatibility for the other producer and query paths.