SSE Consumer API

BoilStream provides a Server-Sent Events (SSE) streaming interface for consuming real-time Arrow data over HTTP/2.

Overview

The SSE consumer interface is designed for:

• Browser-native streaming using the standard EventSource API
• Real-time data delivery with Arrow IPC payloads encoded as base64
• Automatic reconnection with Last-Event-ID for exactly-once delivery
• Schema evolution support with dedicated schema events

Endpoint

GET https://localhost:8443/stream/{token}

Where {token} is a 64-character hex-encoded authentication token with stream purpose, generated via the Web Auth GUI or API.

Protocol Details

Property	Value
Protocol	HTTP/2 with TLS
Port	8443 (default, configurable via http_ingestion.port)
Content-Type	text/event-stream
CORS	Access-Control-Allow-Origin: *
Heartbeat	Every 15 seconds (default)

Event Types

schema

Sent when the client first connects and whenever the schema changes. Contains the Arrow schema as a base64-encoded Arrow IPC stream.

event: schema
data: <base64-encoded Arrow IPC stream>

batch

Contains a batch of rows as a base64-encoded Arrow IPC stream. The id field contains the batch sequence number used for reconnection.

event: batch
id: 42
data: <base64-encoded Arrow IPC stream>

heartbeat

Sent periodically to keep the connection alive. Contains no data.

event: heartbeat
data:

Reconnection

The SSE protocol supports automatic reconnection via the Last-Event-ID header. When a client reconnects:

1. The browser's EventSource automatically sends Last-Event-ID: {last_batch_seq}
2. BoilStream replays all missed batches from the hot tier where batch_seq > last_event_id
3. A fresh schema event is sent before any catchup batches
4. Live streaming resumes after catchup completes

TIP

Browser EventSource handles reconnection automatically. No client-side logic is needed for reliable delivery.

Authentication

Tokens must have the stream purpose. Generate them via:

• Web Auth GUI: Navigate to the token management page and create a token with "stream" purpose
• API: Use the token generation endpoint with purpose: "stream"

Error Codes

Status	Description
200	Success — SSE stream begins
400	Bad token format (not 64 hex characters)
401	Invalid or expired token
403	Token has wrong purpose (not "stream")
410	Topic deregistered or gone

Browser Example

const es = new EventSource("https://your-server:8443/stream/YOUR_TOKEN");

es.addEventListener("schema", (event) => {
  // Decode base64 Arrow IPC → schema
  const bytes = Uint8Array.from(atob(event.data), c => c.charCodeAt(0));
  console.log("Schema received");
});

es.addEventListener("batch", (event) => {
  const batchSeq = event.lastEventId;
  const bytes = Uint8Array.from(atob(event.data), c => c.charCodeAt(0));
  console.log(`Batch ${batchSeq} received`);
});

es.addEventListener("heartbeat", () => {
  console.log("Heartbeat");
});

es.onerror = (err) => {
  console.error("SSE error:", err);
  // EventSource reconnects automatically with Last-Event-ID
};

Node.js Example

import { EventSource } from "eventsource";

const es = new EventSource("https://your-server:8443/stream/YOUR_TOKEN");
// Same event listeners as browser example above

TIP

For a higher-level API with automatic Arrow decoding, see the SSE Consumer SDK guide.

Performance

• Zero-copy forwarding: Single-message batches with pre-encoded IPC data are forwarded directly (base64 only, no decode + re-encode)
• Backpressure handling: If a client is slow, events may be dropped. The client can reconnect with Last-Event-ID to catch up
• Catchup queries: Reconnection catchup uses cached DuckDB VIEWs with generation-based invalidation for efficient database access