HTTP Arrow ingestion

Send an Arrow IPC stream to an existing streaming table. The response reports rows accepted into the ingestion queue, not rows committed locally or published to object storage. See Interface compatibility.

Request

POST https://HOST:PORT/ingest/TOKEN
Content-Type: application/vnd.apache.arrow.stream

Use the HTTP ingestion endpoint from your deployment; its default port is 8443. Generate an ingest token for the table in the dashboard. Pass the returned token unchanged; do not construct or decode it in your application. Keep it out of access logs. The table must already exist and the Arrow schema must match it.

The endpoint uses HTTP/2 over TLS. Configure allowed browser origins in the deployment. Use binary Arrow IPC; sendBeacon text payloads are not supported by the current handler. Neither page-unload delivery nor HTTP/2 itself guarantees ingestion or compression.

With an existing Arrow stream file:

# Set INGEST_URL to the endpoint including your scoped token.
curl --http2 --silent --show-error \
  --header 'Content-Type: application/vnd.apache.arrow.stream' \
  --data-binary @events.arrow "$INGEST_URL"

Check the status and response body. Curl completing successfully is not evidence that all input rows were accepted.

Response and partial acceptance

A successful response has this form:

{"status":"success","rows":100}

Compare rows with the number sent. With multiple record batches, earlier batches can be accepted while another fails. The response can still be HTTP 200 with a smaller count. The count does not identify which rows were accepted; do not assume an accepted prefix.

Status	Meaning and action
200	Check the admitted row count. A partial count needs reconciliation.
400	Invalid content type or schema; correct the request.
401 / 403	Token or authorization/origin failure; refresh credentials or correct access.
404	Target not found; check the provisioned table and endpoint.
429	Rate limit; admission checks may supply retry_after_ms. Keep the batch.
503	Admission capacity unavailable; keep the batch and respect retry guidance.
500, missing response or unreadable response	Outcome may be unknown. Reconcile before replaying.

The server's admission checks occur before processing the body. A response from a proxy or a lost connection may have a different meaning. Do not infer safe replay from a status code alone when the responding component is unknown.

Browser upload example

Encode data with an Arrow library using the target schema, then pass the resulting Uint8Array and its row count to this function. It sends one request, retains the caller's bytes, and reports the outcome. It does not clear a queue or silently retry.

async function uploadArrow(endpoint, bytes, expectedRows, maxBytes = 128 * 1024) {
  if (!Number.isSafeInteger(maxBytes) || maxBytes <= 0 ||
      !(bytes instanceof Uint8Array) || bytes.byteLength === 0 ||
      bytes.byteLength > maxBytes || !Number.isSafeInteger(expectedRows) ||
      expectedRows <= 0) {
    throw new Error('Provide a nonempty, bounded Arrow batch and its row count');
  }

  let response;
  let body;
  try {
    response = await fetch(endpoint, {
      method: 'POST',
      headers: { 'Content-Type': 'application/vnd.apache.arrow.stream' },
      body: bytes
    });
    body = await response.json();
  } catch {
    return { outcome: 'unknown' };
  }

  if (response.status === 200 && body?.status === 'success' &&
      Number.isSafeInteger(body.rows) && body.rows >= 0 && body.rows <= expectedRows) {
    return {
      outcome: body.rows === expectedRows ? 'admitted' : 'partial',
      admittedRows: body.rows
    };
  }
  if (response.status === 429 || response.status === 503) {
    const delay = body?.retry_after_ms;
    return {
      outcome: 'retry-later',
      status: response.status,
      retryAfterMs: Number.isSafeInteger(delay) && delay > 0 ? delay : null
    };
  }
  if ([400, 401, 403, 404, 405].includes(response.status)) {
    return { outcome: 'rejected', status: response.status };
  }
  return { outcome: 'unknown', status: response.status };
}

The 128 KiB value is a client-side example bound, not a promise about server request-size limits. Use bounded batches and one upload at a time. Keep a bounded pending queue or a durable application outbox if data must survive a browser close.

Only remove input from the pending queue after the outcome required by your application. admitted establishes memory admission only. For retry-later, preserve the same bytes; retry a confirmed pre-admission rejection after the indicated delay, with a retry limit. For partial or unknown, retain the input and reconcile it instead of retrying blindly. Application event IDs help reconciliation but do not automatically enable server deduplication.

Completion

Local DuckDB commits, Parquet publication, search commits and live delivery can occur at different times. HTTP success does not establish any of those later boundaries. See Reliability and completion before defining retention or retry policies for your producer.