Send an Arrow file

tutorial

TL;DR

Create an Arrow IPC stream whose schema matches a provisioned table, then upload it with the table’s ingest token. Compare the returned row count with the number sent. This guide sends two events once; a successful response should report both rows admitted, but does not prove durable publication.

Message sequenceAn HTTP response reports admitted rows, not downstream completion

Successful admission should report the full request count. Reconcile unexpected counts or lost responses before replaying.

Prerequisites

Create main.activity using the first workflow, but skip its SQL insert. Obtain the table’s ingest token from the dashboard. Use Python with PyArrow and a curl build that supports HTTP/2.

1. create the arrow file

Install PyArrow in your application environment:

python -m pip install pyarrow

Save and run this Python script in a writable directory. It creates a two-row Arrow IPC stream:

from datetime import datetime
import pyarrow as pa

schema = pa.schema([
    ('event_id', pa.int64()),
    ('event_type', pa.string()),
    ('event_time', pa.timestamp('us')),
])
table = pa.Table.from_pylist([
    {'event_id': 1, 'event_type': 'view', 'event_time': datetime(2026, 1, 1, 12, 0, 0)},
    {'event_id': 2, 'event_type': 'click', 'event_time': datetime(2026, 1, 1, 12, 0, 1)},
], schema=schema)
with pa.OSFile('events.arrow', 'wb') as output:
    with pa.ipc.new_stream(output, schema) as writer:
        writer.write_table(table)

2. send once

Set INGEST_URL in your private environment to the deployment’s HTTP ingestion endpoint including the exact returned token. For a private CA, configure curl’s CA trust before sending.

curl --http2 --silent --show-error \
  --header 'Content-Type: application/vnd.apache.arrow.stream' \
  --data-binary @events.arrow "$INGEST_URL"

3. check acceptance and visibility

For full acceptance, the response reports:

{"status":"success","rows":2}

A smaller count needs reconciliation; it does not identify an accepted prefix. An absent or unreadable response leaves the outcome unknown. Keep events.arrow and use the response reference before retrying.

Query main.activity through PGWire as in the first workflow. Readiness follows the commit cycle; do not resend the file to fix an empty read.

Browser producers

Use the bounded browser upload helper for response classification. Browser applications also need the deployment to allow their origin.