Configuration

Start from the configuration generated by your server version. Edit the settings needed for your deployment and keep the other required fields. The field reference lists accepted keys, types and schema defaults; deployment examples can use different values.

Configuration file

# Generate or use config.yaml in the current directory
./boilstream

# Use an existing configuration file
./boilstream --config /etc/boilstream/config.yaml

# Or select it through the environment
CONFIG_FILE=/etc/boilstream/config.yaml ./boilstream

Supported environment overrides take precedence over YAML values. Use the options provided by your installed version; arbitrary environment-variable names do not become settings.

Use the complete example below as a reference, then adapt storage, endpoints and security settings for your deployment.

Storage configuration

Choose a real storage backend before relying on persisted history. The noop backend is for development and provides no object-storage durability.

For S3, configure a primary backend with backend_type: s3, the bucket, prefix and region. Use use_path_style only when required by the endpoint.

Supply credentials using your deployment's credential mechanism. For a compatible service such as MinIO, set its endpoint and the addressing mode required by that service. See AWS IAM permissions before provisioning AWS storage.

Keep the primary data storage and configured recovery storage available. Do not rename, delete or directly modify service-managed objects. Retain the encryption keys required to read existing data and backups.

Network interfaces

Setting	Purpose
auth_server.port	Login, dashboard and authenticated administration
pgwire.port	PostgreSQL wire-protocol SQL clients
kafka.port	Kafka producers and preview consumers
http_ingestion.port	HTTP Arrow ingestion and preview SSE
server.flight_base_port	Flight deployment configuration
server.consumer_flight_port	FlightSQL query service
metrics.port	Monitoring endpoint

Use the endpoint returned by the dashboard or supplied by the operator, especially in a cluster. A configured port alone does not establish a supported client workflow. See Interface compatibility.

Configure TLS separately for the enabled interfaces. Use trusted certificates for public clients. Keep monitoring and administrative interfaces restricted to the appropriate network and users.

Authentication and authorization

Configure the login methods your organization uses, then assign tenant membership and catalog permissions. Authentication and catalog authorization are separate controls. See Authentication and Catalog sharing.

The Schema Registry is served under /schema-registry on the auth server. Every request requires a Schema Registry bearer token; the Kafka ingest token is not interchangeable. See Schema Registry.

HTTP/2 ingestion configuration

Enable the HTTP interface and allow only the browser origins that need access. Use the field reference for accepted HTTP settings. Applications must use binary Arrow IPC and inspect the admitted row count; see HTTP ingestion.

Data visibility and sizing

duckdb_persistence.commit_interval_ms schedules local commits; processing.snapshot_interval_minutes controls file completion. The example uses 1,000 ms and 10 minutes respectively. These are settings, not end-to-end latency guarantees.

Local commit and publication to stored history happen at different times. Size recent-data retention, batches and storage capacity using your workload. Use the generated defaults for advanced tuning unless measurements show a need to change them. See Reliability and completion.

Optional features

• Full-text search: enable the service, then the required tables and text fields.
• Hydration: preload historical files for repeated reads.
• Metrics: configure the scrape endpoint and monitoring.
• Audit logging: choose the destination and whether queries and IP addresses should be included. Audit logging alone does not establish a compliance certification.

Cluster mode configuration

Use the Kubernetes profile for the production cluster setup. It includes replica count, service endpoints, TLS, storage and recovery settings. A shared bucket alone does not establish a safe cluster configuration.

Keep stable cluster identity and key material across restarts. Follow Cluster mode for quorum requirements and deliberate recovery; do not bootstrap an existing deployment as an empty new cluster.

Validation and upgrades

The server validates configuration at startup. Correct reported errors before admitting production traffic. A YAML file parsing successfully does not prove that its storage, certificates or credentials work.

When upgrading, review the migration before applying it:

./boilstream --upgrade-config config.yaml --dry-run

Keep a backup of the previous configuration and follow the migration instructions for the release. Use boilstream-admin for server and CLI configuration commands; they manage different settings.

Complete configuration example

Example YAML for a single-server deployment

Replace example endpoints and storage values before use. This is not the production cluster profile; use the Kubernetes guide for that setup.

aws:
  region: "eu-west-1"
  access_key_id: null # Optional - uses AWS SDK credential chain
  secret_access_key: null # Optional - uses AWS SDK credential chain
  https_conn_pool_size: 100

storage:
  backends:
    - name: "primary-s3"
      backend_type: "s3"
      enabled: true
      primary: true
      endpoint: "https://s3.eu-west-1.amazonaws.com"
      bucket: "my-bucket"
      prefix: ""
      region: "eu-west-1"
      access_key: null # Optional - uses AWS SDK credential chain
      secret_key: null
      use_path_style: false # true for MinIO
      max_concurrent_uploads: 10
      max_retries: 3
      initial_backoff_ms: 100
      max_retry_attempts: 3

server:
  tokio_worker_threads: null # null = system CPU count
  flight_thread_count: 1
  flight_base_port: 50050
  admin_flight_port: 50160
  consumer_flight_port: 50250
  router:
    spsc_queue_capacity: 2
    batch_size: 500 # Messages per batch
    retry_delay_ms: 1

processing:
  data_processing_threads: 8
  buffer_pool_max_size: 50
  window_queue_capacity: 30000
  max_flush_latency_ms: 10000 # Max time before flush (ms)
  max_part_size_mb: 32 # Size threshold for early flush
  max_file_size_mb: 1024 # Max file size before completion
  snapshot_interval_minutes: 10 # Wall-clock aligned file completion (1-60)
  enable_5mb_part_padding: false # Pad to 5MB for S3 multipart
  include_metadata_columns: true
  schema_validation_enabled: true
  parquet:
    compression: "ZSTD"
    dictionary_enabled: true

duckdb_persistence:
  enabled: true
  storage_path: "./data/duckdb/topics"
  max_writers: 10
  total_databases: 10 # Local database capacity
  commit_interval_ms: 1000 # Transaction commit interval

rate_limiting:
  disabled: false
  max_requests: 15000000 # Per producer
  burst_limit: 20000000
  global_limit: 150000000
  base_size_bytes: 4096

tls:
  disabled: false
  cert_path: "/etc/ssl/certs/server.crt"
  key_path: "/etc/ssl/private/server.key"
  cert_pem: null # Alternative to cert_path
  key_pem: null # Alternative to key_path
  grpc_default_ssl_roots_file_path: null

auth:
  providers: ["cognito"] # cognito, azure, gcp, auth0, okta
  authorization_enabled: true
  admin_groups: ["admin"]
  read_only_groups: ["viewers"]
  max_sessions_per_hour: 1000
  max_concurrent_sessions: 20
  max_session_ttl_hours: 8
  cognito:
    user_pool_id: "us-east-1_example"
    region: "us-east-1"
    audience: "client-id"
  azure:
    tenant_id: null
    client_id: null
    allow_multi_tenant: false
  gcp:
    client_id: null
    project_id: null
    require_workspace_domain: false
  auth0:
    tenant: null
    audience: null
    groups_namespace: null
  okta:
    org_domain: null
    audience: null
    auth_server_id: null

metrics:
  port: 8081
  flush_interval_ms: 1000

logging:
  rust_log: "info"

pgwire:
  enabled: true
  port: 5432
  refresh_interval_seconds: 5
  initialization_sql: |
    SET threads = 8;
    SET memory_limit = '24GB';
    SET max_temp_directory_size = '16GB';
    SET allowed_directories = ['/tmp'];
    SET allow_unredacted_secrets = false;
    LOAD json;
    LOAD httpfs;
    LOAD ducklake;
    LOAD boilstream;
    SET allow_unsigned_extensions = false;
    SET lock_configuration = true;
  metadata_db_path: "./metadata.duckdb"
  query_timeout_seconds: 300 # 5 minutes
  max_connections: 100
  idle_timeout_seconds: 900 # 15 minutes
  connection_timeout_seconds: 1800 # 30 minutes
  tenant_data_base_path: "/tmp/boilstream/tenants"
  metadata_backup_backend: null # Storage backend name
  metadata_backup_interval_seconds: 60
  metadata_backup_path: "pgwire/metadata-backup.tar.gz"
  tls:
    enabled: true
    cert_path: "/etc/ssl/certs/pgwire.crt"
    key_path: "/etc/ssl/private/pgwire.key"
    cert_pem: null
    key_pem: null

kafka:
  enabled: true
  port: 9092
  bind_address: "0.0.0.0"
  tls:
    enabled: true
    cert_path: "/etc/ssl/certs/kafka.crt"
    key_path: "/etc/ssl/private/kafka.key"
    cert_pem: null
    key_pem: null
  rate_limiting:
    disabled: false
    max_requests_per_second: 10000
    burst_limit: 15000
    global_limit: 100000

http_ingestion:
  enabled: true
  port: 8443
  bind_address: "0.0.0.0"
  token_secret_key: ""                   # BLAKE3 MAC key for browser tokens (32 bytes, base64)
  max_request_size: 131072 # 128KB
  tls:
    cert_path: "/etc/ssl/certs/http.crt"
    key_path: "/etc/ssl/private/http.key"
    cert_pem: ""
    key_pem: ""
  cors:
    allowed_origins: []
    max_age_seconds: 3600
    allow_credentials: false
  rate_limiting:
    enabled: true
    global_requests_per_second: 10000
    tenant_requests_per_second: 1000
    tenant_burst_size: 5000
    tenant_overrides: {}
  http2:
    max_connections: 100000
    accept_threads: 0 # 0 = CPU cores
    initial_stream_window_size: 262144
    initial_connection_window_size: 1048576
    max_frame_size: 131072
    max_concurrent_streams: 50
    keep_alive_interval_secs: 20
    keep_alive_timeout_secs: 10
  memory_pool:
    buffer_size: 131072 # 128KB
    buffer_count: 2000

auth_server:
  enabled: true
  port: 443
  tls_cert: "/etc/ssl/certs/auth.crt"
  tls_key: "/etc/ssl/private/auth.key"
  tls_auto_reload: true
  tls_reload_check_interval_seconds: 21600 # 6 hours
  session_ttl_hours: 8
  users_db_path: "data/users.duckdb"
  encryption_key_path: "/etc/boilstream/encryption.key"
  superadmin_password_path: null # For automated deployments
  superadmin_mfa_secret_path: null # Base32 TOTP secret
  app_domain: "app.example.com"
  webauthn_rp_id: "app.example.com"
  webauthn_rp_origin: "https://app.example.com"
  cors_allowed_origins: []
  grants_db_path: "data/grants.duckdb"
  default_user_role: "reader"
  admin_username: null
  users_backup_backend: "primary-s3"
  users_backup_interval_seconds: 60
  users_backup_path: "auth/users.duckdb"
  superadmin_backup_backend: null # Defaults to users_backup_backend
  superadmin_backup_interval_seconds: 60
  superadmin_backup_path: "auth/superadmin.duckdb"
  email_encryption_pgp_public_key_path: null
  email_encryption_pgp_public_key: null
  enable_ducklake: true
  auto_create_default_ducklake: true
  enable_hydration: true
  hydration:
    max_admitted_bytes: 2147483648
    max_queue_depth: 1000
    per_ducklake_quota_bytes: 1073741824
    worker_count: 2
    max_retries: 3

oauth_providers:
  github:
    client_id: "github-client-id"
    client_secret: "github-client-secret"
    redirect_uri: "https://app.example.com/auth/callback"
    allowed_orgs: ["mycompany"]
    team_role_mappings:
      "mycompany/admins": "admin"
      "mycompany/engineers": "write"
      "mycompany/analysts": "read"
    audit_org_teams: false

  google:
    client_id: "google-client-id.apps.googleusercontent.com"
    client_secret: "google-client-secret"
    redirect_uri: "https://app.example.com/auth/callback"
    allowed_domains: ["mycompany.com"]

  oauth2:
    - name: "custom-provider"
      client_id: "client-id"
      client_secret: "client-secret"
      auth_url: "https://provider.com/oauth/authorize"
      token_url: "https://provider.com/oauth/token"
      userinfo_url: "https://provider.com/oauth/userinfo"
      redirect_uri: "https://app.example.com/auth/callback"
      scopes: ["openid", "profile", "email"]

  saml:
    - name: "entra-id"
      enabled: true
      sp_entity_id: "https://app.example.com"
      sp_acs_url: "https://app.example.com/auth/saml/acs"
      sp_slo_url: "https://app.example.com/auth/saml/logout"
      idp_entity_id: "https://sts.windows.net/{tenant-id}/"
      idp_sso_url: "https://login.microsoftonline.com/{tenant-id}/saml2"
      idp_slo_url: null
      idp_certificate: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      sp_certificate: "/etc/ssl/saml/sp.crt"
      sp_private_key: "/etc/ssl/saml/sp.key"
      attribute_mappings:
        email: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"
        username: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
        groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups"
        first_name: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname"
        last_name: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname"
        display_name: "http://schemas.microsoft.com/identity/claims/displayname"

audit:
  enabled: true
  backend: ndjson # ndjson, duckdb, s3, disabled
  log_path: "/var/log/boilstream/audit.ndjson"
  buffer_size: 1000
  sample_rate: null # 0.0-1.0, null = no sampling
  include_query_text: true
  include_ip_address: true

cluster_mode:
  enabled: false
  node_id_path: "./node_id"
  advertised_host: "node1.example.com"
  internal_api_port: 8443
  leader_heartbeat_interval_secs: 30
  leader_stale_threshold_secs: 120
  broker_heartbeat_interval_secs: 30
  broker_stale_threshold_secs: 120