Skip to content
BoilStream

Analyze SCADA data

concept
TL;DR

Use BoilStream as an ingestion and analytical component behind an industrial gateway. Transform events, query recent rows and retain published history in object storage. Define network, permission and recovery boundaries before rollout. BoilStream does not replace the safety, deterministic control or real-time functions of PLC, DCS or SCADA systems.

ArchitectureExport observations without putting analytics in the control loop
Export observations without putting analytics in the control loopThe control system continues its own operational loop. An exporter copies observations to BoilStream, where SQL clients analyze them. No command path from analytics back to the controller is shown or required.Operational systemControllerOwn control loopExporterCopies observationsBoilStreamStores exported eventsAnalytics clientSQL over observationsObserveExport rowsSQL / resultsExport observations without putting analytics in the control loopThe control system continues its own operational loop. An exporter copies observations to BoilStream, where SQL clients analyze them. No command path from analytics back to the controller is shown or required.Operational systemControllerOwn control loopExporterCopies observationsBoilStreamStores exportedeventsAnalytics clientSQL over observationsObserveExport rowsSQL / results
This example is an analytics export path. Keep control, safety and actuation responsibilities in the operational system.

BoilStream can move operational data from industrial networks into durable object storage while keeping SQL-based transformation and recent-data queries close to the ingestion path.

Where it fits

Section titled “Where it fits”
Output
Sensors and PLCs -> SCADA or edge gateway -> BoilStream -> Parquet on object storage
                                            |            -> managed DuckLake
                                            +-> streaming views and hot-tier queries

BoilStream is useful when a deployment needs to:

  • send Arrow or Kafka-compatible data through a controlled network boundary;
  • transform and aggregate streams with SQL;
  • retain analytical Parquet files in S3-compatible or cloud object storage;
  • query recent data through PostgreSQL-compatible or Arrow interfaces; and
  • make historical data available through managed DuckLake catalogs.

It does not replace the safety, control, or deterministic real-time functions of PLC, DCS, or SCADA control systems. Treat it as an ingestion and analytical data component.

Interfaces

Section titled “Interfaces”

Choose an interface based on the producer and operational boundary:

Producer Interface Payload
Existing Kafka producer Kafka protocol Confluent Avro
Arrow-capable service FlightRPC Arrow record batches
Browser or HTTP service HTTP/2 ingestion Arrow IPC
SQL client or BI tool PostgreSQL or FlightSQL SQL queries
Native analytical client DuckDB with the BoilStream extension Managed DuckLake

All public interfaces should use TLS and their documented authentication method. Keep object-storage credentials inside the BoilStream deployment.

Storage and recovery

Section titled “Storage and recovery”

Bulk analytical data is written as Parquet to object storage. Cluster metadata uses quorum replication and completed S3 backups. Hydrated files are replaceable from their source objects; unpublished local hot rows have a separate durability boundary. See Reliability and completion and Cold-tier hydration for the supported boundaries.

Evaluate with representative data

Section titled “Evaluate with representative data”

Throughput, visibility latency, compression, and storage cost depend on the source schema, batch size, transformations, durability settings, hardware, network, and object store. A production evaluation should use representative tag distributions and query patterns and should include:

  • sustained ingestion plus expected bursts;
  • streaming-view and query workloads running together;
  • loss and restoration of a cluster node;
  • temporary object-storage and network failures;
  • authorization and tenant-isolation checks; and
  • complete local-state loss followed by S3-only recovery.

Record the tested release, topology, workload, durability settings, and acceptance criteria before using results for sizing or service objectives.

Next steps

Section titled “Next steps”
  1. Select the source interface and define the Arrow or Avro schema.
  2. Choose the streaming DuckLake and object-storage location.
  3. Define required transformations, retention, and query paths.
  4. Run the representative evaluation above.
  5. Document operational ownership, monitoring, recovery, and rollback procedures.