On Google Cloud, the path from field telemetry to warehouse analytics usually runs through Pub/Sub and lands in BigQuery, a serverless warehouse that scales query compute independently of storage. The appeal for SCADA data is that BigQuery can hold enormous tag histories and still answer time-and-tag queries quickly if the tables are laid out well. This page covers how tag readings travel from the field through Pub/Sub into BigQuery, how time partitioning and clustering keep those queries efficient, and how to weigh the cost of streaming freshness against cheaper batch loading.
SCADA to BigQuery pipeline in one line: A SCADA to BigQuery pipeline is a data path that moves SCADA tag readings into Google BigQuery for warehouse-scale analytics, typically by publishing telemetry to Pub/Sub and then loading it into BigQuery through the Storage Write API or a Dataflow job. Tables are usually partitioned by time and clustered on the tag identifier so that queries filtered to a time range and specific tags scan only the relevant data. The design balances the freshness of streaming inserts against the lower cost of batch loading, with deduplication handling late-arriving telemetry.
The front of a Google Cloud telemetry pipeline is usually Pub/Sub, a managed messaging service that decouples the field from the warehouse. An edge gateway or integration service publishes tag readings as messages to a Pub/Sub topic, and Pub/Sub buffers them durably so that a downstream consumer can read at its own pace without the publisher having to match the consumer's speed. This buffering absorbs the bursty nature of SCADA polling and means a slow or briefly offline loading process does not lose telemetry, because the messages wait in the subscription until they are pulled.
From Pub/Sub, there are a couple of common ways into BigQuery. Dataflow, Google's managed stream and batch processing service, can run a pipeline that reads from the subscription, transforms and validates the readings, and writes them into BigQuery, which is the flexible option when the data needs reshaping, enrichment, or windowed aggregation on the way in. For simpler paths, BigQuery subscriptions can send Pub/Sub messages more directly into a table, or a service can write straight through the Storage Write API. The Storage Write API is the modern high-throughput ingestion interface, supporting both streaming and batch commit semantics, and it is the recommended way to get high volumes of rows into BigQuery efficiently rather than the older row-by-row insert path.
Which route to use depends on how much processing the telemetry needs before it lands. If the readings arrive already clean and consistently shaped, writing them through the Storage Write API with minimal transformation keeps the pipeline lean. If they need parsing, unit conversion, deduplication, or joining against reference data, a Dataflow job earns its place by doing that work in the stream. Many pipelines combine the two, using a lightweight write path for well-formed telemetry and reserving heavier processing for the sources that need it.
How the BigQuery table is defined largely determines how fast and how cheap tag queries will be, because BigQuery bills queries by the volume of data scanned. Time partitioning splits a table into segments by a date or timestamp, so a table of tag readings partitioned on the reading timestamp stores each day or hour in its own partition. A query filtered to a specific time window then scans only the partitions that overlap that window and skips the rest entirely, which for time-series telemetry is the single most important optimization, because most analytical queries are naturally bounded in time.
Clustering complements partitioning by ordering the data within each partition on chosen columns, and for tag data the natural clustering key is the tag identifier, sometimes with the site as an additional key. When a table is clustered on the tag, a query that filters to a particular tag or a small set of tags scans only the relevant blocks within each time partition rather than the whole partition. The combination is powerful: partition by time to bound the scan to a date range, cluster by tag to bound it further to the tags of interest, and a query over one sensor's last week touches a tiny fraction of a table that might hold years of the whole plant.
The layout has to match the query patterns to pay off, so it is worth designing around how the data will actually be asked for. Partitioning on the reading timestamp assumes queries filter on that timestamp, and clustering on the tag assumes queries filter on the tag, both of which are the common cases for telemetry. Where analysts routinely slice by site as well, adding site to the clustering keys helps. The goal throughout is to arrange the physical layout so the warehouse can prune away the data a query does not need, which keeps both the response time and the per-query cost proportional to the answer rather than to the size of the whole table.
Freshness in BigQuery is not free, and the streaming-versus-batch decision is largely a cost decision. Streaming ingestion, through the Storage Write API's streaming path, makes rows queryable within seconds and is the right choice for tags that need to be current in the warehouse, but it is metered on the volume streamed. Batch loading, where rows are collected into files or larger commits and loaded periodically, is far cheaper per unit of data and is the sensible default for the bulk of history that does not need second-level freshness. A pipeline that streams everything when most of it could be batched is a common and avoidable source of cost.
The pragmatic pattern mirrors the ingest question elsewhere: stream the small set of tags that genuinely need to be current for live dashboards or alerting, and batch the rest. Because SCADA telemetry is dominated by a long tail of tags that are only ever looked at in reports, moving that tail to batch loading cuts cost substantially without hurting any real use. Deciding tag by tag, or tag-group by tag-group, which readings need streaming freshness is where the savings come from.
Late and duplicate telemetry is the other thing the pipeline must handle, because field data does not always arrive in order or exactly once. A gateway that reconnects after a network drop may replay readings, and a retry in the pipeline may deliver the same reading twice, so BigQuery tables usually need deduplication on a stable key such as tag plus timestamp. This is often done downstream, loading raw readings into a landing table and then deduplicating into a clean table, or by using merge logic that ignores a reading already present. Handling late arrivals means the clean tables reflect what the field actually measured rather than the accidents of network timing, which is exactly what analysts querying the warehouse need to be able to trust.
Telemetry is usually published to Pub/Sub, which durably buffers the messages, and then loaded into BigQuery either through a Dataflow job that transforms and validates the readings, through a BigQuery subscription, or directly via the Storage Write API. The Storage Write API is the modern high-throughput ingestion path supporting both streaming and batch semantics. Dataflow earns its place when the telemetry needs parsing, enrichment, or aggregation on the way in.
BigQuery bills queries by the volume scanned, so layout controls both speed and cost. Partitioning the table by the reading timestamp lets a time-bounded query scan only the overlapping partitions, which is the biggest win for time-series telemetry. Clustering on the tag identifier orders data within each partition so a query for a few tags reads only the relevant blocks, and the combination keeps a query over one sensor's week touching a tiny fraction of a table holding years of history.
Stream only the tags that genuinely need to be current in the warehouse for live dashboards or alerting, since streaming makes rows queryable within seconds but is metered on volume. Batch load the bulk of history, which is far cheaper per unit and fine for the long tail of tags only seen in reports. Because SCADA telemetry is dominated by rarely queried tags, moving that tail to batch cuts cost substantially without hurting any real use.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.