A historian is excellent at answering operational questions about tags over time, but it is not where data scientists want to train models or where the enterprise runs cross-domain analytics. For that work, organisations move their SCADA telemetry into a cloud data lake, where it can sit as cheap columnar files next to every other dataset the business owns. A SCADA to cloud data lake pipeline is the machinery that gets it there: reading telemetry from the SCADA side, writing it into object storage as Parquet, partitioning it so queries are efficient, and registering its schema so tools can read it. This guide covers those mechanics, the raw landing zone concept, and how a lake differs from a historian, focusing on how the pipeline is built rather than on data lakes in general.
SCADA to Cloud Data Lake Pipeline in one line: A SCADA to cloud data lake pipeline is the pipeline that moves raw and aggregated SCADA telemetry into a cloud data lake, meaning object storage holding the data as columnar files such as Parquet. The pipeline reads telemetry from the SCADA system or historian, writes it into object storage partitioned by dimensions like asset and time, sizes the files sensibly, and registers a schema so query and machine-learning tools can read it. It exists to make industrial data available for ML and enterprise analytics on the same footing as the rest of the organisation's data, which a historian, optimised for operational queries, is not designed to do.
At its core the pipeline takes telemetry from the SCADA side and lands it as files in cloud object storage. The format that has become standard for this is a columnar one such as Parquet, and the reason is a good fit with how the data is queried. Columnar files store each column together rather than each row, so an analytics query that reads only a few columns out of many can skip everything else, and they compress well because values within a column are similar. For telemetry, which is naturally a small number of columns - asset, timestamp, tag, value - repeated across enormous numbers of rows, columnar storage is both compact and fast to scan, which is exactly what a lake needs to serve large analytic and machine-learning workloads economically.
The pipeline usually carries two kinds of data, and it is worth landing both. Raw telemetry preserves the full-fidelity samples exactly as they were recorded, which is what a machine-learning model or a detailed investigation needs, while aggregated telemetry provides pre-rolled summaries that make routine analytics cheaper. Keeping the raw data in the lake, even though it is bulky, is deliberate: object storage is cheap enough to hold years of it, and having the original samples means any future question can be answered without going back to the source. The aggregates are a convenience layer on top, not a replacement for the raw record.
How the pipeline actually moves the data depends on the source. It might export in periodic batches, reading a window of history from the historian on a schedule and writing it out as files, or it might stream continuously, landing telemetry as it arrives. Batch export is simpler and fits the reality that much lake analytics is not time-critical, while streaming keeps the lake more current at the cost of more moving parts. Either way, the writing side has to be careful about how it groups readings into files, because the choice of file layout and size, covered next, largely determines how well the lake performs for the people querying it later.
Partitioning is how a data lake stays queryable at scale, and it means organising the files into a folder structure keyed on the dimensions people filter by. For SCADA telemetry the natural partitions are asset and time, so files are laid out by asset and by date, for example a folder per asset and within it a folder per day. When a query asks for one asset over one month, the query engine can then read only the folders for that asset and those days and ignore everything else, a technique called partition pruning. Without partitioning, every query has to scan the entire dataset, which is slow and expensive; with sensible partitioning, queries touch only the slice they need.
File sizing is the less obvious but equally important half of the layout problem. Object storage and query engines both perform badly when a dataset is split into a huge number of tiny files, because each file carries per-file overhead to open and read, and telemetry written continuously naturally tends to produce many small files. The pipeline should therefore aim for files that are large enough to read efficiently rather than a flood of tiny ones, which usually means buffering readings and writing them out in reasonably sized chunks, and sometimes running a compaction step that merges small files into larger ones after the fact. Balancing partition granularity against file size is a real design tension: partitioning too finely by time produces tiny files, so the two decisions have to be made together.
Schema registration is what lets tools actually read the files as tables rather than as opaque blobs. The pipeline records the columns and their types in a catalog or metadata store, so that a query engine, a notebook, or a BI tool can discover the dataset, understand its structure, and query it by name. This also gives the lake a place to manage schema over time, so that when a new tag or column appears, the catalog reflects it and consumers are not broken. Registering the schema turns a pile of Parquet files sitting in object storage into a discoverable, self-describing dataset that the wider organisation can use, which is the difference between a data lake and a data swamp.
The raw landing zone is a foundational idea in how these pipelines are structured. It is the area of the lake where incoming telemetry is written first, in as close to its original form as possible, before any cleaning or transformation. The point of landing raw data untouched is that it preserves the original record permanently and cheaply, so that later processing steps can be re-run, corrected, and improved without ever having lost the source. If cleaning logic turns out to have a bug, the raw landing zone still holds the true data to reprocess from. This raw-first discipline is what makes a lake trustworthy as a system of record rather than just a convenient copy.
It is worth being clear about how a data lake differs from a historian, because they are easy to conflate. A historian is a purpose-built operational store, schema-on-write and optimised for fast queries over tags and time, holding the data operations needs and enforcing its structure as data arrives. A data lake is schema-on-read and general-purpose: it holds all data, structured or not, in its raw form, and structure is applied when the data is read rather than when it is written. That makes a lake far more flexible for exploratory analytics and machine learning across many datasets, but it does not replace the historian's job of answering operational questions quickly, so the two coexist, with the lake serving analytics and the historian serving operations.
A cloud SCADA platform such as Merobix fits into this by being the source that feeds the pipeline cleanly. Because the platform already ingests and stores telemetry in a structured, queryable form, exporting raw and aggregated data out to a lake, partitioned by asset and time and registered in a catalog, is a natural extension rather than a separate integration project. For field operations that want their equipment and production telemetry available to data science and enterprise analytics alongside everything else the business owns, having the SCADA platform land that data in the lake in a well-partitioned, columnar, schema-registered form means the analytics teams get usable data without the operations team having to build and run the export pipeline themselves.
A historian is a purpose-built operational store that is schema-on-write and optimised for fast queries over tags and time, holding the data operations needs in a defined structure enforced as data arrives. A data lake is schema-on-read and general-purpose, holding all data in its raw form in object storage, with structure applied when the data is read rather than when it is written. The lake is far more flexible for exploratory analytics and machine learning across many datasets, but it does not replace the historian's job of answering operational questions quickly, so the two usually coexist.
Parquet is a columnar format, meaning it stores each column together rather than each row, which suits telemetry well because a query typically reads only a few columns out of many and can skip the rest. Columnar storage also compresses efficiently since values within a column are similar, and telemetry is a small number of columns repeated across enormous numbers of rows. The result is compact files that query engines and machine-learning tools can scan quickly and cheaply, which is what makes holding years of raw and aggregated telemetry in a lake economical.
It is the area of the lake where incoming telemetry is written first, in as close to its original form as possible, before any cleaning or transformation is applied. Landing raw data untouched preserves the original record permanently and cheaply, so later processing can be re-run or corrected without ever having lost the source. If a cleaning step turns out to have a bug, the raw landing zone still holds the true data to reprocess from, which is what makes the lake trustworthy as a system of record.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.