A Parquet partitioning strategy is the plan for how you organize SCADA telemetry files in a data lake so that queries stay fast and cheap as the data grows into billions of rows. Parquet is a columnar file format, which already makes analytical queries efficient, but how you split the data into folders and files decides whether an engine reads a few relevant files or scans the entire history. The two recurring problems are choosing partitions that let queries skip most of the data and avoiding the flood of tiny files that high-frequency writes create. This page covers partitioning by date and site, the small-files problem and compaction, and the column layout that lets query engines prune scans.
Parquet partitioning for SCADA data lakes in one line: A Parquet partitioning strategy organizes telemetry files into a folder hierarchy, commonly by date and site, so a query engine reads only the partitions it needs and skips the rest. Combined with columnar storage, sensible file sizes, and periodic compaction of small files, it keeps queries over huge SCADA histories fast and inexpensive.
Partitioning means physically splitting the data into separate folders keyed on a column's value, so that a query filtering on that column can ignore every folder that cannot contain a match. For time-series SCADA data the natural first partition is the date, because almost every query is bounded in time: a trend for last week, a report for last month, an investigation of a specific day. Laying the data out as folders by year, month, and day means a query for a single day touches one day's folder and never opens the other years sitting in the lake.
Site, or another coarse asset grouping, is the usual second partition key because many queries are also scoped to one location. A partition path that nests site under date, or date under site, lets a query for one site on one day narrow to a single small slice of the lake. The engine reads the partition values straight from the folder names without opening any files, so this pruning is nearly free. The art is choosing partition keys that match how the data is actually queried, since a partition on a column nobody filters by adds folders without ever saving a scan.
The trap at the other extreme is partitioning too finely. Splitting by hour, or by individual tag, or by site and hour and tag together, produces an explosion of partitions each holding very little data, and that fine granularity is what breeds the small-files problem. The right grain leaves each partition holding a substantial amount of data, enough to fill files of a healthy size, while still letting the common query filters prune away everything irrelevant. Date and site together usually hit that balance for SCADA telemetry.
High-frequency telemetry ingestion tends to write many small files, because each micro-batch or streaming flush lands its handful of rows as a new file. Left unchecked, a single day's partition can accumulate thousands of tiny Parquet files, and this is genuinely expensive to query. Every file carries per-file overhead, a footer to read, metadata to parse, an object-store request to make, and when the engine must open thousands of files to answer one query, that overhead dominates and swamps the efficiency Parquet was supposed to provide. Small files also waste Parquet's compression and encoding, which work best over larger row groups.
The remedy is compaction: a periodic job that reads the many small files in a partition and rewrites them as a few large ones, typically targeting files in the hundreds of megabytes range so that each holds sizable, well-compressed row groups. Compaction is usually run on a schedule against partitions that are no longer receiving live writes, for example compacting yesterday's data overnight, so it never fights the ingestion stream. After compaction the same data occupies far fewer, larger files, and queries that previously opened thousands of files now open a handful.
Because the small-files problem is a direct consequence of streaming ingestion, the pattern that works is to separate the write path from the read path. Ingestion optimizes for getting data landed quickly and durably, accepting small files as a by-product, while a downstream compaction step optimizes the layout for querying. Some table formats layered over Parquet automate this bookkeeping, tracking which files make up a partition and swapping the small ones for compacted ones atomically, so readers never see a half-compacted state. Either way, planning for compaction from the start avoids the slow degradation that creeps up on an un-maintained lake.
Partitioning prunes at the folder level, but Parquet also prunes inside each file, and a good column layout makes the most of that. Because Parquet is columnar, a query that selects only a few columns reads only those columns from disk and skips the rest entirely, so a wide telemetry table where a query needs just timestamp and value never pays to read the columns it ignored. This is why storing telemetry in a columnar format beats a row format for analytics, where queries typically touch a few columns across many rows rather than whole rows.
Parquet goes further with row-group statistics. Each file is divided into row groups, and the footer records the minimum and maximum value of each column in each group, so an engine filtering on a range can skip whole row groups whose min-max window cannot contain a match. If the data within each file is sorted or clustered by a commonly filtered column, such as timestamp, these min-max ranges become tight and non-overlapping, and the engine can skip most of the file for a narrow time filter. Writing data already ordered by timestamp within each partition therefore pays off at query time.
For a cloud SCADA service like Merobix storing long histories from many sites, this layered pruning, partitions by date and site, columnar reads of just the needed fields, and row-group skipping on sorted timestamps, is what keeps analytical queries affordable as the archive grows without bound. A well-partitioned, compacted, time-sorted Parquet lake lets an engine answer a question about one tag at one site over one week by touching a tiny fraction of the stored bytes, rather than scanning years of unrelated telemetry, which is the difference between a data lake that stays fast and one that slows to a crawl as it fills.
Partition by the columns your queries most often filter on, which for time-series SCADA data is usually date and site or asset group. Date is almost always a filter because queries are bounded in time, and site is a common second key because analyses are often scoped to one location. Avoid partitioning by very high-cardinality columns like individual tags, which creates far too many tiny partitions.
Streaming ingestion tends to write many tiny files, one per micro-batch, and each file carries fixed overhead to open and read. When a partition holds thousands of small files, that overhead dominates query time and undermines Parquet's compression. The fix is compaction, a periodic job that rewrites the small files into a few large ones sized in the hundreds of megabytes.
Being columnar, Parquet reads only the columns a query selects and ignores the rest. Within each file it stores min and max statistics per row group, so a range filter can skip entire row groups that cannot contain a match. Sorting data by a commonly filtered column like timestamp within each partition tightens those ranges and lets the engine skip most of a file for a narrow query.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.