What Is a Data Lake?

Back to Blog

Schema-on-Read: The Defining Idea

The core distinction of a data lake is schema-on-read versus schema-on-write. A traditional database or warehouse requires you to define the structure before you load data (schema-on-write) - every field typed and modeled up front. A data lake accepts data as-is: CSVs, JSON, sensor streams, images, PDFs, log files, all landed in cheap object storage (typically cloud storage like Amazon S3 or Azure Data Lake Storage) exactly as they arrived. You impose structure only when a specific analysis reads the data.

The advantage is flexibility and cost. You do not have to know in advance every question you will ask, and you can retain everything inexpensively for future use - including data you have no immediate use for. The risk is that a lake without governance, cataloging, and quality controls becomes a 'data swamp': vast, cheap, and unusable because nobody can find or trust what is in it.

Data Lake vs Data Warehouse vs Historian

A data warehouse stores cleaned, structured, modeled data optimized for business intelligence queries - schema-on-write, high governance, higher cost per terabyte. A data lake stores raw data of any type - schema-on-read, low cost, maximum flexibility, less inherent structure. The two are complementary, and modern architectures increasingly blend them into a 'lakehouse' that adds warehouse-like management and query performance over lake storage.

A time-series database or historian is different again: it is highly optimized for one data shape (timestamped tags) with fast trending and compression, but it is not where you would keep well logs, images, or unstructured records. A common pattern is that operational data lives in a historian for real-time and trending use, while a copy also flows into a data lake for cross-domain analytics and machine learning alongside non-time-series data.

Where Data Lakes Fit in Oil and Gas

Upstream operators generate far more than telemetry: drilling data, well files, seismic, maintenance logs, LOWIS/production reports, financial and regulatory records. A data lake gives a single, affordable place to consolidate all of it so data science teams can combine production telemetry with maintenance history to predict failures, or correlate operating conditions with outcomes across the fleet - analyses that no single operational system holds all the data for.

For most field operators, the practical relationship is that SCADA and the historian handle real-time monitoring and process history, while a data lake is an enterprise analytics destination that the operational data feeds into. Merobix focuses on the operational layer - live monitoring and time-series history of your assets - and its data can be exported to feed an enterprise data lake, rather than replacing it.

Getting SCADA Data Into the Lake

The mechanics of feeding a lake from operations are mostly about discipline at export time. Time-series data should land with timestamps in UTC and the original quality flags carried along, because an analyst two years later cannot reconstruct whether a flat-lined tag was a real plateau or a stale value unless quality made the trip. The same goes for units and asset context: a stream of numbers keyed by raw tag ID is unusable outside the SCADA team, so exports should carry engineering units and the asset hierarchy, or a mapping table should live in the lake beside the data. The reasoning behind UTC vs local time storage applies doubly once data leaves the system that created it.

Transport is either scheduled batch export - files of historized values landed on an interval - or a streaming path that forwards values continuously. Batch is simpler and fits most analytics, which run on historical windows anyway; streaming earns its complexity only when a lake-side consumer genuinely needs fresh data. A dedicated pattern for this is described in the SCADA-to-cloud data lake pipeline reference.

Zones: Raw, Curated, Consumption

Mature lakes are layered, not flat. A raw zone holds data exactly as received, immutable, so there is always a ground truth to reprocess from. A curated zone holds validated, standardized copies: units converted, tag names mapped to assets, bad-quality values flagged or filtered by documented rules. A consumption zone holds analysis-ready datasets shaped for specific uses - a well performance table, a compressor health set. Each zone answers a different question when a result looks wrong: was the source bad, the cleaning bad, or the analysis bad?

The governance that keeps this structure honest is unglamorous: a catalog that says what each dataset is, who owns it, and how fresh it is; access control by zone; and automated checks that reject files with missing columns or impossible timestamps at the raw-to-curated boundary instead of letting them flow downstream.

Questions to Settle Before Building One

A short list worth answering in writing before any storage is provisioned:

1. Which analytics use cases justify the lake, and what data do the first two actually need?
2. Which historian tags flow in - at raw resolution or after downsampling - and who decides?
3. How are units, tag naming, and asset hierarchy standardized across source systems?
4. Who owns the catalog and the raw-to-curated validation rules?
5. What are the retention and access rules, especially for data with commercial or regulatory sensitivity?

Lakes that start from a concrete use case and grow outward stay useful. Lakes that start as a mandate to centralize everything tend to fill with data nobody can interpret, which is exactly the swamp scenario.

Frequently Asked Questions