A lakehouse tries to give you the cheap, flexible storage of a data lake with the reliability and query behavior of a warehouse, and Databricks built its platform around that idea using Delta Lake as the storage layer. For SCADA telemetry, this is appealing because raw tag data is voluminous and messy at the point of arrival but needs to become clean and query-ready for analytics. This page covers how tag data lands in a Databricks lakehouse, whether through file drops picked up by Auto Loader or a continuous stream, and how the raw readings are refined in stages into tables that analysts and models can actually use.
SCADA to Databricks ingestion in one line: SCADA to Databricks lakehouse ingestion is the practice of landing SCADA telemetry into a Databricks lakehouse built on Delta Lake, then refining it into analytics-ready tables. Data arrives either as files dropped into cloud storage that Auto Loader incrementally picks up, or as a continuous structured stream read from a source like Event Hubs or Kafka. The raw readings are written into a bronze layer, cleaned and standardized into a silver layer, and aggregated into a gold layer, with Unity Catalog governing access across the tables.
There are two main on-ramps for telemetry into a Databricks lakehouse, and both are built on Spark structured streaming underneath. The first is Auto Loader, which is designed to incrementally ingest files as they arrive in cloud storage. An edge collector or an upstream pipeline drops tag readings as JSON or Parquet files into a storage location, and Auto Loader keeps track of which files it has already processed so that each run picks up only the new ones without rescanning the whole directory. This is the natural fit when telemetry is being landed as files, for instance from an Event Hubs Capture output or a scheduled export, because Auto Loader turns a growing pile of files into a reliable, exactly-once ingestion stream.
The second on-ramp is reading directly from a streaming source with structured streaming. Databricks can consume from Event Hubs, Kafka, or similar message systems as a streaming DataFrame, so tag changes flow into the lakehouse continuously rather than in file batches. This lowers latency compared to waiting for files to accumulate, and it suits pipelines where telemetry is already on a message bus. The two approaches are not mutually exclusive: a plant might stream a small set of important tags directly for low latency while letting the bulk of history arrive as files that Auto Loader sweeps up on a schedule.
Both paths handle the awkward reality that telemetry schemas drift. Auto Loader has schema inference and evolution features so that when a device starts sending a new field, the pipeline can pick it up and add the column rather than failing, and structured streaming can be written to tolerate new fields similarly. Because SCADA sources change over time as firmware and tags are added, an ingestion layer that adapts to new columns instead of breaking is a practical necessity rather than a nicety, and it is one of the reasons these managed ingestion features exist.
The lakehouse convention for turning raw telemetry into something usable is a layered progression often called medallion architecture. The bronze layer holds the raw readings essentially as they arrived, appended without much transformation, which preserves an unaltered record and lets the pipeline reprocess later stages if logic changes. For SCADA data this bronze table is effectively a raw historian of everything the field sent, warts and all, including duplicate readings, odd quality flags, and whatever inconsistencies the sources produced, kept as the source of truth from which cleaner layers are derived.
The silver layer is where the data becomes trustworthy. Here the pipeline cleans and standardizes: deduplicating on a tag-and-timestamp key, filtering out bad-quality readings or flagging them, normalizing tag names to a consistent scheme, casting values to proper types, and resolving late-arriving records into the right time order. The result is a silver table of validated tag readings that downstream consumers can rely on without re-doing the cleanup themselves. Getting this layer right is where most of the engineering effort goes, because raw telemetry carries all the messiness of the field and turning it into clean, well-typed, deduplicated time series is the hard part of the job.
The gold layer serves the specific questions the business asks. It holds aggregated and shaped tables built for particular uses, such as hourly averages per asset, daily production rollups, or feature tables that feed a model, each derived from the clean silver readings. Because gold tables are purpose-built and much smaller than the raw stream, the dashboards and reports that read them stay fast, and analysts work against tidy tables rather than wrestling raw telemetry. This staged refinement, raw to clean to aggregated, is what lets a lakehouse hold both the complete unaltered record and the polished tables that day-to-day analytics actually query.
Operational data brings governance requirements, and Unity Catalog is Databricks' answer to who can see what across the lakehouse. It provides a central place to define catalogs, schemas, and tables with access controls and lineage, so that raw plant telemetry, cleaned readings, and aggregated views can each have appropriate permissions. This matters in an industrial context because raw operational data may be sensitive or restricted, and being able to grant analysts access to gold aggregates while limiting who touches the raw bronze layer keeps the data usable without making it a free-for-all. Lineage tracking also helps trace a number in a report back through the layers to the raw reading it came from.
It is worth being clear that a lakehouse complements a plant's real-time systems rather than replacing them. Databricks is built for analytics at scale, batch and streaming, over large volumes of history, and it is not the system operators watch to run the plant second by second. The lakehouse answers the fleet-wide and historical questions, comparing sites, training models, spotting long-term drift, while live operational control and monitoring stay in the SCADA and monitoring layers where they belong. Treating the lakehouse as the analytics destination rather than the control system keeps each part of the stack focused on what it does well.
The quality of what lands in bronze depends heavily on the collection layer feeding it, which is where a monitoring platform helps. A platform such as Merobix can gather SCADA tags across mixed field protocols, apply consistent naming and carry quality and timestamps, and deliver clean, consistently formatted telemetry into storage or a message bus that Databricks ingests. When the raw layer receives well-shaped data from a single collection layer rather than a scramble of device-specific formats, the silver cleanup is far simpler, and operators keep a live plant view in the monitoring layer while the lakehouse handles the heavy analytics.
Auto Loader incrementally ingests files as they arrive in cloud storage, tracking which files it has already processed so each run picks up only new ones, which fits telemetry landed as JSON or Parquet drops. Structured streaming reads directly from a message source like Event Hubs or Kafka as a continuous stream, lowering latency for telemetry already on a bus. Many pipelines use both, streaming a few important tags while sweeping the bulk of history from files.
Bronze holds raw readings essentially as they arrived, preserving an unaltered record you can reprocess from later. Silver is the cleaned layer, where readings are deduplicated, bad-quality values are filtered or flagged, tag names are normalized, and values are typed and time-ordered. Gold holds aggregated, purpose-built tables like hourly averages or feature sets that dashboards and models query, kept small and fast because the heavy cleanup already happened upstream.
Not for live operations. Databricks is built for large-scale batch and streaming analytics over history, answering fleet-wide and long-term questions, while operators run the plant second by second from the SCADA and monitoring layers. The lakehouse is best treated as the analytics destination that complements those systems, comparing sites and training models, rather than as the control or real-time monitoring system.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.