On-Demand Webinar · 42 min

Data Quality in a Medallion Architecture

Bronze, Silver, and Gold each fail differently, so each needs its own checks. Chris Bergh on where data quality tests belong in a Medallion architecture, plus regression testing in development and scores you can hand to someone else.

Presented by Chris Bergh

What you'll learn 6 points
  • The three layers do different jobs, which is why one quality gate does not cover them. Bronze is the raw landing zone, Silver applies just enough cleaning to produce a unified view of core business entities, and Gold holds refined, aggregated, analysis-ready data in project-specific schemas.
  • The naming is not standard. Some organizations add a Platinum layer after Gold, others call the same three layers L1, L2, and L3 — and Medallion is becoming very popular inside the data lakehouse.
  • Each layer has its own failure mode, so the session walks the challenges and then the tests separately for Bronze, Silver, and Gold rather than treating quality as a single checkpoint.
  • This is not a Medallion-only problem. Traditional raw/staging/warehouse/mart stacks, Kappa and Lambda streaming architectures, data mesh, and plain data lakes all need data quality testing too.
  • Development needs its own testing. Regression testing across the Medallion layers is how you find out you did not break production before you deploy, rather than after.
  • The practices that hold the whole thing together: find data errors before your customers do, find them as early in the processing as possible, automate the checks instead of doing them by hand, count your errors, and start with a data quality circle in a no-blame, no-shame culture.

Prefer to read it? The written version is in Webinar: Data Quality in a Medallion Architecture – 2024.

Slides

42 slides

Questions from this session

What is a medallion architecture?

A medallion architecture organizes data into three layers. Bronze is the initial landing zone for incoming raw data, captured in its unprocessed original form. Silver applies just enough cleaning and transformation to give a unified, enterprise-wide view of core business entities. Gold holds fully refined, aggregated, analysis-ready data, usually in project-specific schemas built for BI and advanced analytics. Some organizations add a Platinum layer after Gold, and some call the layers L1, L2 and L3.

Where should data quality tests go in a medallion architecture?

At every layer, because each one fails differently. Bronze needs tests on the raw data as it lands, Silver needs tests on the cleaned and unified entities, and Gold needs tests on the aggregated, analysis-ready output that customers see. Testing only at the end means a customer finds the error, which is the outcome the whole architecture is supposed to prevent.

Do other data architectures need the same testing?

Yes. Traditional raw and staging, warehouse, and mart stacks need it, as do Kappa and Lambda streaming architectures, data mesh and domain-driven designs, and plain data lakes. Medallion is becoming popular in the data lakehouse, but the requirement is not specific to it: every architecture that moves data through stages needs quality testing at those stages.

How does TestGen generate data quality tests?

TestGen runs a five-step workflow. It profiles tables against 51 profiling characteristics to build a baseline, screens for hygiene problems with 27 hygiene tests, generates tests from 32 auto-selected test types plus two custom types and eight fill-in-the-blank multi-column business rule types, executes them on a schedule or trigger, and then supports review and refinement so parameters can be tweaked and irrelevant tests muted. Execution uses database queries directly: 1,000 tests run in under three minutes and 15,000 in under 20 minutes.

Why do you need regression testing as well as production data quality tests?

Production tests tell you the data arriving today is wrong. Regression testing tells you the change you are about to deploy would break something, before it reaches customers. In a medallion setup that means running the same tests against a development environment with test data, so a developer can confirm the SQL, model or report change did not break a downstream layer, then push safely.

What are the five pillars of data observability?

Across the steps checks runs, order of operations and schedule, for process reliability. Down the stack monitors metrics, logs and cost, for technology status. Data at rest validates data quality automatically with business domain tests such as schema, freshness and volume. Data in use tests the results of models, visualizations and delivery, such as dashboard values or a model's root mean square error. Set expectations compares expected against reality, alerts and analyzes.

Where to go next