Medallion Data Architecture
Your Bronze, Silver, and Gold layers move data forward whether it's correct or not. TestGen and DataOps Observability put a gate at every layer transition and a live view across the whole pipeline, so broken data stops at the break.
Get real test coverage on every table with TestGen
A 100-table Bronze layer needs about 2,200 tests. Silver needs another 2,200. Gold adds 620 plus custom business-metric tests. Writing that by hand is 14 months of full-time work, so most teams skip it and hope. TestGen profiles each table, generates freshness, volume, schema, and drift tests in minutes, and re-baselines the suite when you move from Bronze to Silver to Gold so the tests match the schema at that layer. A junior operator can run it. You get column-level coverage across every layer on day one, instead of the three tests on the dashboard table that everybody knows about.
Add tripwires to your Medallion DAG
Your job finished green. The number was wrong. A tripwire is a task that sits between two layer transforms, runs the TestGen suite for that layer, and exits non-zero if anything fails. Your orchestrator treats it like any other failed task. The next transform does not run. Your production tables stay on the last good data. You get a failed task in the run UI, a log of every test result, tags on the failing table in your catalog, and a deep link back to the TestGen run. Bad data stops at the layer where it broke, instead of showing up in a VP's dashboard at 9am.
See across Bronze, Silver, and Gold with Observability
Test results from one layer do not tell you where the break actually happened. DataOps Observability watches every tool in the pipeline: ingestion, Spark, dbt, Airflow, BI. It ties task runs, test results, timing, and error logs onto a single Data Journey that spans all three layers. When a Gold report looks off, you see which step failed, in which layer, and which test caught it. That is the difference between a lineage guess and knowing. Lineage maps where a problem would spread. A Data Journey shows where one actually is.
Profile and screen new data before it leaves Bronze
Bronze is where bad data enters. Wrong format, missing rows, a schema change nobody mentioned. TestGen profiles each incoming dataset column by column, flags null rates, duplicates, value ranges, and format mismatches, and gives you a hygiene review before it moves to Silver. Four baseline checks run on every refresh: freshness (did we get something new), volume (did we get enough), schema (does it fit), and drift (does it make sense). Problems surface at the source, where they cost one dollar a record to fix instead of one hundred.
Score data quality at each layer and compare them
You can't improve what you don't measure. TestGen gives every table in every layer a data quality score on completeness, freshness, schema conformity, and drift. Compare Bronze to Silver to see whether your cleansing actually cleansed anything. Compare Silver to Gold to catch aggregation bugs and business-logic drift. The scores roll up to a dashboard the whole team can look at, so data quality stops being a vibe and starts being a number.
See TestGen run against your Medallion
We'll profile a real table, generate the test suite, and show you where tripwires and observability slot into your pipeline.
Frequently Asked Questions
Common questions about Medallion Data Architecture
What is a Medallion architecture?
A Medallion architecture organizes a data lakehouse into three layers. Bronze holds raw data as it landed. Silver holds cleaned, conformed, joined data. Gold holds the aggregated tables feeding reports and models. Data moves upward, which means an error in bronze reaches gold unless something between them catches it.
Where do you test in a Medallion architecture?
Every layer, because each fails differently. Bronze needs arrival, freshness, and volume checks. Silver needs record count reconciliation and referential integrity after joins. Gold needs the business rules behind the numbers people report. A single check at the end catches almost none of it.
What is a data production tripwire?
A tripwire is a test that runs between two transformation steps and exits non-zero on failure, so your orchestrator stops the next transform. That is the difference between learning your gold table is wrong and preventing it from being written. Production keeps serving the last good data while somebody investigates.
How much testing does each Medallion layer need?
More than a team can hand-author, which is the practical obstacle. A bronze layer of 100 tables averaging 10 columns needs roughly 200 table-level and 2,000 column-level tests before silver and gold add their own. Generating the mechanical majority from profiling is what makes the target reachable.
Does this work on Databricks and Snowflake?
Yes. Databricks SQL and Snowflake are both supported, along with Azure Synapse Analytics, Azure SQL Database, Microsoft SQL Server, Google BigQuery, Amazon Redshift, Amazon Aurora PostgreSQL, Oracle Database 12c and later, SAP HANA, PostgreSQL, and Salesforce Data 360. Tests run as SQL inside the platform holding the data, so the same approach covers a lakehouse and the warehouse sitting next to it.
What happens when a test fails mid-pipeline?
The tripwire task exits with a non-zero code, the orchestrator halts the DAG, and the downstream transform never runs. You get the failing test, the table and column, and the failing rows. The dashboard is never wrong, because the bad data never reached it.