Open source data quality for OneLake
OneLake data quality for every lakehouse layer.
Bronze, silver, and gold: TestGen profiles and tests each layer. It reads your Lakehouse tables through the Fabric SQL analytics endpoint and generates thousands of tests. Read-only. Free and open source.
Install TestGen
Pick your platform. Free and open source.
# TestGen install for Mac, Linux.
# Download the latest installer.
$ curl -o dk-installer.py \
'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
# Run the install command.
$ python3 dk-installer.py tg install
# Full docs: docs.datakitchen.io/testgen/get-started/install-on-mac-linux/- Double-click
dk-installer.exein your Downloads folder. - If Windows shows a protection warning, choose More info, then Run anyway.
- Pick TestGen, then Docker or pip. The installer seeds demo data, opens the UI, and prints your login.
# Full docs: docs.datakitchen.io/testgen/get-started/install-on-windows/
- 55data profiling column characteristics
- 32data hygiene detectors
- 34test types auto-generated from the profile
- 19business rule and custom SQL test types
Connect TestGen to OneLake
TestGen reads OneLake tables through the Fabric SQL analytics endpoint, which is read-only by design. Point one table group at each layer and you get a quality score per layer.
- What gets tested
- Lakehouse and Warehouse tables in OneLake. Raw files in Azure Blob Storage (CSV, Parquet, JSON) can be tested in place too.
- Sign-in
- Entra ID service principal or Managed Identity.
- Access
- Contributor (or higher) on the workspace. No item-level grants needed.
- One score per layer
- Bronze catches bad input, silver catches broken transforms, gold catches what your reports would show.
Ask your Fabric admin for three things
- An Entra ID service principal (or a Managed Identity) for TestGen. SQL logins don’t work on the Fabric SQL analytics endpoint.
- Contributor (or higher) on the Fabric workspace that holds your Lakehouse or Warehouse.
- The tenant setting Service principals can call Fabric public APIs (formerly “Service principals can use Fabric APIs”) turned on in the Fabric admin portal.
Workspace-level access is enough; no item-level permissions. Details in the access requirements.
OneLake data problems TestGen catches, layer by layer
Catch it in bronze and silver never copies it. Each card lists the TestGen checks that find it.
-
Bronze: the file didn’t arrive
A feed is late or empty.
-
Bronze: the file isn’t shaped right
Numbers as text, odd blanks, placeholder values.
-
Silver: deduplication failed
Business keys that should be unique aren’t.
-
Silver: conformance broke
Codes and keys that no longer match the reference data.
-
Gold: totals drifted from silver
Aggregates that don’t reconcile to the layer below.
-
Gold: the metric jumped
A KPI that moved more than the business did.
What TestGen does in OneLake
Everything below is in the open source release, and all of it works on OneLake. Enterprise adds team access, security, Commercial Pharma tests, and support.
-
Auto-Generated Tests (documentation, opens in a new tab)
Point one table group at each medallion layer and TestGen writes the tests for bronze, silver and gold separately.
-
Data Profiling (documentation, opens in a new tab)
55 characteristics per column in every layer, so you can see where a problem entered: the file, the transform or the aggregate.
-
Hygiene Detection (documentation, opens in a new tab)
Catches bad input in bronze, before silver and gold copy it: numbers as text, odd blanks and placeholder values. Raw CSV, Parquet and JSON in Azure Blob Storage can be checked in place.
-
Anomaly Detection (documentation, opens in a new tab)
Freshness and volume monitors on bronze, schema monitors on silver, metric monitors on gold.
-
Data Catalog (documentation, opens in a new tab)
Every layer’s tables in one catalog, with profiles, test results and Critical Data Elements.
-
Quality Scoring (documentation, opens in a new tab)
One quality score per layer, so “is the lakehouse healthy?” has a number for bronze, silver and gold.
-
MCP Server (the TestGen MCP cheat sheet)
Ask an AI assistant which layer broke and see the rows behind it, through 112 MCP tools that run in your environment.
-
Open Source Core, UI Included
Unlimited tables and a full UI. The core data profiling, testing and monitoring engine is free under Apache 2.0, for one OneLake connection. Enterprise adds more connections, team access, security and support.
From DataKitchen on OneLake and the medallion architecture
-
Blog Testing Every Layer of the Medallion Architecture, Now with OneLake at the Center
DataOps TestGen 5.92.1 connects to Microsoft OneLake through the Fabric SQL analytics endpoint. Here is what Microsoft's current medallion guidance says, why…
- Answer
How do you profile, test and monitor data quality in Microsoft Fabric and OneLake with TestGen?
Connect TestGen to the Fabric SQL analytics endpoint with Entra ID, map medallion layers to table groups, and test Delta tables in place.
- Answer
How do you set up an Entra service principal so a tool can read your Fabric workspace?
Microsoft's steps to register an Entra app, create a secret, enable service principals in Fabric tenant settings, and add the principal to a workspace.
- Answer
How do you test data quality in a Microsoft Fabric Warehouse?
Test a Fabric Warehouse by connecting over its SQL connection string with Entra ID, running data profiling on every table, generating tests, and monitoring freshness…
-
Blog The Four Points in Your Medallion Architecture Where Data Testing Really Matters
Four places to test data in a Medallion: source, ingestion, production tripwires, and CI/CD regressions. Same null check, different job in each place. Skip a point…
-
Blog Data Quality Test Coverage In a Medallion Data Architecture
For data engineering teams serious about delivering production-grade data products, implementing systematic test coverage across their Medallion architecture…
Why teams choose TestGen
-
You Don't Have Time to Write Tests. TestGen Does It Automatically.
You're buried in customer requests. You have no time to write tests, let alone innovate. TestGen scans your data and generates the tests and anomaly detectors for you. No coding, no massive YAML configuration.
-
A lakehouse with no tests is three layers of guessing.
TestGen profiles each layer, writes the tests, and monitors freshness, volume, and schema, so a bad file in bronze doesn’t reach gold unnoticed.
See how TestGen works →
-
Runs inside OneLake
Tests run as read-only SQL where the data sits. Nothing is copied out, and nothing is written back.
-
Apache 2.0, core engine free
Every detector, every characteristic, and every test type but the 12 Commercial Pharma types. In the open source.
-
Self-hosted
Your data never leaves your environment.
-
Bootstrapped since 2013
Profitable, independent, and not pivoting next quarter.
Open Source with Reasonable Enterprise Pricing
Open source TestGen is free for one user, one project, and one database connection. Enterprise is a flat $100 per month per user and database connection, with unlimited tables and data volume. No per-table tax. A year of enterprise data quality costs roughly one data engineer's monthly salary.
See pricingTestGen versus the competition
Head-to-head comparisons with every major data quality and observability tool. Search for the one you are evaluating.
No vendors match. .
- Bristol Myers Squibb
- Eisai
- Catholic Relief Services
- Progeny Health
- X4
- KOA
- CNH
Questions
How do I test each layer of a OneLake medallion architecture?
Point one TestGen table group at each layer (bronze, silver and gold). Each gets its own profiling, tests and quality score, so you can see which layer a problem entered at.
Can TestGen test raw files before they reach a table?
Yes. TestGen reads CSV, Parquet and JSON files in place from Azure Blob Storage, as well as Lakehouse tables through the SQL analytics endpoint.
How does TestGen sign in to OneLake?
With an Entra ID service principal or a Managed Identity that has Contributor (or higher) on the Fabric workspace. SQL logins are not supported on the SQL analytics endpoint.
Does TestGen change anything in my database?
No. TestGen runs read-only SELECT queries to profile data, run tests and collect monitor metrics. It never runs INSERT, UPDATE, DELETE, CREATE, ALTER or DROP, and it stores only computed results (statistics, scores and test outcomes) in its own application database.
What should I test in the bronze layer?
Whether the data arrived and whether it is shaped as expected: Table Freshness, Volume, Row Count, and type checks such as Character Column with Mostly Numeric Values. Fixing it here is cheapest, before silver and gold copy the problem.
What should I test in the gold layer?
Whether the numbers people see are right: Aggregate Balance against silver, Distribution Shift and Average Shift between loads, and Unique Values on business keys.
Is TestGen free?
The open source edition is free under Apache 2.0, for one user, one project and one connection, with every detector and test type except the Commercial Pharma tests. Enterprise is $100 per user per month plus $100 per database connection per month, with no per-table fees. See pricing.
Free book
Not ready to try it out now? Read the free book.
The DataOps Way to Data Quality and Data Observability covers why data quality keeps failing and what to do about it: culture, testing strategy, monitoring, and the seven deadly sins of AI-generated pipelines.
- 620 pages
- 63 chapters
- Updated September 2026
- By Chris Bergh, Chip Bloche, and Gil Benghiat