Open source data quality for Azure Synapse
Azure Synapse data quality, without writing tests.
Point TestGen at Azure Synapse Analytics. It profiles every table, generates thousands of tests, and watches freshness, volume, and schema. Read-only access. Free and open source.
Install TestGen
Pick your platform. Free and open source.
# TestGen install for Mac, Linux.
# Download the latest installer.
$ curl -o dk-installer.py \
'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
# Run the install command.
$ python3 dk-installer.py tg install
# Full docs: docs.datakitchen.io/testgen/get-started/install-on-mac-linux/ - Double-click
dk-installer.exein your Downloads folder. - If Windows shows a protection warning, choose More info, then Run anyway.
- Pick TestGen, then Docker or pip. The installer seeds demo data, opens the UI, and prints your login.
# Full docs: docs.datakitchen.io/testgen/get-started/install-on-windows/
- 55data profiling column characteristics
- 32data hygiene detectors
- 34test types auto-generated from the profile
- 19business rule and custom SQL test types
Connect TestGen to Azure Synapse
TestGen pushes read-only queries into Synapse and stores only results: statistics, scores, and test outcomes. Moving to Fabric later? TestGen connects to Fabric too.
- Sign-in
- SQL login, Azure Managed Identity, or an Entra ID service principal.
- Permissions
- SELECT on the tables you want tested.
information_schemaandsys.partitionsare readable by default, andsys.objectsshows the tables you can already select. - Where TestGen runs
- An Azure VM, a container, or any machine that can reach the workspace.
- Next stop: Fabric
- The same tests carry over when you move to Microsoft Fabric.
Azure Synapse data problems TestGen catches
What breaks in a warehouse fed from files and many sources. Each card lists the TestGen checks that find it.
-
A source file was missing
One feed didn’t land and a fact table quietly shrank.
-
Dates parsed as text
File loads that left date and number columns as strings.
-
Facts that don’t add up to dimensions
Totals and keys out of line between fact and dimension tables.
-
Double loads
A rerun appended a batch that was already there.
-
Moving to Fabric
Proving the data in Fabric matches what was in Synapse.
-
Stale tables
A table nobody noticed stopped updating.
What TestGen does on Azure Synapse
Everything below is in the open source release, and all of it works on Azure Synapse. Enterprise adds team access, security, Commercial Pharma tests, and support.
-
Auto-Generated Tests (documentation, opens in a new tab)
Point TestGen at Azure Synapse and it writes the tests for every warehouse table, then runs them inside Synapse.
-
Data Profiling (documentation, opens in a new tab)
55 characteristics per column, using Synapse’s own sampling on large tables, so data profiling doesn’t pull the warehouse out to read it.
-
Hygiene Detection (documentation, opens in a new tab)
Finds the leftovers of file loads: dates and numbers parsed as text, odd blanks, and placeholder values.
-
Anomaly Detection (documentation, opens in a new tab)
Freshness, volume, schema and custom-metric monitors flag a missing source file or a fact table that quietly shrank.
-
Data Catalog (documentation, opens in a new tab)
Profiles, test results, PII flags and Critical Data Elements for every Synapse table, and a record to compare against when you move to Fabric.
-
Quality Scoring (documentation, opens in a new tab)
Scorecards by dimension and Critical Data Element for the warehouse, before and after a move to Fabric.
-
MCP Server (the TestGen MCP cheat sheet)
Ask an AI assistant which Synapse tables failed and why, through 112 MCP tools that run in your environment.
-
Open Source Core, UI Included
Unlimited tables and a full UI. The core data profiling, testing and monitoring engine is free under Apache 2.0, for one Synapse connection. Enterprise adds more connections, team access, security and support.
From DataKitchen on Synapse and Azure
- Answer
How do you profile, test and monitor data quality in Azure Synapse with TestGen?
TestGen connects to Azure Synapse Analytics with a SQL login, managed identity, or service principal, and runs profiling and generated tests in place.
- Answer
How do you validate a SQL Server to Microsoft Fabric migration?
Profile SQL Server before the move, run comparison tests on the Fabric warehouse or lakehouse after it, and sign off when every failure matches a planned change.
- Answer
What is the best data quality and data observability tool on Azure?
On Azure, pick a tool that profiles Synapse, Azure SQL, Fabric, and Blob files, generates tests, and monitors freshness and volume in your tenant.
- Answer
How do you monitor freshness and row counts on SQL Server tables?
T-SQL to read row counts from sys.partitions, find last writes, keep a history table, and alert on drops, plus how monitors learn each table's normal pattern.
-
Blog The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure
The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. Is it overkill? Paradox of choice? Or the right tool for the right job? We discuss.
-
Blog There Is No Microsoft Data Quality Tool in 2026. There Are Nineteen.
You went looking for the Microsoft data quality tool. There isn't one. There are nineteen features across five products, twelve of them manual, and they don't talk to…
Why teams choose TestGen
-
You Don't Have Time to Write Tests. TestGen Does It Automatically.
You're buried in customer requests. You have no time to write tests, let alone innovate. TestGen scans your data and generates the tests and anomaly detectors for you. No coding, no massive YAML configuration.
-
A warehouse this big needs more tests than anyone will write.
TestGen profiles every column and generates the tests, then monitors each table for freshness, volume, and schema changes.
See how TestGen works →
-
Runs inside Synapse
Tests run as read-only SQL where the data sits. Nothing is copied out, and nothing is written back.
-
Apache 2.0, core engine free
Every detector, every characteristic, and every test type but the 12 Commercial Pharma types. In the open source.
-
Self-hosted
Your data never leaves your environment.
-
Bootstrapped since 2013
Profitable, independent, and not pivoting next quarter.
Open Source with Reasonable Enterprise Pricing
Open source TestGen is free for one user, one project, and one database connection. Enterprise is a flat $100 per month per user and database connection, with unlimited tables and data volume. No per-table tax. A year of enterprise data quality costs roughly one data engineer's monthly salary.
See pricingTestGen versus the competition
Head-to-head comparisons with every major data quality and observability tool. Search for the one you are evaluating.
No vendors match. .
- Bristol Myers Squibb
- Eisai
- Catholic Relief Services
- Progeny Health
- X4
- KOA
- CNH
Questions
How does TestGen sign in to Azure Synapse?
With a SQL login, an Azure Managed Identity, or an Entra ID service principal. The account needs SELECT on the tables you want tested.
We are moving from Synapse to Fabric. Does that matter?
TestGen connects to both, so you can run the same profiling and tests on Synapse before the move and on Fabric after it, and compare the results.
Does TestGen change anything in my database?
No. TestGen runs read-only SELECT queries to profile data, run tests and collect monitor metrics. It never runs INSERT, UPDATE, DELETE, CREATE, ALTER or DROP, and it stores only computed results (statistics, scores and test outcomes) in its own application database.
Does TestGen work on very large Synapse tables?
Yes. TestGen runs its queries inside Synapse, and sample-based data profiling uses database-native sampling on Synapse, so it doesn’t pull the table out to profile it.
Can TestGen run after a Synapse pipeline?
Yes. Schedule test runs in TestGen or trigger them through its CLI or API as the last step of a pipeline, so each load gets tested before anyone reads it.
Is TestGen free?
The open source edition is free under Apache 2.0, for one user, one project and one connection, with every detector and test type except the Commercial Pharma tests. Enterprise is $100 per user per month plus $100 per database connection per month, with no per-table fees. See pricing.
Free book
Not ready to try it out now? Read the free book.
The DataOps Way to Data Quality and Data Observability covers why data quality keeps failing and what to do about it: culture, testing strategy, monitoring, and the seven deadly sins of AI-generated pipelines.
- 620 pages
- 63 chapters
- Updated September 2026
- By Chris Bergh, Chip Bloche, and Gil Benghiat