How do you profile, test and monitor data quality in Microsoft Fabric and OneLake with TestGen?

DataOps TestGen connects to Microsoft Fabric through the SQL analytics endpoint with an Entra ID service principal or managed identity, runs profiling and data tests as read-only SQL against your lakehouse and warehouse Delta tables, monitors freshness, volume, and schema, and scores each medallion layer. It's Apache 2.0, and OneLake support is in the free build.

Key points

  • DataOps TestGen has connected to Microsoft OneLake since version 5.92.1, in open source and Enterprise, through the Fabric SQL analytics endpoint, which does not accept SQL logins; TestGen authenticates with an Entra ID service principal or a managed identity.
  • TestGen's Access Requirements page, updated October 2026, lists Contributor or higher on the Fabric workspace and the tenant setting 'Service principals can use Fabric APIs' enabled in the Fabric admin portal.
  • Microsoft says the lag between a lakehouse write and its SQL analytics endpoint is normally under one minute but can run to minutes, so a test fired the instant a Spark notebook finishes can read stale data.
  • A TestGen table group is one schema, so bronze, silver, and gold built as three schemas in one lakehouse fit one connection with three table groups. Microsoft's architecture guidance puts each layer in its own lakehouse, and since a TestGen connection names one database, we expect that layout to need a connection per lakehouse, which takes TestGen Enterprise.
  • TestGen profiles 55 characteristics per column, flags 32 data hygiene issues, generates tests from 34 test types, and stores only results and metadata, never table rows.

This is the setup page. The Fabric tool comparison covers whether to use TestGen at all.

Connect through the SQL analytics endpoint

Every lakehouse in OneLake gets a SQL analytics endpoint, a read-only T-SQL view of its Delta tables over TDS, the same protocol SQL Server uses. TestGen connects there. Since 5.92.1 it’s in open source and Enterprise.

The Access Requirements page lists what it takes:

RequirementWhere
Entra ID service principal or managed identityThe endpoint doesn’t accept SQL logins
Contributor or higherThe Fabric workspace
”Service principals can use Fabric APIs”: EnabledTenant settings, Fabric admin portal

Ask your Fabric admin for the tenant setting first. It’s the step that stalls people for a week, and the data team can’t flip it. Make the principal TestGen’s alone and grant it only the workspaces you test.

TestGen runs no INSERT, UPDATE, DELETE, CREATE, ALTER, or DROP. The endpoint is read-only for table data anyway.

Lakehouse or warehouse

The endpoint takes its schema and data from the underlying lakehouse or warehouse, per TestGen’s docs, so both are testable. That matters because one of the two patterns in Microsoft’s medallion guidance puts gold in a Fabric Warehouse. Our OneLake release post walks through both patterns. If gold is a SQL Server your finance team won’t give up, TestGen connects to that too, alongside Azure SQL and Synapse.

Map medallion layers to table groups

A table group is one schema, and a connection names one database. So the layout you picked decides the setup. Microsoft’s materialized lake views tutorial builds the three layers as schemas in one lakehouse; its architecture guidance recommends a lakehouse per layer, ideally each in its own workspace.

Your layoutTestGen setup
Bronze, silver, gold as schemas in one lakehouseOne connection, three table groups
A lakehouse per layerA connection per lakehouse, each with its own table group
Gold in a Fabric WarehouseIts own connection

The second and third rows are our reading of those two documented facts, not a tested DataKitchen recipe. Open source allows one connection, so those layouts need Enterprise. Each layer gets its own scorecard either way.

Profile

Profiling captures 55 characteristics per column and flags 32 kinds of hygiene issue: placeholder blanks like “N/A”, text columns full of dates, possible PII. Profile sampling, if you turn it on, applies only to pattern, character, and frequent-value analysis; counts, minimums, maximums, and percentiles read every row. Purview’s profiler reads a 1-million-row sample.

Profile bronze; don’t gate it. Shortcuts and mirroring bring data in with no ETL you wrote, so there’s no pipeline log to read. The profile is how you see what arrived. For a large Delta table, profile a view over the recent window, as the docs recommend.

Test

Test generation writes tests from 34 test types against the latest profile. You add 19 business-rule and custom SQL types by hand. Every test runs as T-SQL through the endpoint.

Watch the sync lag. Microsoft says a lakehouse write normally reaches the endpoint in under a minute, sometimes minutes, and the sync stops after 15 minutes idle. A test that fires the moment a notebook finishes can read the old data. From a Fabric pipeline, call the endpoint’s metadata refresh API, then trigger TestGen through its REST API (POST /api/v1/test-suites/{test_suite_id}/test-runs) and poll GET /api/v1/jobs/{job_id}. Open source authenticates with an OAuth 2.1 client: one-hour access tokens from a refresh token that lasts 30 days, then a browser sign-in to issue a new one. Personal access tokens are Enterprise. Fail the pipeline on a failed test and you have a tripwire between bronze and silver.

Observe

Monitors learn each table’s update schedule, train volume models over about 30 runs, and report every schema change. Alerts go to email, Slack, Microsoft Teams, or a webhook through notifications.

DataOps Observability has documented agents for Azure Data Factory and Power BI dataset refreshes. It has no documented agent for Fabric pipelines or notebooks as of October 2026. Export TestGen results to see them next to the refresh that read them.

Dashboard

Each table group gets a scorecard, so bronze, silver, and gold each carry a quality score and a CDE score for critical columns. Issue reports go to whoever owns the fix.

TestGen's hygiene issues view: a list of issues found during profiling, each with a likelihood rating, the affected table and column, and a suggested action.

The screenshot shows TestGen’s hygiene issues list. It isn’t from a Fabric tenant, and it looks the same on every database.

Walkthrough: the general steps

DataKitchen has no Fabric quickstart yet, so these are the general Connect your Data steps applied to OneLake.

  1. Have a Fabric admin enable “Service principals can use Fabric APIs.”
  2. Create the service principal and add it to the workspace as Contributor.
  3. Install TestGen.
  4. Add a Microsoft OneLake connection with the endpoint’s SQL connection string and the principal’s credentials. Click Test Connection.
  5. Create a table group per layer schema. Leave CDE and PII detection on. Run profiling.
  6. Generate and schedule tests, configure monitors, and click Finish Setup.

TestGen vs Fabric’s native data quality

DataOps TestGenPurview data qualityMaterialized lake view constraints
Rules34 generated types, plus 19 hand-configuredAI-suggested and hand-attached, 200 per assetHand-written CHECK constraints
ProfilingFull table1M-row random sampleNone
Freshness, volume, schemaLearned monitors on every tableOne freshness ruleNone
CoversLakehouse and warehouse via the endpointLakehouse, shortcuts, mirroringOne view, at refresh
CostFree for one connectionPay-as-you-go processing unitsSpark refresh

Microsoft details checked October 21, 2026. The Microsoft roundup covers the rest of Microsoft’s options. The four test points post covers where tests go in a medallion. For the full list, see supported sources and the Azure overview.

The core profiling, testing, and monitoring engine is free and open source. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support. Open source covers one user, one database connection, and one project, with unlimited tables. Enterprise is $100 per user per month plus $100 per database connection per month.

Documentation: Connect your Data, Access Requirements, Manage Connections, Manage Table Groups, Manage Test Suites, Run Profiling and Tests (API), OAuth 2.1 Clients, Notifications, Connect to Observability.

Frequently asked questions

Short answers to the follow-up questions people ask next.

What access does TestGen need in Microsoft Fabric?

An Entra ID service principal or managed identity with Contributor or higher on the Fabric workspace, per TestGen's Access Requirements page in October 2026. A Fabric admin must also enable the tenant setting 'Service principals can use Fabric APIs'. The Fabric SQL analytics endpoint does not accept SQL logins, so a username and password connection won't work.

Why do TestGen tests fail right after a Spark notebook writes to a lakehouse?

The SQL analytics endpoint trails the lakehouse. Microsoft says the sync lag is normally under one minute and can run to minutes, and the background sync stops after 15 minutes without query activity. Call Microsoft's Refresh SQL analytics endpoint metadata API, or wait, before you trigger the TestGen test run.

Does any data leave my environment?

No. DataOps TestGen runs profiling, tests, and monitors as read-only SQL through the Fabric SQL analytics endpoint, from a TestGen instance you host. Only results and metadata are stored in TestGen's own database, including profiling statistics such as minimum and maximum values and each column's most frequent values, which the Data Catalog displays. TestGen never writes to OneLake.

Can one TestGen connection cover bronze, silver, and gold in Fabric?

Yes, if the three layers are schemas in one lakehouse, which is how Microsoft's materialized lake views tutorial builds them. A TestGen table group is one schema, so you get three table groups on one connection. Microsoft's medallion architecture guidance recommends a separate lakehouse per layer instead. A TestGen connection names one database, so we expect each lakehouse to need its own connection, and open-source TestGen allows one connection.

Does TestGen test a Fabric warehouse as well as a lakehouse?

Yes. TestGen's OneLake connection reads through the Fabric SQL analytics endpoint, which TestGen's documentation says derives its schema and data from the underlying lakehouse or warehouse. If gold lives in SQL Server or Azure SQL Database instead, TestGen connects to those directly, and since 5.92.1 Azure SQL also accepts Entra ID service principals.

Try it on your own data

DataOps TestGen is free and open source. Profile a table and generate its tests this afternoon.