Data Factory tells you the copy succeeded and how many rows it moved. It has no opinion on whether the customer key in those rows is null. Put DataOps TestGen between the copy and the notebook that reads it.
TestGen runs from Data Factory through Web activities that call its REST API. No connector, and none needed. The Airflow answer and the CI/CD answer cover the same calls in other tools. For what TestGen tests in the database itself, see the Synapse and Fabric and OneLake answers.
The pipeline
Copy, test, then transform. The endpoints come from Run Profiling and Tests:
| Activity | What it does |
|---|---|
| Copy data | Loads the table |
| Web: start run | POST /api/v1/test-suites/{test_suite_id}/test-runs, returns a job id |
| Until | Wait 15 seconds, then Web GET /api/v1/jobs/{job_id}; exit when status is completed, error, or canceled |
| Web: read result | GET /api/v1/test-runs/{job_id}, returns result.result_counts and result.score |
| If Condition | Failed or error count above zero goes to Fail; otherwise on to the notebook |
| Notebook | Runs only when the data passed |
The start activity, with turnOffAsync set so ADF returns the job ID instead of chasing the 202:
{
"name": "Start TestGen run",
"type": "WebActivity",
"typeProperties": {
"method": "POST",
"url": "@concat(pipeline().parameters.testgenApi, '/api/v1/test-suites/', pipeline().parameters.suiteId, '/test-runs')",
"headers": { "Authorization": "@concat('Bearer ', variables('tgToken'))", "Content-Type": "application/json" },
"body": "{\"metadata\": {\"system\": \"Azure Data Factory\", \"run_id\": \"@{pipeline().RunId}\"}}",
"turnOffAsync": true
}
}The If Condition expression on the read-result activity:
@greater(add(activity('Read TestGen result').output.result.result_counts.failed,
activity('Read TestGen result').output.result.result_counts.error), 0)Its true branch is a Fail activity, which ends the pipeline with an error message and code you choose. The notebook never starts. That’s the tripwire pattern DataKitchen’s OneLake post describes for Data Factory and Fabric pipelines.
Three ADF details that bite
The Web activity times out after 1 minute with no response, up to 10 with httpRequestTimeout. That’s why the start call returns at once and the Until activity waits instead. Give the Until a timeout too. Microsoft’s docs (checked October 2026) say an inner activity’s failure doesn’t stop the loop.
Warnings don’t block. A TestGen test set to warn returns warning, not failed, per the testing docs. The expression above ignores them on purpose.
Tokens. Open source authenticates with an OAuth 2.1 client: a first Web activity posts the stored refresh token to /oauth/token with the client ID and secret as Basic authentication and gets a one-hour access token. A refresh token lasts 30 days, then someone signs in through the browser to issue a new one. Enterprise adds personal access tokens. Keep either in Azure Key Vault, not in the pipeline JSON.
Where results land
In TestGen. The Test Results page shows every result, and the metadata you sent appears under Run details, so a failed test carries the ADF run ID. The optional body also takes an external_url, shown under Link, if you want a clickable path back to the run.
DataOps Observability has a documented Azure Data Factory agent. It reads ADF events from Azure Event Hubs, fed by diagnostic settings, and puts pipeline runs on the same timeline as TestGen results exported to Observability. The docs mark access to the agent’s private Docker repository as Enterprise. DataKitchen’s nineteen Microsoft features post maps what Data Factory, Purview, and Fabric check on their own, and the Fabric tool comparison weighs the alternatives. The supported sources list covers what TestGen tests once the copy lands.
The core profiling, testing, and monitoring engine is free and open source. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support.
Documentation: TestGen API, Run Profiling and Tests, API Authentication, OAuth 2.1 Clients, Connect to Observability, Observability Azure Data Factory agent, Add Azure Diagnostic Settings