In April 2026, DataKitchen published a Databricks tripwire that reached TestGen by opening an SSH session to its VM and running docker compose exec inside the container. The post was blunt about the cost: key rotation, port whitelisting, a security review. It said an HTTP API was coming and would turn the whole thing into a REST call.
It shipped. DataOps TestGen now runs from Databricks Workflows, which Databricks calls Lakeflow Jobs, through its REST API. No connector, no SSH. The Airflow answer and the CI/CD answer cover the same calls elsewhere. Setting up the Databricks SQL connection TestGen tests through is in the Databricks how-to.
The job
bronze_to_silver (notebook) -> testgen_silver (Python script) -> silver_to_gold (notebook)The middle task, from Run Profiling and Tests:
import sys, time, requests
from databricks.sdk.runtime import dbutils # dbutils outside a notebook
suite_id = sys.argv[1]
API = dbutils.secrets.get("testgen", "api-url") # http://<host>:8530 by default
HDR = {"Authorization": f"Bearer {dbutils.secrets.get('testgen', 'token')}"}
job_id = requests.post(f"{API}/api/v1/test-suites/{suite_id}/test-runs",
headers=HDR, timeout=30).json()["id"]
deadline = time.time() + 1800
while time.time() < deadline:
job = requests.get(f"{API}/api/v1/jobs/{job_id}", headers=HDR, timeout=30).json()
if job["status"] in ("completed", "error", "canceled"):
break
time.sleep(15)
else:
sys.exit(f"TestGen job {job_id} did not finish")
if job["status"] != "completed":
sys.exit(f"TestGen job {job_id}: {job['status']} {job['error_message']}")
result = requests.get(f"{API}/api/v1/test-runs/{job_id}", headers=HDR, timeout=30).json()["result"]
counts = result["result_counts"]
print(f"score {result['score']}, counts {counts}")
if counts["failed"] or counts["error"]:
sys.exit(f"FAIL: {counts['failed']} failed, {counts['error']} errors in suite {suite_id}")sys.exit with a message is the enforcement, same as the April version. Only the transport changed. The token and URL sit in a Databricks secret scope, not in the script.
Block the next task
Leave silver_to_gold on the default Run if condition, All succeeded. When testgen_silver fails, Databricks marks silver_to_gold Upstream failed and skips it (docs checked October 2026). Gold stays on the last good data. Add a task with At least one failed if you want a cleanup or notification step on the failure path.
Warnings pass through. A TestGen test set to warn returns warning, not failed, per the testing docs, so the data steward decides what blocks without a redeploy. DataKitchen’s medallion post puts one of these tasks between each layer.
Where results land
In TestGen’s Test Results page, with the job’s printed score and counts in the task log. The run accepts an optional body with external_url and metadata, shown under Link and Run details; send the Databricks run URL so a failed test links back to the job.
Profile on a schedule, not in every run. POST /api/v1/table-groups/{table_group_id}/profiling-runs works from a weekly job, or use TestGen’s own scheduler.
Tokens and Observability
The script reads one bearer token from the secret scope. Personal access tokens are an Enterprise feature, and they are the simple fit. Open source authenticates with an OAuth 2.1 client instead. Its access token expires after one hour, so store the client ID, client secret, and refresh token in the scope and trade the refresh token at /oauth/token at the top of the script. The refresh token lasts 30 days, after which someone re-authorizes the client in a browser.
DataOps Observability has a documented Databricks agent that watches the workflow runs themselves. The docs mark access to its private Docker repository as Enterprise. TestGen results join that view through Connect to Observability. For how TestGen compares with Lakehouse Monitoring and the managed vendors, see the Databricks tool comparison.
Documentation: TestGen API, Run Profiling and Tests, API Authentication, OAuth 2.1 Clients, Databricks quickstart, Connect to Observability, Observability Databricks agent