How do you run data tests in CI/CD before an analytics release?

Add a job to your release pipeline that starts a TestGen test run through the REST API, polls until the run finishes, and fails the pipeline when any test fails. Tests with Warning severity get reported without blocking. TestGen runs from GitHub Actions, GitLab CI, or Azure DevOps through that API, so no connector is needed.

By Chris Bergh Published

Key points
  • A CI gate for TestGen is four documented REST calls: exchange a refresh token for an access token, POST /api/v1/test-suites/{test_suite_id}/test-runs, poll GET /api/v1/jobs/{job_id}, and read result_counts from GET /api/v1/test-runs/{job_id}.
  • A TestGen test run summary reports passed, failed, warning, error, log, and dismissed counts plus a score, so a CI job can block on failures and errors and only print warnings.
  • Passing external_url when you start a TestGen test run puts a link back to the CI job on the Test Results page, under Link and Run details.
  • TestGen OAuth access tokens expire after one hour and refresh tokens after 30 days, so a CI pipeline stores the refresh token as a secret and trades it for a fresh access token on every run.
  • DataOps Observability has no agent for GitHub Actions, GitLab CI, or Azure DevOps as of October 2026; TestGen exports test results to it as Test Outcomes events instead.

Run the tests your release depends on as a step in the release pipeline, and stop the release when they fail. TestGen runs from the CI system through its REST API; there’s no connector to install, and you don’t need one. How do you test data before a release covers what to test. This page covers the wiring, for any of the 14 databases TestGen supports.

One script, four documented calls

Everything below comes from TestGen’s Run Profiling and Tests and OAuth Clients pages. Register an OAuth client once, store the refresh token as a CI secret, and save this as scripts/testgen-gate.sh. It needs curl and jq.

#!/usr/bin/env bash
set -euo pipefail
# 1. Trade the stored refresh token for a one-hour access token
TOKEN=$(curl -sf -X POST "$TG_URL/oauth/token" -u "$TG_CLIENT_ID:$TG_CLIENT_SECRET" \
  -d grant_type=refresh_token -d refresh_token="$TG_REFRESH_TOKEN" | jq -r .access_token)
AUTH="Authorization: Bearer $TOKEN"
# 2. Start the test run, linked back to this CI job
JOB=$(curl -sf -X POST "$TG_URL/api/v1/test-suites/$TG_SUITE_ID/test-runs" -H "$AUTH" \
  -H "Content-Type: application/json" -d "{\"external_url\": \"$CI_RUN_URL\"}" | jq -r .id)
# 3. Poll until completed, error, or canceled, for up to 30 minutes
for i in $(seq 1 180); do
  STATUS=$(curl -sf "$TG_URL/api/v1/jobs/$JOB" -H "$AUTH" | jq -r .status)
  case "$STATUS" in completed|error|canceled) break ;; esac
  sleep 10
done
[ "$STATUS" = completed ] || { echo "TestGen run ended as: $STATUS"; exit 1; }
# 4. Block on failures and errors; report warnings
COUNTS=$(curl -sf "$TG_URL/api/v1/test-runs/$JOB" -H "$AUTH" | jq .result.result_counts)
echo "TestGen results: $COUNTS"
[ "$(echo "$COUNTS" | jq '.failed + .error')" -eq 0 ]

TG_URL is the TestGen API URL, port 8530 by default, not the UI URL. Open-source TestGen authenticates the API this way, with an OAuth 2.1 client: access tokens last one hour, and refresh tokens last 30 days, so re-run the one-time setup, browser sign-in included, before yours expires. Enterprise users can use a personal access token instead.

GitHub Actions, GitLab CI, Azure DevOps

Each system calls the same script after the build step. Keep the token and client secret in its secret store: GitHub secrets, GitLab CI/CD variables, or Azure Pipelines secret variables.

# GitHub Actions: a step in the release job
- name: TestGen gate
  run: scripts/testgen-gate.sh
  env:
    TG_URL: ${{ vars.TG_URL }}
    TG_SUITE_ID: ${{ vars.TG_SUITE_ID }}
    TG_CLIENT_ID: ${{ secrets.TG_CLIENT_ID }}
    TG_CLIENT_SECRET: ${{ secrets.TG_CLIENT_SECRET }}
    TG_REFRESH_TOKEN: ${{ secrets.TG_REFRESH_TOKEN }}
    CI_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

# GitLab CI: a job in the test stage (TG_* set as masked CI/CD variables)
testgen_gate:
  stage: test
  script:
    - CI_RUN_URL="$CI_JOB_URL" scripts/testgen-gate.sh

# Azure DevOps: a script step (secrets mapped into env explicitly)
- script: scripts/testgen-gate.sh
  displayName: TestGen gate
  env:
    TG_CLIENT_SECRET: $(TG_CLIENT_SECRET)
    TG_REFRESH_TOKEN: $(TG_REFRESH_TOKEN)
    CI_RUN_URL: $(System.CollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)

A non-zero exit fails the step in all three, and the release stops.

Blocking tests and warning tests

Every TestGen test definition carries a severity. The run summary counts failed and warning results separately, so the script blocks on the first and prints the second. Block on what breaks the release: row counts, keys, the business rules a dashboard depends on. Leave a slow trend you want a person to look at as Warning. The team in DataKitchen’s TestGen case study started with production incident checks and is moving toward this kind of pipeline gate. The argument for doing it is older: DataKitchen’s Add DataOps Tests to Deploy with Confidence.

Where results land

The run shows up on TestGen’s Test Results page, with your CI link under Link and Run details. Notifications send it to email, Slack, Microsoft Teams, or any HTTPS webhook, including a Jira automation rule. TestGen also exports results to DataOps Observability as Test Outcomes events. Observability has no CI agent, so the release itself shows up there only if you post events to its API. For orchestrators, see running TestGen from Airflow. For the earlier half of the line, see shift-left testing and Flip the Script.

Documentation: API, API Authentication, Run Profiling and Tests, Onboard a Dataset, Import and Export Tests, OAuth Clients, Notifications, Connect to Observability

Frequently asked questions

Short answers to the follow-up questions people ask next.

Does any data leave my environment?

No. TestGen runs profiling and tests as SQL inside your database, and TestGen runs on your own infrastructure. Your CI job receives only the job status and the result counts, so no rows pass through the runner or its logs unless you add a step that queries them. TestGen stores computed results plus, for display, the most frequent values and the minimum and maximum text values that profiling records.

Should a warning fail the build?

No. Block on failed tests and on tests that ended in error, and print warnings in the job log. Set a test's severity to Warning when its result needs a person's eye but shouldn't stop a release, such as a row count drifting toward its limit. If a warning gets ignored for weeks, either raise it to a failure or remove it.

Can CI run TestGen tests against a development schema?

Yes, with a table group that points at that schema. A TestGen table group reads one schema, and the match_schema_name option on a test run changes only the second table of referential tests. To test a branch's schema, create a table group and test suite for it through the API's Onboard a Dataset workflow, or change the table group's schema.

Do I need a GitHub, GitLab, or Azure DevOps connector for TestGen?

No. TestGen runs from any CI system that can make an HTTPS request, through its REST API. The runner needs network access to the TestGen API, port 8530 by default, so a TestGen behind your firewall needs a self-hosted runner or agent inside your network. Store the refresh token and client secret in the CI system's secret store.

More answers

Platforms How do TestGen and DataOps Observability send alerts and issues to Slack, Teams, email, and Jira? Platforms How do you profile, test and monitor data quality in Aurora PostgreSQL with TestGen? Platforms How do you profile, test and monitor data quality in Azure SQL Database with TestGen?

See every answer

Try it on your own data

DataOps TestGen is free and open source. Profile a table and generate its tests this afternoon.

Install Open Source TestGen Request a Demo