On-Demand Webinar · 56 min

Automated Test Generation: Why Data Teams Need It

Chris Bergh and Chip Bloche introduce DataOps TestGen: why data engineers have neither the time nor the business context to write data quality tests by hand, and how profiling a database can generate them instead.

Presented by Chris Bergh, Chip Bloche

What you'll learn 6 points
  • TestGen was built for two problems: data journeys that keep producing embarrassing errors, and data engineers who lack the data and business understanding needed to write validation tests for hundreds of datasets.
  • 52% of data engineers surveyed said errors are a significant source of burnout.
  • Automatically generated tests cast a wide net on purpose — sensors on every entrance, rather than watching the one window a burglar might try — to stay sensitive to real problems while limiting false positives.
  • The June 2023 release counted 28 automatically generated data quality checks, 11 fill-in-the-blank business-rule tests, 51 profiling characteristics collected, and 13 bad-data detector tests.
  • Bad-data detector tests point at patching work rather than alerts: invalid ZIP formats, leading spaces, multiple data types under one column name, no values present.
  • The customer example was a fully automated cell-therapy cancer analytics process: 70+ external sources of varying quality, analyst-ready data every 30 minutes, built by 3.5 FTEs over a year, with TestGen checks at every step.

Slides

38 slides

Questions from this session

What is automated data quality test generation?

It is deriving data quality tests from a profile of the data rather than writing them by hand. TestGen profiles the tables in a database, collecting 51 characteristics per column such as averages, date characteristics, minimum and maximum values, numeric counts, percentiles, and unique values, then uses that profile to generate 28 data quality checks. The engineer reviews and tunes the result instead of starting from a blank file.

Why don't data engineers write their own data quality tests?

Not for lack of willingness. They deal with hundreds of datasets and diverse customer needs, carry a backlog on the daily task list, and are held responsible for data they do not control upstream. Writing a good validation test requires knowing both the data and the business meaning behind it, and there is no time to learn that for every dataset.

What are the five pillars of Data Journeys?

Across the steps checks runs, order of operations, and schedule, giving process reliability. Down the stack monitors metrics, logs, and cost, giving technology status. Data at rest validates data quality with business domain tests. Data in use tests the results of models, visualizations, and delivery. Set expectations compares expected against reality and alerts. TestGen sits in the data at rest pillar.

What is the difference between automatically generated tests and fill-in-the-blank tests?

Automatically generated tests are derived from prior data and cast a wide net: average shift, value present in a list of values, distinct value change, future date. Fill-in-the-blank tests cover the rules that cannot be inferred from data at all, such as Data Match, Prior Match, and Aggregate Match No Drops. The logic is already written and verified, so a business data steward configures the rule rather than coding it.

Why do automatically generated tests cast a wide net?

The analogy used is a home burglar alarm: sensors go on every possible entrance even though a burglar will only try one window. Broad coverage is what keeps maximum sensitivity to real problems, and the tests are tuned so that breadth does not come at the cost of a flood of false positives.

What is a bad data detector test?

Bad data detector tests, also called profile screens, confirm how closely the assumed structure of the data matches the actual profile of each column. Examples include invalid ZIP code format, leading spaces, multiple data types under one column name, and no column values present. Their results point at patching work that makes a dataset usable, rather than raising a production alert.

Where to go next