The Periodic Table of Data Quality

Ask a governance lead how to organize data quality tests and you get dimensions: validity, completeness, timeliness. Ask the engineer on call and you get symptoms: the schema changed, the row count fell off a cliff, the table went stale. The person who wrote the SQL thinks in algorithms. The person who has to configure the test wants to know how much it needs from them.

They’re all right. That’s the trouble with a folder tree. A test lives in one folder and belongs in four.

So shop for tests the way you shop on Amazon. Nobody scrolls every product in Electronics. You tick a brand, a price range and a rating, and the list shrinks to something a person can read. Tests have facets the same way products do, which is why we rebuilt TestGen’s Add Test screen as a faceted picker. Below is the same idea laid over all 130 checks the TestGen documentation lists, every test type, table monitor and data hygiene issue, drawn as a periodic table. Search, or tick a facet on the left, and watch the table rearrange. Every tile opens its entry in the TestGen documentation.

Six ways to sort a data quality test

Quality dimension

The governance vocabulary: accuracy, completeness, consistency, timeliness, uniqueness, validity. It’s the axis everybody reaches for first, and the tile colors use it. It is also the least helpful one at 2am, when what you know is the symptom and not the category.

Impact dimension

The kind of operational risk a failure carries: conformance, reliability, regularity or usability. It’s lopsided: conformance holds 80 of the 130 checks. TestGen’s quality scores break down along this axis too, so it is the one to sort by when you want to know where the risk sits.

Health dimension

The symptom: schema drift, data drift, volume or freshness. Start here when something is already wrong and you want every check that would have seen it coming.

Algorithm

How the check computes its answer. Counting, a boundary check, a regex, a set lookup, a statistical drift measure, an aggregate reconciled against a baseline, or custom SQL. Aggregate reconciliation is the biggest group, 38 tiles, and 33 of them are the Match tests that compare a target table against a source baseline. That is how you prove a data migration worked. Statistical technique goes one level deeper for the drift tests, down to Cohen’s D or Jensen-Shannon divergence.

Test scope

What the check points at. Of the 130, 77 run against a single column. Ten run against a table, eight compare one table to another, and the Schema monitor watches a whole table group.

Criteria

How much you have to tell the test. A defined rule needs nothing but switching on: Duplicate Rows already knows what a duplicate is. A defined threshold needs a number. A reference dataset needs a second table to compare against. The rail lists them in that order, least setup to most.

One tile, six answers

Pick any tile and it answers all six at once. Row Count is a reliability check on completeness. Its symptom is volume, it counts, it runs on a table, and it needs one number from you: the threshold. Filed in one folder, five of those six facts go missing.

How the facets combine

Ticked values inside one facet widen the set: tick Column and Table under Test Scope and you see tests of either scope. Ticked values across facets narrow it: add Counting under Algorithm and only the column and table checks that count stay on the table. A tile with no value on a ticked facet drops out. Profiling finds data hygiene issues, and nobody configures them as tests, so they carry an impact and a quality dimension and nothing else, and they leave the moment you tick an algorithm.

The counts beside each value are tiles. They follow the search but not the other facets, the same as in the product.

The Periodic Table of Data Profiling

Most of the checks above start from a profile. TestGen finds hygiene issues in it and generates tests from it, and the documentation lists 55 characteristics a profile captures. Four of them name the column instead of measuring it: table name, column name, column type and position. The other 51 get their own table.

None of them pass or fail. They are raw material. Sort them by the column type that gets them, by the section TestGen files them under, or by whether they hold raw values, which TestGen redacts for anyone not allowed to see a PII column.

Where to start

You don’t need all 130. You need the few that would have caught last month’s incident, and facets are how you find them. Start from the symptom you already lived through under Health Dimension. Narrow to the scope you can actually test. Then let Criteria show you which checks cost nothing but a click to switch on.

After that, point TestGen at the table and let the profile write the first draft of the suite. The periodic table is for the tests a profile can’t guess.

FAQ