DataOps TestGen Technical Overview

Introduction

Does this sound like your team? Are you a member of a stressed-out, overworked data team that directly provides insight to customers by both building and running complex data journeys? Do your data journeys produce embarrassing data errors regularly? Are you aware that only 22% of data engineers’ time is spent on innovation, but 78% on errors and manual execution (Gartner 2022)? Are you aware that a survey of 700 data engineers conducted by DataKitchen & data.world in 2022 found that 52% of Data Engineers said errors are a significant source of burnout?

Imagine having a simple and fast way to generate and execute data quality tests in your data journeys without hassle. With DataKitchen DataOps TestGen, you can avoid tedious and time-consuming manual test development processes. Our software is designed to be simple, fast, and highly efficient and deliver many valuable data quality validation tests in minutes. It automates the entire data profiling, test generation, and execution process, saving you valuable time and resources. With DataOps TestGen, you can focus on what matters most — delivering trusted insight to your customers — while TestGen ensures data accuracy and quality. Say goodbye to the complexity of writing data tests yourself. DataOps TestGen takes care of that for you, automating the terms and conditions of your data contract through simple, automatically generated data test creation and execution.

DataOps TestGen is the “Easy Button” for data quality monitoring and anomaly detection. Our robust software streamlines the entire testing process, making it a breeze to identify and rectify any data errors or anomalies.

The Problem DataOps TestGen Solves

Make embarrassing and costly data errors a thing of the past. Most data engineers are so busy that they don’t have time to write data quality validation tests or the business domain expertise needed. As a result, tests go unwritten, and production systems have problems, reducing end-user trust. Their lack of time and understanding means they need a standard set of data tests created automatically and a process to tweak existing production tests when new data comes in. For more advanced business-specific testing, they need a set of tests configured easily in concert with data stewards. Together, those two kinds of test let data teams detect problematic data records efficiently.

Problem #1: Costly production data errors are found by customers — or not at all. Data errors cause your customers to mistrust the data, are hard to find, take time to investigate, and waste your time on re-work. Data errors can cause compliance risk and have an opportunity cost for the business because of downtime during errors. And incorrect data or wrong reports and models can cause costly business mistakes.

Problem #2: Data teams don’t have the time or business understanding to create data quality validation tests. Data engineers deal with hundreds of data journeys, diverse data sets, tools, and diverse customer needs. They have backlogs on their daily task lists. So they don’t have time or energy to learn about each data set or customer to create robust production data tests. They need help creating tests that fit their data and customers’ needs without taking significant time to set up or babysit once they go live. They also need help adding new types of tests for the business and adjusting tests when new data arrives.

Essential Product Features

One of the banes of the data team is manual data testing. Manual testing takes time and effort and is error-prone. Always test your data automatically, not manually. Execute automatic data validation tests with parameters that can be changed based on inputs and business conditions — saving re-work and duplication. DataOps TestGen creates and executes data tests automatically.

DataOps TestGen automatically tests data before, during, and after data production. Within minutes of installation, its components cover all three phases:

Before production begins	During production, TestGen executes	After production ends, TestGen lets users
Creates data profile — identifies 51 baseline characteristics per column	Auto-generated data tests	Evaluate test results
Bad data detection tests find data problems	Custom business rule data tests	Review and refine test parameters
Creates auto-generated tests from data profile characteristics	Any existing data tests already in use, even those generated outside of TestGen	When new data arrives, refresh the data profile and the auto-generated thresholds
Optionally configure custom business rule tests

Data Profiling

Data profiling is the periodic x-ray of tables in a schema to gather extensive information about the contents of each column. Results are stored in a standard table in DataOps TestGen. This table is available for direct review and is used for rules derivation downstream. As more versions of a dataset are profiled, the derived testing gets better.

Because profiling conducts a separate query on each column of every table analyzed, profiling is not meant to be part of the production data validation testing process. Typically, you might profile data on a weekly or monthly basis. The profiling step allows you to perform profiling on a random percentage of records within a table. But we recommend analyzing the entire table where possible, even if the process takes some time, to capture the fullest picture of the data.

The 51 baseline characteristics gathered per column are grouped as follows:

Type	Data profiling characteristic name(s)
Averages	Average Embedded Spaces, Average Length, Average Value
Column & table types & names	Column Name, Column Type, Functional Data Type, Functional Table Type, General Type, Table Name
Date characteristics	Before 1yr Date Count, Before 5yr Date Count, Data Type Suggestion, Date Count, Date Days Present, Date Months Present, Date Weeks Present, Future Date Count, Within 1mo Date Count, Within 1yr Date Count
Minimum/maximum value	Maximum Date, Maximum Length, Maximum Text, Maximum Value, Minimum Date, Minimum Length, Minimum Text, Minimum Value, Minimum Value Gr0
Numeric counts	Boolean True Count, Embedded Space Count, Filled Value Count, Fractional Sum, Lead Space Count, Numeric Count, Record Count, Value Count
Percentiles	Percentile 25, Percentile 50, Percentile 75
Position	Position
Top N	Top Frequent Values, Top Patterns
Unique values	Distinct Pattern Count, Distinct Standard Value Count, Distinct Value Count, Distinct Value Hash
Zero length or nulls	Null Value Count, Zero Length Count, Zero Value Count

Bad Data Detection Tests

Once data profiling is complete, Bad Data Detection Tests can be run. The Bad Data Detection Tests automatically confirm how closely data structures and assumptions match the actual contents of each column. Results can be used to assist the Data Engineer in refining data structure definitions and target the addition of data ‘patching’ steps which help to generate a more usable, analyzable dataset.

A reasonable approach would be to review Bad Data Detection Tests to start and make any changes that may be required to address issues raised. Then, when you have a reliable dataset, you can rerun profiling and the Bad Data Detection Tests repeatedly.

Below is the list of the 13 Bad Data Detection Tests:

Test name	Description
Invalid Zip Code Format	The zip code column doesn’t match the expected format. A column with “zip” in the name that is not defined as alpha, or where contents have a minimum length below 5 or over 10 characters.
Leading Spaces	Leading spaces in an existing character column.
Mostly Dates In String	Character column contains over 95%, but less than 100%, date values — invalid data might be present, or the data type could be changed.
Mostly not null, empty, or filled values	The column is more than 97% populated but with some representations of missing values — invalid data might be present.
Mostly Numeric In String	Character column contains over 95% but less than 100% numeric values — invalid data might be present; for example, the data type could be changed.
Mostly One Value	The column is over 97%, but less than 100%, a single value — invalid data might be present.
Multiple Data Types Per Column Name	Multiple data types exist per column name across the schema.
No Column Values Present	The column is entirely unpopulated.
Non-standard Blank Values	There are many ways to denote blank or missing values that can come through in an external dataset. This is especially true when data is integrated from multiple sources, when data-entry validation checks are inefficient, or when users can bypass the checks.
Pattern Inconsistency	Flags cases where alpha-numeric patterns may be inconsistent within the column, or with columns of the same name in other tables. This is only applied where two or fewer patterns exist for one column version. Patterns consist of upper-case or lower-case alphas represented by “A” or “a”, numeric digits represented by “N”, and non-alpha-numeric characters represented directly.
Potential Duplicates	Column values are almost all unique, but some duplicate values are present.
Suggested Data Type	Suggests potential new data types that better match column contents. For example, a date type would be suggested if a column is defined as alpha but all contents are valid dates. Where warranted, the test also offers smaller sizes for alpha values, allowing a reasonable buffer for new, longer values. Sometimes a slightly larger size may be suggested to accommodate the buffer.
Too Many Boolean Values	The column contains multiple representations of boolean values.

Automatically Generated Data Tests

Once profiling is complete, optionally, TestGen can automatically derive a set of data quality validation test rules on all columns profiled. Typically between two and nine tests are generated per column. The total tests generated can be in the thousands for a full schema. Although profiling itself can take some time, test generation is high-speed.

The goal of Automatically Generated Data Tests is to cast a wide net for data problems that can’t be predicted by targeted testing devised in advance. It’s the same way you might set up a burglar alarm in your home by deploying sensors at all possible entrances to catch a burglar who would only try one window. Your goal in refining these tests is to maintain maximum sensitivity to real problems while minimizing false positives that are not worth the follow-up.

Does this take some effort? Of course! The benefit is that it’s much less work to refine a small number of tests that miss the target. It’s much more work — if not impossible in the real world — to manually create a blanket of relevant tests that can alert you when anything goes wrong. And when you are refining tests, you have direct access to the result history for that test and profiling data for that column, which makes it even easier.

Initially, the value of the tests is only as good as the quality of the baseline dataset. By definition, subsequent tests conducted on the same baseline data should all pass. Accordingly, TestGen’s test generation process allows you to base your tests on a prior version of profiling results — based on an older version of data — so you can immediately evaluate and refine your tests on the current version of your dataset. Or you can use the current data as your baseline and initiate testing against subsequent versions of data.

You can regenerate your tests as often as you like. You may choose to do this periodically. But the point is not to keep regenerating tests that change. It’s to build a consistent set of rules that can be applied to new data.

The following 28 test types are automatically generated, as dictated by characteristics in profiling:

Test name	Test description	Example test results
Alpha Truncation	Tests that the maximum count of characters in a column value has not dropped vs. baseline data.	Baseline_Value=7, Threshold_Value=0
Average Shift	Tests for a statistically-significant shift in the mean value for the column from the average calculated at baseline.	Baseline_Value_Ct=45706, Baseline_Avg=2513.885, Baseline_SD=1451.376, Threshold_Value=0.5
Constant Value Present	Tests that all values in the column match the constant value identified in baseline data.	Baseline_Value=45, Threshold_Value=0
Daily Record Count	Tests for the presence of every calendar date within the min/max date range, per baseline data.
Decimal Truncation	Tests for decimal truncation by confirming that the sum of fractional values in data is no less than the sum at baseline.	Threshold_Value=22623.810
Distinct Date Count Change	Tests that the count of distinct dates referenced in the column has not dropped vs. baseline data.	Baseline_Value=45706, Threshold_Value=45706
Distinct Value Change	Tests for any change in the distinct values present for a column from baseline.
Distinct Value Count Change	Tests that the count of unique values in the column has not changed from baseline.	Baseline_Value_Ct=2, Threshold_Value=2
Email Format	Tests that non-blank, non-empty email addresses match the standard format.	Threshold_Value=0
Future Date	Tests that the maximum date referenced in the column is no greater than the test date, consistent with baseline data.	Threshold_Value=0
Future Date One Year	Tests that the maximum date referenced in the column is no greater than one year beyond the test date, consistent with baseline data.	Threshold_Value=0
Incremental Average Shift	Tests for a statistically-significant shift in the mean of new values for columns compared to the average calculated at baseline.	Baseline_Value_Ct=45706, Baseline_Sum=114899627.812, Baseline_Avg=2513.885, Baseline_SD=1451.376, Threshold_Value=2
Minimum Date	Tests that the earliest date referenced in the column is no earlier than the baseline data.	Baseline_Value=1993-02-21 00:00:00, Threshold_Value=0
Minimum Value	Tests that the minimum value present in the column is no lower than the minimum value in baseline data.	Baseline_Value=0.11, Threshold_Value=0
Missing Percentage	Tests for a statistically-significant shift in percentage of missing values in the column vs. baseline data.	Baseline_Ct=45707, Baseline_Value_Ct=45705, Threshold_Value=2
Monthly Record Count	Tests for the presence of at least one date per calendar month within the min/max date range, per baseline data.	Threshold_Value=0
Outlier Percentage Above	Tests that the percent of outliers over 2 SD above the mean doesn’t exceed the threshold.	Baseline_Avg=33.6726190476191, Baseline_SD=77.6098545076732, Threshold_Value=0.05
Outlier Percentage Below	Tests that the percent of outliers over 2 SD below the mean doesn’t exceed the threshold.	Baseline_Avg=33.6726190476191, Baseline_SD=77.6098545076732, Threshold_Value=0.05
Pattern Match	Tests that all values in the column match the same alpha-numeric pattern identified in baseline data.	Baseline_Value=[0-9][0-9][0-9][0-9][0-9], Threshold_Value=0
Recency	Tests that the latest date in the column is within a set number of days of the test date.	Threshold_Value=14
Required	Tests that a non-null value is present in each record for the column, consistent with baseline data.	Threshold_Value=0
Row Count	Tests that the count of records has not decreased from the baseline count.	Baseline_Ct=45707, Threshold_Value=45707
Street Address Pattern	Tests for the percent of records matching the standard street address pattern.
Unique	Tests that no values for the column are repeated in multiple records.	Threshold_Value=0
Unique Percentage	Tests for a statistically-significant shift in the percentage of unique values vs. baseline data.	Baseline_Unique_Ct=54, Baseline_Value_Ct=45705, Threshold_Value=0.5
US State	Tests that the recorded column value is a valid US state.	Threshold_Value=0
Value present in List-of-Values	Tests that all values in the column match the list of values identified in baseline data.	Baseline_Value=('No', 'Yes'), Threshold_Value=0
Weekly Rec Count	Tests for the presence of at least one date per calendar week within the min/max date range, per baseline data.	Threshold_Value=0

Business Rule Configurable Data Tests

Business Rule Configurable Data Tests allow you to configure data quality validation tests that can’t be gleaned automatically from prior data. The idea is to test that new versions of data conform to the rules you define. The number of Business Rule Data Tests you set up will likely be much smaller than the number of Auto Generated Data Tests, but specifically targeted to leverage your understanding of your data and address the unique requirements of the domain you are working in.

Although you have to define criteria, there are critical advantages of configuring tests in TestGen compared to writing code to develop tests. It is faster and easier to set up Business Rule Configurable Data Tests than to program custom SQL:

• Business Rule Data Test logic is already programmed, tested, and verified to work.
• Test tables can be configured and shared with business users, not database programmers, to collaborate on rules and documentation.
• Data engineers can manually add a new instance of an Auto-Generated Test Type when the business requires it.
• A Business Rule Data Test includes a custom query test: this allows you to define a custom SQL condition and include it as a standard test within the TestGen reporting framework.
• A Business Rule Data Test includes several Multi-Table Test Types.

The Multi-Table Test Types are:

• Aggregate Match: tests for matches in aggregated values distributed per one column or a combination of columns — for example, the sum of sales dollars per product per day vs. the same values in another table. (This is similar to a historical balance test.)
• Data Match: tests for the presence of values, or combinations of values, from the current table versus another table.
• Prior Match: tests for the presence of the same distinct column values, or combination of values, in a new build as are present in the same table of a prior build or different schema.
• Window Match: tests for the presence of the same column values, or combination of values, in the most recent time window of a transactional table vs. the prior time window of the same table in the same schema.

The 11 Business Rule Data Tests are as follows:

Test name	Test description
Aggregate Match No Drops	Tests that aggregate values for each set of column values have not diminished vs. a reference dataset.
Aggregate Match Same	Tests for an exact match in aggregate values for each set of column values vs. a reference dataset.
All values in the List-of-values are present	Tests that all values match a pipe-delimited list of expected values, and that all expected values are present.
Custom Expression	Custom — tests that all values for the column match a pre-defined, custom condition.
Custom Query	Custom business-rule test.
Data Match	Tests the presence of the same set of column values in a reference table.
Prior Match	Tests that the same column values are present in the current dataset as in a different, prior schema.
Relative Entropy	Tests the closeness of the match between two distributions of aggregate measures across combinations of column values, using the Jensen-Shannon Divergence test.
Row Count Percentage	Tests that the count of records is within a percentage above or below the baseline count.
Time Window Match No Drops	Tests that column values in the most recent time window include at least the same as the prior time window.
Time Window Match Same	Tests for the presence of the same column values in the most recent time window vs. the prior time window.

Summary of DataOps TestGen Profiling, Bad Data Detectors, and Data Quality Validation Testing

Capability	Count
Data profiling characteristics per column	51
Bad data detection tests	13
Automatically generated data test types	28
Business rule (configurable) data test types	11
Tests typically generated per profiled column	2–9

DataOps TestGen Profiling and Data Quality Validation Tests Organized by Data Quality Dimension

How can you assess your data quality? Data quality meets six dimensions: accuracy, completeness, consistency, timeliness, validity, and uniqueness. DataOps TestGen profiling and test results can help compare your data against these definitions of the data quality dimensions:

1. Accuracy — how well does a piece of information reflect reality?
2. Completeness — does it fulfill your expectations of what’s comprehensive?
3. Consistency — does information stored in one place match relevant data stored elsewhere?
4. Timeliness — is your information available when you need it?
5. Validity — is data in a specific format; does it follow business rules, or is it unusable?
6. Uniqueness — is this the only instance in which this information appears in the database?

Data quality dimension	Data quality validation tests	Data profiling characteristics
Accuracy	Aggregate Match No Drops, Custom Query, Incremental Average Shift, Outlier Percentage Above, Outlier Percentage Below	Average Value, Maximum Value, Minimum Value, Minimum Value Gr0, Boolean True Count, Numeric Count, Fractional Sum, Percentile 25/50/75, Standard Deviation Value
Completeness	Daily Record Count, Missing Percentage, Monthly Record Count, No Column Values Present, Potential Duplicates, Required, Row Count, Row Count Percentage, Weekly Rec Count	Filled Value Count, Null Value Count, Record Count, Value Count, Zero Value Count
Consistency	Aggregate Match Same, Average Shift, Distinct Value Change, Leading Spaces, Multiple Data Types Per Column Name, Non-standard Blank Values, Pattern Inconsistency, Prior Match, Relative Entropy, Time Window Match No Drops, Time Window Match Same	Filled Value Count, Null Value Count, Record Count, Date Count, Value Count
Timeliness	Distinct Date Count Change, Future Date, Future Date One Year, Recency	Before 1yr Date Count, Before 5yr Date Count, Within 1mo Date Count, Within 1yr Date Count, Future Date Count, Date Days Present, Date Weeks Present, Date Months Present, Maximum Date, Minimum Date
Uniqueness	Unique, Unique Percentage	Distinct Value Count, Distinct Standard Value Count, Top Frequent Values, Distinct Value Hash, Distinct Pattern Count, Top Patterns
Validity	All values in List-of-values are present, Alpha Truncation, Constant Value Present, Custom Expression, Data Match, Decimal Truncation, Distinct Value Count Change, Email Format, Invalid Zip Code Format, Minimum Date, Minimum Value, Mostly Dates In String, Mostly not null/empty/filled values, Mostly Numeric In String, Mostly One Value, Pattern Match, Street Address Pattern, Suggested Data Type, Too Many Boolean Values, US State, Value present in List-of-Values	Column Name, Column Type, Data Type Suggestion, Functional Data Type, Functional Table Type, General Type, Position, Table Name, Zero Length Count, Average Length, Maximum Length, Minimum Length, Maximum Text, Minimum Text, Lead Space Count, Embedded Space Count, Average Embedded Spaces

DataOps TestGen Architecture and Requirements

The current version of DataOps TestGen has two major components: a Docker container, which contains the software executables for the TestGen engine, and a PostgreSQL database that persists the test results, test runs, and other test information.

Component	Role
Docker container	The software executables for the DataOps TestGen engine — profiling, test generation, and test execution.
PostgreSQL database	Persists test results, test runs, test definitions, and profiling results.
Target database connection	Snowflake or Amazon Redshift, reached with supplied connectivity and credentials.
Deployment	On-premise or in DataKitchen’s cloud.

Tests and profiling may be created, edited, run, and their results reviewed via a series of command line arguments:

• execute_cat_tests_query, execute_tests_query
• gen_tests_query, profiling_query
• run_execute_cat_tests, run_execute_tests
• run_generate_tests, run_get_entities
• run_launch_db_config
• run_observability_exporter
• run_profiling_bridge, run_test_parameter_validation

DataOps TestGen may be installed on-premise or in DataKitchen’s cloud. Database connectivity and credentials are required.

DataOps TestGen currently supports two databases: Snowflake and Amazon Redshift. Support for more databases such as Databricks, Google BigQuery, Azure Synapse, and SQL Server will be released through 2023. Later in 2023, DataOps TestGen will add a User Interface. An example of that UI is shown below.

DataOps TestGen integrates with DataOps Observability to provide a complete view of your Data Journey from data source to resulting delivery to your customer across your data toolchain.

Example DataOps TestGen Use Cases

Data Production Testing

In your day-to-day analytics production, you may use various tools to load, transform, predict, and report on data. Modern data architectures often forget the most crucial part — automatically testing data during production.

Black Box ETL Testing

DataOps TestGen allows you to build credible, refinable testing when business information is limited or unreliable.

For example, you may have a legacy, black box ETL process developed long ago, with the developers reassigned or gone. You may not know the ETL process and have little or no documentation. DataOps TestGen lets you apply credible testing to dispersed, legacy data pipelines and tech stacks. It shines a light on the black box.

Understanding Distributed Data Mart Quality

Companies often have multiple databases scattered around an organization, each containing unique and vital datasets. Data engineers are sometimes tasked with building common data standards and developing comparative data quality metrics for data across databases and servers. They can also be tasked with enforcing in-place data quality standards. DataOps TestGen spans multiple data marts and databases so you can create and enforce a common set of data rules and standards. You can also maintain a common set of data test results that can help enforce in-place data standards.

Coordinating Test Definition Development with Data Stewards and the Business

One of the big challenges data engineers have is understanding what tests are needed and configuring the parameters of those tests. That knowledge is often in the hands of business users or Data Stewards. DataOps TestGen allows you to get those individuals looped in early. You can share responsibility in editing data test definitions and configuration with people who know your business context the best. You can develop fast first-pass test specs to communicate with data stewards and SMEs. You can empower non-technical people to configure tests.

Test Individual Raw Data Sources and Give Immediate Feedback

Data Engineers often have dozens — or hundreds! — of individual data providers. Many enterprises need more data tests and controls on data sources that allow data to flow through their data factory. “Hoping for the best” is not an effective manufacturing strategy. You want to catch errors as early in your process as possible. Ensure the data and artifacts you create from data are correct before you start your data integration and production process. Why? Fast feedback holds upstream data providers responsible. It prevents you from getting caught holding the bag for data errors you can’t control. DataOps TestGen allows you to screen for unanticipated problems, the unknown-unknowns in the data that comes from your many data sources. Catch the unpredictable from your data providers right away when the data lands.

Multi-Layered Database Structures

Some data teams today organize their data warehouse or lake into L1, L2, and L3 layers. The layers generally refer to different stages of data processing, storage, and access. The naming and specifics of these layers can vary somewhat depending on the context and the specific data warehousing approach, but here is a generalized description:

• L1 — Operational Data Layer (ODL): the Source layer.
• L2 — Integration Layer (IL): the Harmonization or Transformation layer. Data from different source systems are cleaned, transformed, and integrated.
• L3 — Access Layer (AL): the Presentation or Information layers. This layer is where business users interact with data, often through reporting, dashboarding, and analytics tools.

Data Teams have specific challenges in understanding and testing data in each warehouse layer. They want to ensure that each layer has not lost or mangled data during its process layers. They must ensure that data properties match business logic at each processing layer. DataOps TestGen allows you to create a test that spans each layer of the database, ensuring consistency and accuracy at each level.

DataOps TestGen As Part Of Observing Your Entire Data Journey

You have many data tools: Talend, Azure Data Factory, DataBricks, custom tools, custom testing tools, ETL tools, orchestrators, data science tools, dashboard tools, databases, bucket stores, servers, and more. Those tools work together to take data from its source and deliver it to your customers. We call that multi-tool process a ‘Data Journey.’ The Data Journey is about observing what you have done, not changing your existing data estate. Data Journeys track and monitor all levels of the data stack, from data to tools to servers to code to tests across all critical dimensions. It supplies real-time statuses and alerts on start times, processing durations, test results, and infrastructure events, among other metrics. With this information, you can know if everything ran on time and without errors, and immediately detect the parts that didn’t.

DataOps Observability provides the Data Journey abstraction, expectations, alerts, and analysis. DataOps TestGen provides in-database data testing results shared with DataOps Observability. DataOps Automation provides tool, model, and API-level testing shared with DataOps Observability. Their relationship is shown below.

Why DataKitchen’s DataOps TestGen?

What is the goal of DataOps TestGen? Automatically generate data quality checks that identify errors in your data before your customer finds them. Efficiently test data during the production process directly in the database. Allow data engineers to understand their data and detect problematic data records. Give data engineers room to grow from automatic data tests to a list of configurable ‘fill in the blank’ data tests that require business context.

So what makes DataOps TestGen a sustainable framework?

• Simple to use — doesn’t add complexity or clog up mental bandwidth.
• Self-documenting — encourages transparency and collaboration.
• Extensible — responds to changing needs without hard coding.
• Instantiates a plan of action — not just reporting.
• Refinable over time — separates signal from noise.
• Minimal impact on production performance.

DataKitchen is the Complete Toolkit For Testing All Aspects of Your Data Journey

Errors in production don’t just happen in your data; they can take place anywhere on the Data Journey from data source to value delivery. Data Journeys represent the myriad paths data take from source to the insight value you deliver to your customer. Data Journeys track and monitor all levels of the data stack, from data to tools to code to tests across all critical dimensions.

While DataOps TestGen focuses on the critical piece of testing data in your database, it is always important to test every step of the Data Journey:

1. Test raw data.
2. Test integrated data.
3. Test the data changed by your tools — reports, models — acting upon data.
4. Test the entire orchestrated process across all your tools and data stores from source to value.
5. Test in production and development.

DataKitchen has a complete software solution for testing and observing every part of your Data Journey:

Task	DataKitchen tool
Data profiling and bad data detection testing	DataOps TestGen: 51 characteristics, 13 tests
Automatically generated (from data profile) tests	DataOps TestGen: 28 tests
User configurable test templates	DataOps TestGen: 11 tests
Complex SQL, Python, or language containerized tests	DataOps Automation
Tool (ETL, BI, data science) API data testing	DataOps Automation
Overall Data Journey process monitoring, alerting, and analytics	DataOps Observability

Additional Resources

• DataOps Observability – Principles and Ideas
• DataOps Observability – Technical Product Overview
• DataOps Observability and Automation Software
• DataOps TestGen Documentation
• Install DataOps TestGen today — free and open source, running on a laptop in minutes

Related Reading

• DataOps TestGen White Paper — the problem this product was built for, told as ‘the mystery box full of data errors’
• A Guide to DataOps Tests — the test taxonomy behind the generated tests
• Data Quality: The DataOps Way — the method that surrounds the tooling
• The Syntax, Semantics, and Pragmatics Gap in Data Quality Validation Testing — why generated tests can only cover the first layer
• The 2026 Open Source Data Profiling Software Landscape — where profiling tools stand now
• Scaling Data Reliability: The Definitive Guide to Test Coverage — how much testing is enough
• Test Coverage Beats Data Lineage — why a static blueprint is not a smoke detector
• The Seven Deadly Sins of Data Quality — the habits that leave databases untested
• Announcing DataOps Data Quality TestGen 3.0 — where the product went after this paper

---

FAQ