DataOps TestGen Software
‘The Mystery Box, Full of Data Errors’ – how can you fix it?
Say goodbye to the complexity of writing data quality validation tests yourself. DataOps TestGen takes care of that for you, automating the terms and conditions of your data contract through simple, automatically generated data test creation, execution, and profiling.
DataOps TestGen delivers simple, fast data quality test generation and execution. DataOps TestGen profiles your data and algorithmically generates dozens of data validations. Your Data Journey starts with verifying that you can trust your data and re-verifying it every time you refresh it in production.
Twenty-Eight Simple, Fast, Automatically Generated Data Quality Checks
Data Engineers don’t need detailed knowledge of your enterprise data or customer needs. Auto-test generation means you can start quickly and easily.
Eleven Best Practice, Business Context Data Tests Configured With Fill-In-The-Blank Simplicity
Best practice data validations allow your team to scale as customers and data complexity increase. Data Stewards can help adjust tests or add new business tests.
Fifty-One Data Profiling Characteristics Collected With Thirteen Bad Data Identification Tests
Data Engineers get an understanding of the characteristics of every column of data. You can identify prominent problem rows of data before your production begins.
Efficient, Understandable In-Database SQL Test Execution
not displayed
51 Data Profiling Column Characteristics
Data profiling is the periodic X-ray of tables in a database to gather extensive information about the contents of each column. Results are stored in a standard table in DataOps TestGen. This table is available for direct review and is used for rules derivation downstream. Examples include:
• Averages
• Column & Table Types & Names
• Date Characteristics
• Min/Max Value
• Numeric Counts:
• Percentiles
• Positions
• Unique Values
28 Auto-Generated Data Tests
The goal of Automatically Generated Data Tests is to cast a wide net for data problems that can’t be predicted by targeted testing devised in advance. It’s the same way you might set up a burglar alarm in your home by deploying sensors at all possible entrances to catch a burglar who would only try one window. Your goal in refining these tests is to maintain maximum sensitivity to real problems while minimizing false positives that are not worth the follow-up. Examples of Test Are:
- Alpha Truncation
- Average Shift
- Constant Value Present
- Daily Record Count
- Value present in List-of-Values
- Distinct Value Change
- Value present in List-of-Values
- Future Date
- Incremental Average Shift
13 Bad Data Detector Tests
Once data profiling is complete, Bad Data Detection Tests can be run. The Bad Data Detection Tests automatically confirm how closely data structures and assumptions match the actual contents of each column. Results can be used to assist the Data Engineer in refining data structure definitions and target the addition of data ‘patching’ steps which help to generate a more usable, analyzable dataset. Examples Include:
- Invalid Zip Code Format
- Leading Spaces
- Mostly Dates In String
- Mostly not null, empty, or filled values.
- Multiple Data Types Per Column Name
- No Column Values Present
- Non-standard Blank Values
11 Business Rule Data Tests
Business Rule Configurable Data Tests allow you to configure data quality validation tests that can’t be gleaned automatically from prior data. It is faster and easier to set up Business Rule Configurable Data Tests than to program custom SQL. Business Rule, Data Test logic is already programmed, tested, and verified to work. To collaborate on rules and documentation, they can be configured and shared with business users, not database programmers. Examples include:
- Data Match
- Prior Match
- Aggregate Match No Drops
Complete Coverage. No data duplication because tests are run in your production database quickly and with low impact. You can understand the test queries clearly.
Simple To Use, Self-Documenting, And History Make Refinements Over Time Easy
Tests are run along with your existing tools simply in production testing can be integrated with your ETL or workflow tool easily. Learn and improve the analytics over time.
Extends To Changing Data And Business Customer Requests
As your data complexity and customer expectations grow, DataOps TestGen grows with you. Bundled tests meet most of your testing needs. Test results and profiling history make refining tests a breeze. Integrates to DataOps Observability to help observe your entire Data Journey.
DataOps TestGen Is Part of a Suite of Data Observability Products
Read More About DataOps TestGen
DataKitchen provides software to observe and automate every data journey in an organization, from source to customer value, in development and production, so that teams can deliver insight to their customers with few errors and a high rate of new insight creation.
Our software allows data and analytic teams to observe, test, and automate the tools, data, processes, and environments in their entire data analytics organization, providing massive increases in quality, cycle time, and team productivity.
Want to See Our DataOps Observability, TestGen, and Automation Software Products in Action?
