DataOps TestGen Software

‘The Mystery Box, Full of Data Errors’ – how can you fix it?

 

Say goodbye to the complexity of writing data quality validation tests yourself. DataOps TestGen takes care of that for you, automating the terms and conditions of your data contract through simple, automatically generated data test creation, execution, and profiling.

TestGen UI  - test results

DataOps TestGen delivers simple, fast data quality test generation and execution. DataOps TestGen profiles your data and algorithmically generates dozens of data validations. Your Data Journey starts with verifying that you can trust your data and re-verifying it every time you refresh it in production.

Twenty-Eight Simple, Fast, Automatically Generated Data Quality Checks

Data Engineers don’t need detailed knowledge of your enterprise data or customer needs.  Auto-test generation means you can start quickly and easily.

Eleven Best Practice, Business Context Data Tests Configured With Fill-In-The-Blank Simplicity

TestGen UI -- Profiling

Best practice data validations allow your team to scale as customers and data complexity increase. Data Stewards can help adjust tests or add new business tests.

Fifty-One Data Profiling Characteristics Collected With Thirteen Bad Data Identification Tests

TestGen UI -- Profiling

Data Engineers get an understanding of the characteristics of every column of data. You can identify prominent problem rows of data before your production begins.

Efficient, Understandable In-Database SQL Test Execution

not displayed

51 Data Profiling Column Characteristics

Data profiling is the periodic X-ray of tables in a database to gather extensive information about the contents of each column. Results are stored in a standard table in DataOps TestGen. This table is available for direct review and is used for rules derivation downstream. Examples include:
• Averages
• Column & Table Types & Names
• Date Characteristics
• Min/Max Value
• Numeric Counts:
• Percentiles
• Positions
• Unique Values

28 Auto-Generated Data Tests

The goal of Automatically Generated Data Tests is to cast a wide net for data problems that can’t be predicted by targeted testing devised in advance.  It’s the same way you might set up a burglar alarm in your home by deploying sensors at all possible entrances to catch a burglar who would only try one window.  Your goal in refining these tests is to maintain maximum sensitivity to real problems while minimizing false positives that are not worth the follow-up.   Examples of Test Are:

  • Alpha Truncation
  • Average Shift
  • Constant Value Present
  • Daily Record Count
  • Value present in List-of-Values
  • Distinct Value Change
  • Value present in List-of-Values
  • Future Date
  • Incremental Average Shift

 

13 Bad Data Detector Tests

Once data profiling is complete, Bad Data Detection Tests can be run. The Bad Data Detection Tests automatically confirm how closely data structures and assumptions match the actual contents of each column. Results can be used to assist the Data Engineer in refining data structure definitions and target the addition of data ‘patching’ steps which help to generate a more usable, analyzable dataset.  Examples Include:

  • Invalid Zip Code Format
  • Leading Spaces
  • Mostly Dates In String
  • Mostly not null, empty, or filled values.
  • Multiple Data Types Per Column Name
  • No Column Values Present
  • Non-standard Blank Values

 

11 Business Rule Data Tests

Business Rule Configurable Data Tests allow you to configure data quality validation tests that can’t be gleaned automatically from prior data. It is faster and easier to set up Business Rule Configurable Data Tests than to program custom SQL. Business Rule, Data Test logic is already programmed, tested, and verified to work. To collaborate on rules and documentation, they can be configured and shared with business users, not database programmers.  Examples include:

  • Data Match
  • Prior Match
  • Aggregate Match No Drops

Complete Coverage.  No data duplication because tests are run in your production database quickly and with low impact.  You can understand the test queries clearly.

Simple To Use, Self-Documenting, And History Make Refinements Over Time Easy

Tests are run along with your existing tools simply in production testing can be integrated with your ETL or workflow tool easily. Learn and improve the analytics over time.

Extends To Changing Data And Business Customer Requests

TestGen UI -- Profiling

As your data complexity and customer expectations grow, DataOps TestGen grows with you.  Bundled tests meet most of your testing needs.  Test results and profiling history make refining tests a breeze.  Integrates to DataOps Observability to help observe your entire Data Journey.

DataOps TestGen Is Part of a Suite of Data Observability Products

DataKitchen provides software to observe and automate every data journey in an organization, from source to customer value, in development and production, so that teams can deliver insight to their customers with few errors and a high rate of new insight creation.

Our software allows data and analytic teams to observe, test, and automate the tools, data, processes, and environments in their entire data analytics organization, providing massive increases in quality, cycle time, and team productivity.

Want to See Our DataOps Observability, TestGen, and Automation Software Products in Action?

 

Webinar

Data Quality Testing Techniques Webinar »

Monitor every Data Journey in an enterprise, from source to customer value, in development and production.

Simple, Fast Data Quality Test Generation and Execution. Your Data Journey starts with verifying that you can trust your data.

Orchestrate and automate your data toolchain to deliver insight with few errors and a high rate of change.

dataops-cookbook-download
recipes for dataops success

DataOps Learning and Background Resources


DataOps Journey FAQ

DataOps Observability basics

Data Journey Manifesto

Why it matters!

DataOps FAQ

All the basics of DataOps

DataOps 101 Training

Get certified in DataOps

Maturity Model Assessment

Assess your DataOps Readiness

DataOps Manifesto

Thirty thousand signatures can't be wrong!

 

DataKitchen Basics


About DataKitchen

All the basics on DataKitchen

DataKitchen Team

Who we are; Why we are the DataOps experts

Careers

Come join us!

Contact

How to connect with DataKitchen

 

DataKitchen News


Newsroom

Hear the latest from DataKitchen

Events

See DataKitchen live!

Partners

See how partners are using our Products

 

DataKitchen DataOps Consulting Services


Product Implementation

We help you succeed with DataKitchen's products.

Commercial Pharma Data Engineering

We build, operate, train, and transfer data products.

DataOps Transformation Advisory

We help you bring DataOps to your organization.

Customer Data Platform Engineering

We build an open CDP, then transfer it to you.