On-Demand Webinar · 1 hr 23 min

Managing Stress in Data Engineering: Data Quality and Testing Techniques for Data Observability

Why 78% of data engineers wish their job came with a therapist. Chris Bergh and Chip Bloche walk through data quality and testing techniques — observability in production, regression testing in development — with examples a team can copy.

Presented by Chris Bergh, Chip Bloche

What you'll learn 6 points
  • The stress is measurable. Gartner reports 60 percent of data projects fail altogether and 87 percent of data science projects never get to production, Eckerson reports 79 percent have too many errors, and DataKitchen's own survey found 78 percent of data engineers are stressed enough to need a therapist.
  • Error rates are a bigger category than data quality. Bad data quality is one source; lateness that misses an SLA, a data processing issue, a code change that broke something, and broken artifacts such as reports, models, and governance outputs are the others. Testing is what lowers all five.
  • Tests have a dual nature and most do double duty. Tests in production monitor the accuracy and quality of raw data, integrated data, and everything built from data during the production run. Tests in development check the functionality, performance, and correctness of code acting on test data, and are the same tests reused as impact, regression, and unit tests.
  • Process QC and Data QC catch different failures. Process QC runs synchronously, shadows the application steps, traces straight back to the cause, and catches the issues you knew to look for. Data QC can run asynchronously, shadows the data architecture, analyzes aggregate results, works backwards to the cause, and catches the unexpected: aggregate trends and shifts, schedule cadences, and conditions too slow to check in line.
  • Data QC in production comes in three techniques. Qualification screening reviews source data against profiling and a data dictionary, catching numbers in alpha columns, blank value representation, and string pattern inconsistency. Business rule testing turns domain expertise into parameterized tests, and the rules become the documentation. Consistency testing derives rules automatically from a profiled baseline and can generate thousands of tests: minimum date constant, dates present each week, no shift in percent missing, distribution shift versus baseline.
  • The team rules are simple: everyone writes tests, from data engineers and data scientists to BI analysts, data stewards, and operations; write them before, during, and after development; spend roughly 5 to 10 percent of your working time on testing; parameterize tests instead of writing manual ones; and when something goes wrong, add a test so it never happens again.

Slides

55 slides

Questions from this session

What is the difference between data testing and data quality?

Data quality is one input to your error rate, not all of it. The other sources are lateness that misses an SLA, a data processing failure, a code change that broke something downstream, and broken artifacts such as reports, models, and governance outputs. Testing covers all five, which is why testing is the larger discipline.

What is the difference between Process QC and Data QC?

Process QC is a set of synchronous checks at predefined checkpoints in the pipeline. It shadows the application steps, verifies expected inputs and results, and traces a problem directly to the step that caused it, but it only catches issues you knew to look for. Data QC can run asynchronously, shadows the data architecture rather than the process, analyzes aggregate results, and catches the unexpected: trends, shifts, and slow error conditions.

What is consistency testing?

Consistency testing compares current data against a baseline built from profiling, and infers the rules and tests automatically rather than having someone write them. Typical checks include required and missing value percentages, uniqueness, list-of-values against baseline, date recency, record counts within time windows, min and max values, average shift, and distribution shift. Warnings are flags to investigate and refine, not errors in themselves.

How much time should a data team spend on testing?

About 5 to 10 percent of working time. That budget covers writing tests before, during, and after development, parameterizing them so they can be reused rather than rewritten, and adding a new test every time an incident reveals a gap. Unit tests on fake data alone are not enough to prevent regressions.

Who should write data tests?

Everyone who touches data. Data engineers test raw data and the process, data scientists test models, BI analysts test dashboards and data preparation, data stewards configure test parameters, and operations tests the production process. Specialist roles such as DataOps engineers and QA automation testers write tests too, but they are not the only ones who can.

Who should be alerted when a data test fails?

Route each error to the party who owns the thing that broke. Data engineers get problems with their data, data scientists get problems with their models, BI analysts get problems with their dashboards, and operations gets every error. Data stewards are brought in only after data engineers have inspected the failure. Classifying errors by severity keeps the routing useful instead of noisy.

Where to go next