How This Research Started (And Why We’re Telling You)
DataKitchen co-founder Gil Benghiat recently posted a question to r/dataengineering — one of the largest and most active data engineering communities on the internet. The question was genuine: “What is actually stopping teams from writing more data tests?”
The problem? He didn’t disclose who he was. Within hours, a lead data engineer named SearchAtlantis spotted that just one week earlier, Gil had posted in a separate thread and explicitly said: “full disclosure, I co-founded DataKitchen and we built TestGen for exactly this problem.” The callout got 102 upvotes. The replies were swift and merciless. “His picture tells a thousand words,” wrote another commenter. The community, as it tends to do, did not let it slide.
To Gil’s credit, he acknowledged it directly: “That is fair. I co-founded DataKitchen and should have said so here. The question is still genuine, though.”
And it was genuine. The thread went on to generate over 60 thoughtful comments from working data engineers, and the insights that emerged are some of the most candid and practically useful we’ve seen on the topic. We’re sharing them here — with full transparency about their source.
Lesson learned. On to the findings.
What’s Actually Stopping Teams From Writing Data Tests?
1. Writing Good Tests Is Genuinely Hard
The most upvoted response in the entire thread cut straight to the heart of it: writing tests that catch real bugs without being noisy is just difficult. This isn’t false modesty — it’s a real skill gap that takes time and painful experience to develop.
The problem isn’t the mechanics of writing a test. It’s knowing which tests are worth writing. Teams that skip this step and add tests just to “check a box” often end up worse off: tests that fire on irrelevant conditions, break jobs for illegitimate reasons, and eventually get deleted. Bad tests erode trust in the whole testing practice. Once engineers stop believing their alerts mean something, the entire system collapses.
One engineer described the downstream effect plainly: “I tried implementing a basic testing practice at my workplace… my manager insisted we need some version of unit testing on every table too as part of the policy. Just adding tests for the sake of claiming we test has been unnecessary extra work… then when those tests eventually fail and break a job it turns out we actually don’t care or the dev never verified the rule was legitimate in the first place.”
The compounding problem: DAG timing makes this worse. When pipelines have multiple arms refreshing on different cadences, false positives multiply. Engineers start ignoring alerts. And once alerts get ignored, you’ve lost the whole point.
2. No One Has Thought Through What Happens When a Test Fails
One of the most insightful comments in the thread reframed the entire problem: “Writing tests isn’t the hard part. It’s deciding the operational playbook for each of the tests that gets tricky.”
The commenter laid out the question chain that most teams never work through:
- You have 100 tests. Which ones stop the pipeline when they fail?
- For those who don’t stop the pipeline, what do you do with the results?
- Is anyone actually reading the report sent for every pipeline run?
- How do you decide which non-critical failing test deserves action?
Without answers to these questions, tests become noise. And in the absence of a clear playbook, teams default to not writing them at all. It’s not laziness — it’s rational risk avoidance.
Gil’s own response in the thread is worth noting here: “Not all tests are equal. Some should stop the pipeline, some should be warnings that are investigated later or documented as release notes for data consumers, and others are more metric tracking than tests — a way for the data engineer to stay calibrated on how the data behaves. If a test is constantly failing and nobody acts on it, then remove it.”
3. Feature Delivery Always Wins
At least four separate commenters called this out directly, and it’s probably the most universal barrier on the list. During a sprint, tests lose. Every time.
The dynamic is simple: delivery has a deadline, stakeholders, and visibility. Testing has none of those things — until something breaks in production and suddenly it has all three.
One senior data engineer summed it up bluntly: “From what I’ve seen, it is more of that last one. No one cares, there’s too much work and too little time to do it in. Just deploy and if there are bugs, take them up in subsequent sprints.”
This isn’t just an individual failure. It’s an organizational one. When teams are too small for the workload, testing is the first thing to get cut. And data testing gets cut silently, without anyone explicitly making that decision.
4. Tests Rot — And Nobody Budgets Time to Maintain Them
Even teams that invest in testing often find that investment decays. Upstream schemas change. Business logic shifts. A vendor updates their API without warning. Tests that were accurate six months ago now fire constantly — or, worse, silently pass bad data because the world has changed around them. One commenter described it well: “a lot of teams don’t have stable expectations, schemas and upstream logic shift, so tests rot fast unless you budget time to maintain them.”
Maintenance rarely gets prioritized. It doesn’t appear on the roadmap, doesn’t get sprint points, and doesn’t have a deadline. So tests slowly rot, confidence erodes, and eventually someone deletes the whole suite and starts over — or doesn’t bother starting over at all.
One commenter captured the fatigue well: teams that have been burned by upstream changes enough times learn to write schema validation and bail early rather than invest in elaborate fixtures, because fixtures require their own maintenance, which requires its own tests — a matryoshka doll of testing infrastructure that most teams don’t have the bandwidth to sustain.
5. You Don’t Know What to Test Until It’s Already Broken
Data testing knowledge is largely reactive. Most engineers learn what to test by getting burned — a pipeline that produced zero rows, a join that silently dropped 40% of records, a schema change from upstream that nobody announced.
Until you’ve lived through a few of those, it’s hard to know where the landmines are. And without that experiential knowledge, tests tend to be shallow — row counts, maybe nulls — and miss the things that actually matter.
One commenter put it plainly: “Lack of experience. Those of us who have been burned like you were with that four-hour pipeline that did nothing write data tests.” Another echoed it: “You don’t realize you need them until you’ve scrambled to get shit fixed cuz things are broken and the heat is on and it’s embarrassing. That’s when you realize oh wow it doesn’t have to be like this.”
The deeper version of this problem is domain knowledge. One commenter observed that they’d seen teams with hundreds of tests still miss “revenue went to 0”—because nobody on the data team understood the business well enough to recognize that it was a possible failure mode worth testing for. As Gil put it in response: “Many data teams just move data and don’t understand the business.” Technical competence and business context are both required, and the latter is often missing.
6. Nobody Senior Enough Cares — Until It’s Too Late
Code breaks loudly. A build fails, a deploy errors out, and an exception gets thrown. Data failures are different. A pipeline that produces garbage rows completes successfully. The job turns green. No alert fires. The wrong numbers sit quietly in a dashboard until someone notices them in a meeting — usually the wrong meeting, at the wrong time, in front of the wrong people.
One commenter nailed the asymmetry: “Code breaks loudly. A pipeline that silently produces garbage? That one sneaks through.”
Because data failures are invisible until they’re embarrassing, there’s rarely organizational pressure to prevent them in advance. Leadership doesn’t feel the cost of missing tests until a crisis makes it undeniable. And by then, the damage is done.
7. Catching the Error Makes It Your Problem
This one is uncomfortable but widely recognized. One commenter articulated it clearly: “If you caught an error, it turns into your problem. If you delivered wrong data where the root cause is not the pipeline, it is not your problem.”
The incentive structure in many organizations actively punishes engineers who look too hard. Finding a problem means owning the investigation, the fix, the communication to stakeholders, and the post-mortem. Not finding it means the problem is the upstream’s fault.
Rational engineers in irrational systems respond accordingly.
Gil’s take: the only fix is leadership that actively rewards people who surface problems rather than penalizing them for finding them. This is ultimately a culture-and-values issue, not a tooling one.
8. At Scale, Comprehensive Testing Must Fit
One commenter described processing close to an exabyte of data per day. At that volume, full validation at every stage is too slow, too expensive, and practically impossible. Their solution — sampling, anomaly detection, and back-testing on pipeline deployments — is the pragmatic answer to a genuine constraint. As they put it: “At our volume something that is 1 in a billion will happen several times a day, so if it can happen, it will happen — plan for the possibility.”
This is worth naming clearly: for some teams, the barrier to comprehensive testing isn’t skill or culture. It’s physics and economics. The right answer at hyperscale looks very different from the right answer at a mid-market company running 50 pipelines.
The Pattern Underneath All of It
Across every comment in the thread, one pattern emerges consistently: data testing is treated as optional until a crisis makes it mandatory.
Most teams build test coverage reactively — one incident at a time — rather than proactively as a standard engineering discipline. The barriers are a mix of technical difficulty, organizational culture, misaligned incentives, and resource constraints. Tooling is rarely the limiting factor.
Gil’s own closing summary of the thread named six core impediments:
- Test noise — false positives erode trust in the entire suite
- Time pressure — feature delivery consistently wins over quality investment. No one has time to write tests.
- Scale — at high volume, writing and maintaining comprehensive testing becomes impractical
- Test rot — upstream changes break tests, and maintenance is never prioritized
- Perverse incentive — catching errors makes them your problem
- Domain knowledge gaps — without business context, it’s hard to know which tests actually matter
These aren’t problems that disappear when you hand a team a testing tool. They require investment in process, culture, and clear operational norms around what testing means and what happens when it fails.
We’re Data Engineers Too — This Is Why We Open-Sourced TestGen
If you’ve read this far, you’ve probably recognized your own team in at least a few of these barriers. We did too — and it’s exactly why we decided to open source a full-featured version of TestGen.
The most common reasons teams don’t test aren’t technical. They’re about time, trust, incentives, and not knowing where to start. A tool sitting behind an enterprise paywall doesn’t fix any of that. Giving engineers direct, no-commitment access to a production-ready testing tool — one that profiles your data and generates tests automatically, so you’re not starting from a blank file — actually does.
TestGen is free to download, free to use, and built to address the specific gaps this community named: it reduces the expertise barrier for knowing what to test, cuts the time cost of writing tests from scratch, and gives teams a structured framework for deciding which tests are critical, which are warnings, and which are just calibration signals. We’ve been doing data engineering for a long time, and have run into every issue listed in the article. And we built TestGen as th free antidote.
For teams that want to go further — shared test suites, collaboration across engineers, and enterprise support — our flat-rate pricing means you’re not getting hit with per-user or per-usage fees that scale against you as your team grows. One predictable number, regardless of how many tables, pipelines, or tests are involved.
Get started with open source TestGen →
Because the community was right: the problem is real, it’s widespread, and it’s been undersolved for too long.
Source: r/dataengineering thread, “What is actually stopping teams from writing more data tests?” — community responses plus OP follow-up from DataKitchen co-founder Gil Benghiat.
The Bottom Line
The Reddit community named eight specific reasons data testing doesn’t happen. TestGen was built to dismantle all eight — through one-button automated test generation, in-database execution, ML-based anomaly detection, dataset screening, shareable issue reports, quality scoring, and flat-rate pricing that doesn’t punish scale. It is Apache-2.0-licensed and available as open source, with no feature gates or usage limits. For teams that want shared scorecards, cross-engineer collaboration, and enterprise support, pricing remains predictable regardless of the number of tables, pipelines, or tests involved.
FAQ
What are the key points in this blog?
We asked r/dataengineering why data engineers do not test more and got a candid answer. The community named eight barriers, and most are human and organizational rather than technical: writing good tests is hard, nobody has a playbook for failures, feature delivery always wins, tests rot, you do not know what to test until it breaks, failures are invisible, finding an error makes it yours, and thorough testing does not scale.
Why don’t data engineers write more tests?
The top-voted reason was that writing tests which catch real bugs without generating noise takes expertise most teams do not have. After that came the absence of a playbook for what to do when a test fails, and the fact that feature delivery beats testing every sprint. Tooling was rarely the complaint; time, expertise, and ownership were.
What happens when nobody knows what to do when a data test fails?
Tests get muted. Without an agreed answer to which failures stop a pipeline, which page someone, and which are logged and ignored, every alert becomes a judgment call. Judgment calls are expensive, so people stop making them and start filtering the channel. That is how a working test suite becomes decorative without anyone deciding to abandon it.
Why do data tests rot over time?
Because they are written against a snapshot of a schema that keeps moving. Upstream adds a column, changes a type, or renames a field, and tests written for yesterday’s structure start failing for reasons that have nothing to do with data quality. Suites decay fastest in exactly the environments that change most, which are the ones that needed testing.
Why are data failures invisible until they are embarrassing?
Because bad data does not throw an error. Code breaks loudly; a pipeline producing garbage completes successfully and turns green. The wrong numbers sit quietly in a table until somebody makes a decision on them. That asymmetry with software engineering is why data teams underinvest in testing relative to how much damage the failures cause.
Why does catching an error make it your problem?
Because in many organizations whoever surfaces a data issue inherits responsibility for fixing it, regardless of who caused it upstream. That incentive quietly discourages looking. It is the sharpest observation in the whole thread, and it explains behavior that otherwise looks like carelessness: not testing is rational when finding something costs you your week.
