Taming Organizational Complexity in Your Data-Driven Organization with DataOps

Tobias Macey invited DataKitchen CEO Chris Bergh back onto The Data Engineering Podcast to argue a position that is unpopular with technologists: the reason analytics projects fail is not the database, the orchestrator, or the model. It is the shape of the organization that builds them. Four kinds of specialist, scattered across IT and the lines of business, each own a slice of a pipeline that has to connect end to end before anyone gets an answer. Nobody owns the seams. This page is our summary of what Chris argued. The recording, the show notes, and the transcript are Tobias’s, and they live on his site.

Four roles, and no agreement about where they sit

Chris starts with the roster rather than the technology. Most data organizations contain four buckets: data engineering, which moves data into a usable shape; data science, which applies algorithms to it; visualization and BI, which explains it to a business audience; and governance and security, which keeps it from hurting the company. What varies is not the roster, it is the reporting lines. Engineering often sits under the CIO with the source systems. The people who turn data into a decision often sit in marketing, sales, or finance. A new data science group sometimes reports to the CEO because nowhere else fits.

His observation is that you could distribute those four roles across an org chart more or less at random and find a real company using that arrangement. That is the setup for the rest of the conversation, because the arrangement is not cosmetic.

Conway’s law, and why it is not a compliment

Chris reaches for Conway’s law: a system’s design copies the communication structure of the organization that built it. When he was a software engineering manager, saying a codebase had been shaped by Conway’s law was an insult. It meant the interfaces had been drawn by whoever divided the team, not by whoever understood the problem.

The same thing happens in analytics, and nobody calls it out. Every group owns pipelines: data pipelines, model pipelines, visualization pipelines, governance pipelines, each in its own tools. Getting value to a business customer means clipping those pipelines together like Lego. The joints land wherever the org chart put them.

This is the argument our white paper on reducing organizational complexity with DataOps develops in writing, with the McKinsey distinction between institutional and individual complexity behind it. The podcast is the spoken version, and it came first.

The consequence a business person sees

Chris describes the moment the structure becomes visible to everyone else. A stakeholder cannot understand why changing a line chart to a bar chart happens while they watch, and adding one new data file takes four months. The discontinuity is not explainable without a tour of the stack, so it reads as incompetence.

Underneath the discontinuity is a real design question: where does the logic live? The same customer segmentation can be a random forest, a clustering model, a calculated field in a BI tool, an attribute on a Kimball dimension, or an if statement in a Python script. Each location has a different cost to build and a different cost to change.

Chris’s position is that logic should earn its way down the stack. Put the rule in the fastest layer available, show it to the person who asked, and only push it to the bottom once it has proved it is the rule the business actually wanted. Starting at the bottom is defensible architecture and, when the requirement turns out to be wrong, an expensive way to be wrong. He is careful that both camps have a point: the architect who only builds enterprise features is right that eight definitions of the same term is a failure, and the analyst iterating in a BI tool is right that a four-month cycle answers nothing.

Many-to-many Dev and Ops

The cleanest technical point in the episode is about DevOps, and why data teams cannot simply copy it. In software the relationship is close to one-to-one: a development team, an operations team, a defined interface between them. In a data organization it is many-to-many. There is a central ops team, there are line-of-business ops teams, and there are development teams running their own production. Self-service users are production too, whether or not anyone counts them.

So the organizational barrier DevOps broke down is not the barrier in front of data teams. Theirs is the many-to-many one, plus the centralization-versus-freedom argument, plus deployment and environment management on top.

The button

Chris is blunt about what the coordination problem costs. Change a column name in a warehouse and the only way to learn what breaks is to send email and talk to people. His summary of what technical people actually want: a button that says whether changing this thing broke other things. Not a meeting, not better ticket hygiene, a button.

That is also his answer to the complaint that organizational complexity is just how the world is. It is not a culture problem to be solved with more meetings and more documentation. Those are what you get when the system cannot answer the question, and the cost lands on your most senior person, who ends up spending nine tenths of their week in review meetings holding the whole map in their head, until they leave and take the map with them.

Test in both directions, and measure the team

Two concrete practices follow. First, automated testing in both directions: monitoring in production, where the code is fixed and the data changes underneath it, and regression and unit testing in development, where the data is fixed and the code changes. The two test sets largely overlap, and Chris puts a number on the effort: about 20 percent of the team’s time, every sprint. Our two open-source tools split along the same seam, DataOps TestGen for the data side and DataOps Observability for the production side.

Second, run the data team on data. Test coverage, deploy rate, cycle time, SLA attainment, who opens the reports, and error rate with a cause recorded against each error. He calls it hypocritical that analytics leaders rarely have a one-page dashboard about their own operation when they would never hire a software manager who lacked one. The practical value is not the dashboard, it is that a shared error log converts a territorial argument into a ranked list, which is much easier to fix. He invokes W. Edwards Deming for the underlying point: when something goes wrong, the process the team lives in is the usual cause, not a lazy person.

The recession framing, dated to when he said it

The episode was recorded in April 2020, in the first weeks of the pandemic, and Chris’s closing prediction carries that stamp. He expected speculative analytics projects to stop getting funded and expected teams to be asked what the return on fifty or a hundred data people actually is. His conclusion was that the field would have to compete on operational efficiency rather than on the next two-letter acronym. Read it as a 2020 forecast, not as a current claim.

The through-line is the one he keeps coming back to. Build the system that runs the system: the machine that makes the machine, in the phrase he borrows from Tesla. Put your strongest people on it, because productivity infrastructure is the highest-leverage work available, and put the code and configuration that drive every one of those pipelines somewhere shared and versioned. That is what DataOps means, and it is the argument the DataOps Manifesto makes in eighteen principles.

Chris returned to the show four years later to talk about what came out of it, in our post on that episode.

FAQ