What Is DataOps Observability?

Why DataOps Observability?

Problem #1:  Many tools and pipelines – too many errors and delays:

• No enterprise-wide visibility of 100’s, 1000’s tools, pipelines, and data sets
• No end-to-end quality control
• Hard to diagnose issues
• Errors in data and elsewhere create distractions that limit new insight development

Problem #2:  Data and analytic teams have friction fixing these problems:

• Very Busy: teams are already busy and stressed – and know they are not meeting their customer’s expectations.
• Low Change Appetite: Teams have complicated in-place data architectures and tools (which may include other logging and observability technologies). They fear change in what is already running.
• No single pane of glass: no ability to see across all tools, pipelines, data sets, and teams in one place.
• Teams don’t know where or how to check for data or artifact problems.
• Teams see a lot of blame and shame without shared context to see and diagnose problems in real-time.

What problem is DataOps Observability trying to solve?

Many companies are stuck in a “hope and pray” culture that their changes and integrations won’t break anything. They practice “firefighting” when things inevitably go wrong. They wait for customers to find problems. They blindly trust their providers to deliver good data on time without changing data structures. They interrupt the daily work of their best minds to chase and fix a single error in a specific pipeline. They do not know if the other thousands of data pipelines and tasks are failing. 

It’s a culture of productivity drains, which results in customers losing trust in the data. Another result is rampant frustration within data teams. A survey of data engineers conducted by DataKitchen in 2022 revealed some shocking statistics: 78% feel they need to see a therapist, and a similar number have considered quitting or switching careers. The industry is experiencing a shortage of people who do data work due to a supply problem and the exodus of people leaving the field because it’s so stressful.

Teams do not have visibility across the entire data journey from source to value and in-depth diagnosis of data, hardware, code, or software problems to find and analyze problem impact.

What is a Data Journey?

The data journey is about observing what you have done, not changing your existing data estate. Data Journeys track and monitor all levels of the data stack, from data to tools to code to tests across all critical dimensions. It supplies real-time statuses and alerts on start times, processing durations, test results, and infrastructure events, among other metrics. With this information, you can know if everything ran on time and without errors and immediately identify the specific parts that didn’t. Journeys provide a context for the many complex elements of a pipeline.

Why is the ‘modern data stack’ so complicated?

The industry is replete with data, automation, and data science / analytic tools. But none of these tools fully addresses the problem. You have to monitor your entire data estate and all the reasons why pipelines may succeed or fail, not just servers, load testing, and testing individual tools and transactions. It is a $60 Billion a year industry with hundreds of vendors.

​https://a16z.com/2020/10/27/blueprints-for-data-infrastructure-lakehouse-storage-data-is-not-the-new-oil-more/

How does ‘DataOps Observability’ differ from ‘Data Lineage’?

Data lineage answers the question, “Where is this data coming from, and where is it going?” It is a way to describe the data assets in an organization. It is a description used to help data users understand where data came from and, with a data catalog, the content of specific data tables or files. Data Lineage does not answer other key questions, however. For example, data lineage cannot answer questions like: “Can I trust this data”? “And if not, what happened during the data journey that caused the problem??” “Has this data been updated with the most recent files”? Understanding your data journey and building a system to monitor that complex series of steps is an active, action-oriented way of improving the results you deliver to your customers. DataOps Data Journey Observability is ‘run-time lineage.’ Think of it this way: if your house is on fire, you don’t want to go to town hall and get the blueprints of your home to understand better how the fire could spread. You want smoke detectors in every room so you can be alerted quickly to avoid damage. Data Lineage is the blueprint of the house; a Data Journey is the set of fire detectors sending you signals in real-time. Ideally, you want both run time and data lineage.   

Why should I care about errors and bottlenecks in the Data Journey?

You have a significant investment in your data and the infrastructure and tools your teams use to create value. Do you know that it’s all working correctly, or do you hope and pray that a source data change, code fix, or new integration won’t break things? If something does break, can you find the problem efficiently and quickly, or does your team spend days diagnosing the issue? Do you fear that phone call or email from an angry customer who finds the problem before you do? How can you be confident that nothing will go wrong and that your customers will continue to trust your deliverables? Do you dread that first-morning work email and the problems it could bring?

Don’t current IT Application Performance Monitor tools do this already?

To see across all the journeys that data travels and up/down the tech and data stack requires a meta-structure that goes beyond typical application performance monitoring and IT infrastructure monitoring software products. While quite valuable, these solutions all produce lagging indicators. For example, you may know that you are approaching limits on disk space, but you can’t say if the data on that disk is correct. You may know that a particular process has been completed, but you can’t see if it was completed on time or with the correct output. You can’t quality-control your data integrations or reports with only some details.

Since data errors happen more frequently than resource failures, data journeys provide crucial additional context for pipeline jobs and tools and the products they produce. They observe and collect information, then synthesize it into coherent views, alerts, and analytics for people to predict, prevent, and react to problems.

What are the components of a solution for DataOps Observability?

1. Represent the Complete Data Journey: Be able to monitor every step in the data journey in-depth within tools, data, and infrastructure and over complex organizational boundaries.
2. Production Expectations, Data and Tool Testing, and Alerts:  Be able to set time, quality control, and process step order rules/expectations with proactive (but not too noisy) push notifications.
3. Development Data and Tool Testing: validate your entire data journey in the development process. Enable team member to ‘pull the pain forward’ to increase the delivery rate and lowers the risk of deploying new insight.
4. Historical Dashboards and Root Cause Analysis: store data over time about what happened to every data journey to learn from your mistakes and improve
5. A User Interface Specific for Every Role: Easy to understand, the role-based UI allows everyone on the team — IT, managers, data engineers, scientists, analysts, and your business customer to be on the same page.
6. Simple integrations and an open API: a solution should include pre-built, fast, easy integrations, and an open API drives quick integrations without replacing existing tools.
7. Monitor Costs and end-user usage tracking:  Be able to include specific cost items (e.g., server costs) as part of your data journey. Likewise, be able to monitor user usage data for tables and reports. This data can help your tell if a data journey’s costs outweigh its benefits.
8. Accelerators and Automators:  team wants to start quickly with understanding their data journey and creating a set of data tests and expectations that can tell them where things are going wrong. DataOps observability should be able to generate =, automatically a base set of data tests and expectations so that team can get value quickly.  

Who cares about DataOps Observability?

Data and analytic teams and their leaders (CDOs, Directors of Data Engineering / Architecture / Enablement / Science). Second, small data teams that develop and support customer-facing data and analytic systems. Data teams that care about delivering insight to their customers with no errors and a high rate of change. Finally, any group wanting to work with less embarrassment, hassle, and more time creates original insight.

How does this compare with Data Quality?

Many enterprises have few process controls on data flowing through their data factory. “Hoping for the best” is not an effective manufacturing strategy. You want to catch errors as early in your process as possible. Relying upon customers or business users to catch the mistakes will gradually erode trust in analytics and the data team. 

A sole focus on source data quality does not fix all problems. In governance, people sometimes perform manual data quality assessments. These labor-intensive data quality evaluations are done periodically, providing a snapshot of quality at a particular time. DataOps Observability which focuses on lowering the rate of errors ensures continuous testing and improvement in data integrity. DataOps Observability works 24×7 to validate the correctness of your data and analytics journey.

Our source data is in excellent shape, so we don’t need DataOps Observability, right?

It’s important to check data, but errors can also stem from problems in your workflows, end-to-end toolchain, or the configuration/code acting upon data. For example, correcting data that is delivered late (a missed SLA) can also be a big problem. The DataOps Observability approach takes a birds-eye view of your data factory and attacks errors on all fronts.

We do a lot of manual QC checks in development, so we don’t need DataOps Observability.

Avoid manual tests. Manual testing is performed step-by-step by a person. Manual testing creates bottlenecks in your process flow. Manual tests are expensive as it requires someone to create an environment and run tests one at a time. It can also be prone to human error. Automated testing is a significant pillar in DataOps Observability. Build automated testing into the release and deployment workflow. Testing proves that analytics code is ready to be promoted to production. Most of the tests used to validate code during the analytics development phase are promoted into production with the analytics to verify and validate the operation of the data journey.

How is DataOps Observability different from Data Observability?

Data Observability tools test data in the database. While this is a fine thing, DataKitchen has been promoting the idea of tests for many years (and in our DataOps Automation Product!). You need to correlate that information with other critical elements of the data journey – where a fundamental understanding is required.

What Is DataOps?

DataOps is a collection of technical practices, workflows, cultural norms, and architectural patterns that enable:

• Rapid innovation and experimentation delivering new insights to customers with increasing velocity
• Extremely high data quality and very low error rates
• Collaboration across complex arrays of people, technology, and environments
• Clear measurement, monitoring, and transparency of results

Where can I learn more about DataOps Observability?

You’ve come to the right place! A great place to start is to read The DataOps Cookbook. Other useful resources include:

• [Product] DataOps Observability Software
• [Webinar] Mission Control For DataOps
• [Conference] MIT CDO Session on DataOps Observability
• [Blog] DataOps Observability
• [Learn More] What is DataOps?