“Stick Little Thermometers in your Data Journeys”

The first step in solving your data team's pain is to observe what's happening with your data and analytics 'estate' and stick little thermometers at various points in the process.

by | Oct 27, 2022 | Blog

 

Question:  What is something the data industry is missing?

I think it’s observability-led DataOps. I’ve come to believe that we, as an industry, will not change how people build things they’ve already made. They’re already being Heroes and have pain, unhappiness, and poor results.

 

 

The first step to enlightenment

The first step in solving that pain is to observe what’s happening with your data and analytics ‘estate’ and stick little thermometers at various points in the process and measure. Have those thermometers check while your data and tools are running in production a few things, for example:

  • Did the production process start? End? Is it late?
  • Is the raw data correct?
  • Is the integrated data right?
  • Is the model still predicting accurately?
  • Is the dashboard showing accurate information?
  • Will the server run out of space?
  • And a few more things.

You are going to be surprised at what you see in that information. And that data is going to drive you and your team’s behavior.

You can’t improve what you can’t measure.

And what you will see is that those checks/tests/monitors find a variety of problems. Maybe your data providers are giving you bad data. Maybe the code your data engineer is running is faulty. Perhaps the data scientist needs to tune up their model. It could be that your Analysts messed up the Tableau dashboard. Or maybe everything is perfect, and your dashboards are not being used. So you can take them down and save some money.

 

Stick Little Thermometers in your Data Journeys

“Stick Little Thermometers in your Data Journeys” by DiffusionBee

 

So the idea of Observability first DataOps is to stick a bunch of thermometers all over your data pipelines, models, Vis, and tech stack, then measure all that stuff. Then look at the data and find where the bottlenecks and errors are. Then you get evidence to do the work to fix those problems through automation and testing.

Why Observability first DataOps? Data convinces data people. Otherwise, our experience is that people will continue to “hero out.”  They will build these systems where they rush to get something done, are afraid to change it once it’s running, and live with a constant stream of problems from their customers until frustration makes them quit.

Don’t change what you have. Just observe it, get evidence and incrementally improve.

I believe that data can convince our team members to make better decisions. For this approach to work, we need measurements throughout the entire process and an understanding of what’s happening with your data estate as it runs in production. Hence, you know when there are problems or opportunities ahead.

Sign-Up for our Newsletter

Get the latest straight into your inbox

Monitor every Data Journey in an enterprise, from source to customer value, in development and production.

Simple, Fast Data Quality Test Generation and Execution. Your Data Journey starts with verifying that you can trust your data.

Orchestrate and automate your data toolchain to deliver insight with few errors and a high rate of change.

recipes for dataops success

DataKitchen Consulting Services


Assessments

Identify obstacles to remove and opportunities to grow

DataOps Consulting, Coaching, and Transformation

Deliver faster and eliminate errors

DataOps Training

Educate, align, and mobilize

Commercial Pharma Agile Data Warehouse

Get trusted data and fast changes from your warehouse

 

dataops-cookbook-download

DataOps Learning and Background Resources


DataOps Journey FAQ
DataOps Observability basics
Data Journey Manifesto
Why it matters!
DataOps FAQ
All the basics of DataOps
DataOps 101 Training
Get certified in DataOps
Maturity Model Assessment
Assess your DataOps Readiness
DataOps Manifesto
Thirty thousand signatures can't be wrong!

 

DataKitchen Basics


About DataKitchen

All the basics on DataKitchen

DataKitchen Team

Who we are; Why we are the DataOps experts

Careers

Come join us!

Contact

How to connect with DataKitchen

 

DataKitchen News


Newsroom

Hear the latest from DataKitchen

Events

See DataKitchen live!

Partners

See how partners are using our Products