Beyond Data Observability: Simple, Complex, and Personalized Data Journeys

00:00:00

Hello everyone. My name is Chris Bergh. I'll be your host and commentator for today's webinar. Um, and so today's webinar is called Go Beyond Data Observability, um, simple, complex and Personalized Data Journeys. Uh, and before we start, just, um, let me do a little bit of housekeeping. So again, my name is Chris, I'm CEO and head chef of DataKitchen.

Uh, the webinar is targeting about 45 ish minutes. We, we certainly won't go more than an hour. Um, when you have questions, feel free to add them to the chat window. I'll try to answer them, uh, if I have time during, um, the discussion or at end, and then slides in the video will be emailed to you, uh, this, this week.

Um, so, so let's walk through what we're gonna talk about today. So, uh, you know, buzz Lightyear talks about going to Infinity and beyond. Um, and I think the sort of theme of today's webinar is looking beyond the state of the art and data observability. And I've grouped that metaphorically into three buckets. Um, and the first bucket is looking beyond, um, data anomalies and, and problems that you find with your data.

And you need breadth and depth to find active errors. So we're gonna talk about that. The second beyond bucket is, it is beyond data pipelines. You need the support for the many patterns of how data teams assemble their tools and their organizations to deliver value. And then lastly, I think you have to look beyond data.

You actually have to look to how your customers want to understand where their data comes from and following specific, um, data patterns for specific customers. So let's get into the three topics today. So first, why, why observe data at all? Um, well, I mean, you could, you know, first of all, the raw data that we get, typically 10 or hundreds of data sources to do analytics is frankly broken.

And, um, your, our customers know it. So let's say you could ma wave a magic wand and have perfect raw data. Um, you'd still have a huge failure rate. You'd still have too many errors. You'd still have people not have data trust, and you'd still have data teams stressed. And so why is that perfect? Data isn't the answer.

It's, it's certainly a help, but we assemble perfect data into systems, out of tools and, and, um, actions in code. And, and that is a, a, a great source of error. And so we're gonna talk about the ratios and talk about why that is. And so the, that goes kind of leads into the first topic about why it's just looking at beyond, uh, data anomalies.

And so I think if we talk about the sort of state of the art of data observability, let's talk about, well, what, what is data observability? And typically, and I've, I've heard this from people talking about the five pillars of observability, and the first four are about schema, freshness, volume, and quality. And so the idea here is has your data been updated?

Um, has the schema been changed? Has the volume of your data increased? Has someone given you crappy data? And, and those are all important things.

00:05:00

Um, um, but it's not enough to prevent customer found data errors. Um, and in fact, where and when you find it is also as important as, as checking for those things. And then the, the fifth sort of pillar is data lineage. And, and we've talked, uh, in these webinars before about how data lineage is kind of a blueprint of where your data is.

And, and as a result, if you find, for instance, that someone has changed the schema on you, you can then use that to go off in lineage to find out who to blame, either from on the input side or, or, or who's gonna be affected by it. However, data lineage is, is, uh, equivalent to looking at a blueprint When you've got a fire in your house, what you really need are fire alarms.

You really need to actively check and find error errors all throughout your house. So not to say data lineage is bad, it's just to say that, that it's incomplete. So let's talk about this as a a metaphor. So let's talk about, I'm gonna use this two ideas here. One about data in place and the other in data in use.

And, and these little black box boxes here in the diagram represent data. So let's say you're working in the cloud and in Amazon, and you get some data from an FTP files file source, and you put it into an S3 bucket, and then it ends up in, in some tables, maybe multiple layers, maybe you're using, um, uh, some other kind of schema format.

Now, that same data actually ends up being used. It, it's used by a Python model. Maybe there's a query that pulls, it could actually be exported or queried into a, a dashboard. And then likewise, it could be exported or queried into some export process so that those same data sets to sort of follow the bouncing ball of where the data is in place in a database and data is in use.

And, and, and sometimes the data in use is in place, right? Because it could be cached in a report or certainly could be, uh, put in a, in an export. So the other part too is these, these data sets are actually encapsulated in tools. Maybe the tool is five trend to export. Maybe you're doing your transformations and, and, um, in Airflow or DVT, maybe the data is in Snowflake.

Uh, maybe your models of Jupyter Notebook, maybe your dashboard is, is Power BI. And maybe you've got some kind of export process going on. So you've got some tools, again, applying to that data. And then those tools don't automatically work, right? They've got code that is defining the tools. Maybe it's YAML or sql or a notebook or some workbook, XML or Python code, however you do your work.

And then lastly, all those things have to happen in order. There's, there's workflow, right? There's this has to happen before that there are steps that we, you know, you talk about tools like Airflow or, um, uh, other sort of scheduling tools, even manual steps. And then of course, we live in a world where physical things exist, right?

It has to run in some server or multiple servers. You've got networks. And so now the question comes when your customer says, this is funny. Um, this doesn't look right. Well, where is the problem? And so, um, it's really becomes and across and down problem because it could be any, it could be in a data in place, or data in use, or it could be somewhere down the stack.

Someone put the wrong Python notebook in. Someone did a bad configuration of Tableau. The Tableau server export didn't run the server has, um, not enough CPU, and it's just running slow. Lots and lots of cases where things were wrong. And again, this, uh, has the point. If you waved a magic wand and you made that file FTP source perfect, you could still have all these places that things are gonna go wrong.

Um, and so well, and, and great data is always a start. And of course, garbage in and garbage out. But that goes to the point that there's problems everywhere. And the argument here is to look at it from your customer's perspective, your customer. Um, and in my experience, your customer does not care how you've actually sort of made their analytics.

Um, just like when you go into a restaurant, you don't particularly care how the person cooked your food. You want it to be fresh, you want it to taste good, you don't want to get sick from it. And of course, you, you want it to be on time. And similarly, our, um, people and whether our customers are analysts, whether our customers are executives looking at dashboards, uh, data scientists, they are not happy when things aren't working.

And, um, and so the question becomes then how do you find where the problem is quickly? Um, and where in this sort of complicated mess do you get it? And so let's go through some examples here of like concretely where if we classify problems, so going back to data in place. So let's say I've got some files and tables, I've got files in a schema.

Well, you could have problems with raw data and sort of freshness and volume and row counts. Um, you could also have business problems. That data is being integrated with other data sets. And look, sales are down 50%.

00:10:00

Maybe that's because something happened in Thel that was assembling it, or maybe because your raw data is screwed. Likewise, you could have a a notebook that's doing a prediction and maybe the model is no longer predicting correctly. Maybe your Tableau dashboard is empty. Um, maybe your export has the wrong data. In all these are classes of errors that I, I, I personally have experienced and have seen teams have.

And so these are all sort of basically data errors that can be checked with, with data qa. Now, there's another, um, which is sort of, think of it as processing errors. Did the tool with that code, did it run? Did it start, did it end? Did it have a failure, um, during its process?

Did it start on time? Did it end on time? Was did it take too long? Did one tool happen, need to happen and finish before the other tool started? All these sort of classes of processing errors are also a source of problems and data and analytics. And, and likewise, um, just the basic log files, like, did your server fall over?

Uh, is your server, um, running outta disc space? And then, uh, things associated with cost or, or, or usage. And all these levels of problems are prevalent in, uh, data and analytics production. And so where, where are these problems? And so I guess I've been doing, and my company's been doing data and analytics now for 10 years, I've been probably doing it for 20.

I've managed dozens and dozens if not hundreds of people and talked to thousands of customers. And so, uh, this is sort of my breakout, and I honestly, I don't have a survey, but like I, I do know one thing is that it changes. So at first, the raw data, the data that you get from your customers, that's like at the very beginning of a project.

Um, that's where most of the problems are. Now, you get three, four weeks into the problem and you've profiled your data, you've shaken out all the columns, and you've patched it or talked to your supplier and they fixed it. Um, now all of a sudden you start integrating the data and there's data, and then you start using the data.

And so I think projects sort of shift over time. The errors initially are, uh, when you initially use data, are very high in, in raw data, but over time they shift sort of shift, right? And so if you look at the project as a whole, like a year long project, um, my, you know, only 20% of the errors in my mind has to do with raw data.

Maybe 30% come from putting data together and 50% come from all the other stuff happening on data. Um, and so you may not agree with this. You, you, you may not, but certainly, um, this is, this is my sort of lived experience on where, uh, data errors lie and where you find problems. Um, and so the, the, the challenge with that, right, is, is that we all hope that if we get the data in place, perfect.

Like let's say we get our raw data right, and we get our database right, that everything's great, right? And, and okay, and maybe we do this defensive work and have a data contract and all this other stuff around the data. It's, is it being processed on time? Is it being used on time? Look, that's what the customer sees.

And so I think your customer, um,

I was just checking, someone had raised their hand, so I was just checking to see. Um, so, so your customer kind of doesn't know where these errors lie. So errors shift over time. So kind of where's the state-of-the-art and observability. So I think a lot of data observability vendors, and now there's probably 15 or 20 to, um, VC industry sort of fell in love with it.

There's been like 600 million or more in financing. Um, and, uh, you know, on the hope that they can make the next sort of, uh, data dog or Dynatrace for data and analytics. Um, and, and that's not a bad bet overall because I think there are a lot of data and analytic problems, but they've really focused primarily on raw data.

Um, and, and haven't focused on these other cases, sort of finding problems actively when a dashboard isn't filled. Um, finding problems actively in a model, not predicting, um, but kind of looking down and looking at order of operations. Did one tool happen after another, um, uh, kind of monitoring the SLA of the whole system.

And, and, you know, some are some different, some do more work on sort of basic it log and monitor errors. Um, some don't. Um, but the, the idea here is, is, uh, current, um, data observability really focuses on the upper left, but the whole system is needed to find problems. So that's where, um, where we've come in and, and built this idea of a data journey, which kind of spans across and down your analytic production.

00:15:00

And so here's an example of, of a data journey. And, and this is something that is really a set of observations. And so I'm just gonna, uh, give a quick breakdown on what we do with the data journey and just show you our, show you our software for a second, um, just to talk about it.

So, let's see, escape, uh, talk about our product. So this is, uh, this is a data journey. And so, uh, it, it doesn't run anything. It looks like a dag, like a Airflow dag or a workflow process. The, you, you've got a lot of tools that do that, that run steps. Um, you know, the, the, the first idea is it's, it's a set of expectations about the way the world should be, and the first level of expectations kind of down on that bottom level of the chart.

Like, did it run or did you get an error log? Um, did it run on time? And that's what all these pieces are. We've got, and is Azure a data factory job that has some stuff embedded in it, Python segmentation, um, uh, a notebook that runs, and then an export to a Power BI dashboard also running in, in Databricks.

And so there's this level of like, just checking the base production. Did it run? Are there errors? And, and we do that. 'cause for every step in the process, we're checking to see each step runs. And then for each step, like here's, um, the Azure Data factory, we're, we're sort of watching the substeps, and then we're all the events that happen from it.

And so for us, some of those events are sort of log files, right? Checking to see if there's any errors in the logs that could be a problem. Um, we're also just checking to see if there's any numbers, sort of metrics like files written. Um, and then we're also checking the run status and the schedule.

And here you can see it's running and, and not running. Um, and also you can see the substeps are running and not running. And then, and we sort of allow you to link into the, the actual tool to see those, those substeps. So the idea here is to be able to kind of look at what's happening.

Um, and then of course, um, another part that's very important is the stuff that's happening inside each one. And, and we call that a test. Um, you know, it's a, it's a data quality validation, a runtime check of data, um, and in order to prove that things are right, and so we, we think that actually should happen all over, right?

It should happen, um, not just, uh, at the beginning, an est, but it should happen throughout the process. And that's why you see, uh, data tests. And so one is that you've got each one of these components running. Uh, another level ex expectation is in order of operations, did each one of them run in order one after the other?

And, and that's a problem that, that I've seen. One thing takes too long and the other thing starts and you get this sort of crazy error that no one understands what's going on. And then, and then lastly, you, you sort of are saying, okay, this system, it runs started, it stopped on time, it ran an order.

That's great. But you've gotta look at what's inside. And that's what these sort of tests are. There are runtime checks to actually see that things are right. Um, and all these things are, are sort of captured in this idea of a data journey we store it for you to look at. And then we actually, uh, allow you to sort of build a whole bunch of rules on top of it to understand what's going on.

Like alert rules on failed status sent to, uh, particular people, um, a late start, um, percentage order of operations, a way to kind of manage by expectation. So the idea of a data journey, kind of going back to our, our, our discussion here is that, um, you know, errors happen across and down, and you need an abstraction to be able to put that on top to see where it is.

Um, and, and sort of why do you need that ABS abstraction? And I, and I think it, this goes to the core benefit of, of why do observability at all. And I, I think of, if I boil down the problems of data and analytic teams, um, they really come to failure because teams are wasteful, um, because they're not productive.

And one source of productivity, if we take that from a metaphor, we have the dueling 23-year-old problems, the first problem is like, can you get a 23-year-old to make a change in a complicated system and put it into production without having, uh, on their first week of work? And a lot of software teams benchmark themselves that way.

Hire a 23-year-old, very smart, can they do a bug fix and can without a lot of work and a a lot of supervision, they can, they can do a small bug check and get it into production. Um, that's one thing. And I, I think most data and analytic teams are very poor on that, right?

They can, um, you know, they sort of hope things work. They, this, the person can make a change to a column name, um, but they have no idea if it's gonna break all the reports, uh, and all the models that are fitting in. And, and most, uh, salient to this, the second prob 23-year-old problem is, well, where something goes wrong, where is it?

Um, you know, where is it in our stack? You know, is it across, is it down? Was it the server? Was it the code? Um, and instead we end up with these large sort of, um,

00:20:00

you know, these sort of red flag meetings or, uh, you know, uh, war rooms where the smartest people in your team are hitting log files. They're querying, they're looking, they're like trying to find out where the problem is. And that, that's just like a complete waste of time. Um, certainly because it's your most experienced people who could be creating things.

And, and second, let's, let's have a 23-year-old find where the problem is. And and maybe they can fix it first, um, and then keep the thing running and hopefully even find it on time before your customer, before your customer sees it. So that, that's really the first, um, bucket here. And, and, and it's not just, you know, you can have perfect data, but you still can have a lot go wrong.

And a lot of it happens, um, across the stack, sort of a, a breadth and depth. And so you need to actively check, you can't just trust lineage to give you the blueprint. You actually have to sort of put smoke alarms on every one of those pieces, um, and, and find out if there's problems.

And, and so that needs an abstraction, uh, for you to organize it. And, and so you need a data journey that's, that's pretty clear to us and why we built the product. Now, the second is sort of beyond data ingestion. Now, I, I think there's a lot of ways that people have a lot of patterns that people use for data teams and, and, and tool chains.

And, and so the first pattern is basically little boxes everywhere. I mean, what does that mean? Like if you look at, um, kind of these pictures and say, well, you know, what's our modern data stack? What's the stack on Google Cloud? What's the stack on Azure? What's the stack on AWS, what was our traditional stack?

It's little boxes of technology, right? It's a database, it's a BI tool and ETL tools and orchestrator. And, and so, um, and then there, there, there's significant overlap into what functionally each box does. You have BI tools that do ETL and do data science. You have data prep tools that are ETL tools that also do data science.

You have databases now, like, um, Databricks and, and that do workflow and, and it's just, it's very confusing because these little boxes overlap and people have their favorites, but I have never seen, and I keep, it's multiplying. There's just little boxes are happening everywhere. And, and, and that's I guess normal 'cause we're trying to maximize it.

But of course, you, you hit an X on one of those little boxes and you have a couple, uh, you have a problem. Um, and so that's one part. That's one part. Um, and the second part is, um, we have how our teams are organized. Um, it changes. And so we have the sort of centralization or data enablement pattern where, um, they, there's a team that produces data.

Maybe it's just raw data, maybe it's integrated data, maybe it's master data. And then you have consumers of that data who take it and do a lot. And so you have this sort of multiple producer consumer relationships and, and this fan out that happens. I produce some data and somebody else uses it. Um, and so sometimes call that data enablement.

Sometimes people call that hub and spoke, um, where you have a hub team, a data enablement team, and they have lots of spoke teams using, using it. Um, and this is, um, a, a great way for data and analytic teams to scale because they're been, the smoke teams are closer to the business, and the hub teams are, are more technical.

And so you have data lakes, data lake houses, um, and, and this pattern, uh, I I've seen in a lot of companies, um, and, uh, you know, smaller teams, they, they don't have this, 'cause the, the hub and the spoke are combined into one organization. Um, now that actually leads to great productivity, right?

Um, having your hub and spoke teams combined, but they can't do everything right 'cause they're a bottleneck. And so two or three years ago, people, uh, applied the idea of domain des design or sort of microservices from software and said, well, you know, hub and spoke, you know, we have single points of failure.

Let's, let's have the, let's have little groups of sort of small teams do everything. They're the hub and the spoke. They, they integrate the data, they access the data, they share it with other people. They, they service their customers. So, so they are, um, decentralized and, and that has the sort of topology of many to many relationships.

And that, uh, data mesh design pattern, I've seen a, uh, some organizations work well with, and some organizations struggle just like in software teams implementing domain design between some of the, uh, central and monoliths. They're, um, that they struggle. And, and, and these are, um, good engineering discussions that have to do with the size of your data and your type of customer.

But you do need to, you do end up with this organizational patterns of producer, consumers, lots of tools, producers, consumers, hub and spoke, uh, data mesh. And then lastly, the sort of, i, I don't know if it's a pattern or an anti-pattern, I just don't know. But a lot of data teams basically inherit an out of control database.

And that is, they are in, they, they're the ones who get blamed when the data's wrong, yet they don't control the data going in. Some other group is pushing data into their database or some other group is doing changes to the data.

00:25:00

And so typically, you know, I have, I've talked to some people where I'm getting blamed, but there's some people who've got a bunch of containers who are shoving things in, and I don't know what there is. And there's a bunch of analytic engineers using DBT who are making changes and I don't know what that, what's there and the whole thing's sort of out of control and, and they're looking to get that back.

And, and I think that's, that's a reasonable any pattern, right? 'cause um, or reasonable, uh, uh, thing to want. Um, they're left to blame for bad data, yet they don't control the, the sort of inputs or, or outputs or only control some of them. So how do you handle all these cases, all these design patterns, right?

Of producer, consumer hubs, spoke, uh, data mesh, um, uncontrolled data stores. Well, I think you need another pattern to handle all these. 'cause a lot of organizations will end up with multiple versions of the same. They'll have not, one company won't have more than one. Um, and I think honestly, the, the idea of a data journey, um, bridges all those.

And so one is a sort of, if you've got this sort of uncontrolled database, well, you can have a journey that just monitors your data or just monitors your data and your injust processes. Um, the second is like if you have a team that kind of works for one boss, right? Where all the tools are under one, one boss, you can have a end-to-end data journey that encompasses all the tools and all the data sets.

Likewise, if you've got a hub and spoke or a data mesh, you can have these one to many or many to many relationships between data journeys. Um, and that's, um, and, and the these patterns I think are, are really good. And, and the way I look at it is, again, the, the current observability is focused on this one pattern, which is sort of data teams with dangerously outta control databases, which is, you know, true, um, uh, it's not everybody, but it's true.

But they also have these cases of sort of, uh, end-to-end product teams, uh, hub and spoke teams. And so our, our focus has, has always been to, to make sure that we can cover, um, all these cases from big companies or small, and also that we can make it easy for people to start with one pattern, um, and then, uh, go to the other.

Um, and so how does that like, look in our, in our software? Well, I'm just gonna kind of give you a, a, a quick walkthrough. So, um, you know, if we go back to this project and I look at some, some journeys today. And so it's a, a little bit hard to see. Let me just make this, um, a bit bigger.

So I've got a couple things here. I've got my, um, daily data load that happens every day about 6:00 AM and it's scheduled for tomorrow, uh, to run. And, and what does it do? Well, if I go into it, it in this sort of simple demo world sort of loads up four tables, right?

And, and has Airflow loading it. And here we've got a whole bunch of, um, operations. And some of them are yellow, like there's 742 tests against this table, 2 79. And, and the, these tests and making sure that these data tests are, are important 'cause they actually prove that the data's right. And the, and so this is not only sort of data health, um, or data quality, but also things like schema and, and, and volume, et cetera.

And, and so those tests actually are collected into our tool here. And you can see all the tests and you can see the ones that passed and failed. Um, and they're actually generated either from tests that you write yourselves or we have an engine that can automatically create tests for you. We have, um, uh, an engine called TestGen and it actually builds tests for you.

And, and, um, I don't know how, you know, if we look at sort of what, what our TestGen tool does, it really kind of does a couple things. It it, it'll start off with looking at all the tables in your database or the tables that you can configure and profile them. And then it just kind of a hygiene review.

It says for this table, uh, here's some things that look weird, right? You've got, um, some addresses that aren't aligned. You've got some names of things that aren't right, you've got some blank fields, you've got some zip code problems and, and this looks weird. Um, and then what it does is take that information and then actually algorithmically generate a whole bunch of data quality validation tests, about 30 of them that that run.

Um, and those can be, um, uh, are used. And then it actually does production data quality testing. So, um, which could mean running kind of during your before or during your, after your production process sort of, uh, along with your orchestrator. Um, and lastly, it could kind of continually monitor also just sort of pull your database and, and it's producing these sort of production data errors or, or data monitoring anomalies.

And, and so, um, you know, it does this by having kind of 51 characteristics of data profiling has 13 interesting data hygiene de detectors has 30, um, now auto-generated data tests and has 11 tests that are sort of more business rule related. Um, and all of these can be sort of configured by your data engineers or, or your data experts.

And that's our, our, our TestGen tool. And so that, that goes back to if you're gonna, uh, observe your system and all these patterns that people work well, you want to prove that it's right, like you want your daily data load to be right. You, you wanna know that that data's good. And the way to do that is to check it with, with tests.

And, um, we're happy if you have tests already

00:30:00

and we also have a tool to write those tests for you. Um, and, and then second is, well, um, I've got the one that I just showed you, this sort of database and, and model production that uses this data, daily data load. Here's the hub and here's the spoke. Um, and then the, the third case is I do have a dastardly dark database, um, running and, and here's a case where I'm running are, um, these tests and I'm kind of running them over and over and over again.

And so here I can see one where I've got a, a number of roses down a threshold. This is sort of a volume break, uh, that goes into it. And, and I can kind of see that, um, in, uh, also in my TestGen tool by kind of going in and, and looking at, um, uh, the test result and kind of seeing that failure and warning and seeing the test result that goes by and seeing that this has been a problem for, for a while.

And again, this all runs on prem, so we don't have, have, uh, uh, mess with your, uh, with your data security. And then of course we've got all sorts of other tasks that we're running against your data in the database, not copying your data, bringing it together. Um, so, so the whole point here of, of this, uh, of what I'm talking about is that these patterns that we have and how we run our teams, we have lots of tools.

We have, um, producer, consumer hub, and spoke data mesh, um, single teams, and then we've got sort of uncontrolled data stores. All of these are cases where data errors happen. And so you have to look just beyond sort of the top part, just beyond kind of, um, uh, your dastardly dark database to all these other use cases.

Um, and, and there's one more use case, um, that I think is actually really interesting and, and that's, um, having to do with, it's looking beyond data. And this use case is really, um, lastly about trying to find specific data for your customer. Now, in general, data and data and analytic work is an aggregate activity, right?

You're pulling multiple data sets together and building some other kind of insight for a customer. Now, what happens if you've got two or three customers or two or three use cases embedded in that data, um, and you want to find out if they're good, sort of, uh, but you're doing this aggregate work. And so, um, you know, we've got this challenge is sort of the idea of a personalized data journey.

And sort of like, I, I am sure you've heard this sort of, where's my data? But your data's being processed with all the other data. I don't know where your data is. Um, and there's some cases where this is so important. Um, rather than kind of giving, your data's gonna be ready when everyone else's data's ready.

Um, they need to, to actually find out what's happening with their data, um, and sort of insights, uh, into exactly why they need that. Now you could say, well, wow, my customers, you're just gonna have to wait. Well, there, there are customers and, and needs. So for instance, we're, uh, we're working with a, um, a drug discovery company, and they have this sort of eight, they, they have a process where their, uh, data scientist is building up a whole bunch of data about a molecule that they're looking on, and it's a bunch of text and other data sources, and it goes through several databases all to actually help accelerate drug discovery.

Um, and, you know, they, they're sort of synthesizing this data for them and, and it hops and takes days and sometimes it gets stuck on where it goes. Um, and the, the, the scientists or the chemist is like, sort of, where's my data? Where's my, where's the data for my molecule? And again, then you have this sort of multi people, smart people log sort of groping and finding problems instead of being able to find out exactly where it's, um, and, and that's really, um, you know, you're assembling data thinking.

It's like, where's, think of it? Like, I'm, I'm, um, it's not acceptable to say, um, like, let's say you bought a new car, like, when is your car gonna be ready? You don't really care that, like your factory's producing 500 cars new days. Like, when is my car gonna be ready? Is it halfway done?

I wanna know. Um, and, uh, uh, similarly, like you don't care if UPS is, you know, delivering a million packages a day in your neighborhood. You wanna know when your package arrives. And in that similar set, that similar spirit of what customers want. And then on the supply side, it's, it's similar too. Like there are, there are people who have very high importance data sets and they want to know if that data set is moving through.

And instead of looking at it from the customer, it's actually looking at it from the dataset side to be able to find out what's happening with that payload. And so we've, uh, invented something called the payload journey that we just released this, um, uh, this last few weeks. Um, and it's really being able to sort of get the status of their data or the data as it moves across the data journey, sort of where it is and where the problems are with a personalized level of monitoring and alerts.

So it's not like, Hey, it's sort of the where's my package? As opposed to what's the status of, of, uh, FedEx? Um, and how does that look in our, our software?

00:35:00

I actually think it's, it's pretty cool, um, the way it looks. And so if we go back to our list of data journeys here, um, uh, instances, you can actually see something, um, interesting and, and I'm just gonna pick one here just to clear this UI up. So I've got this dashboard in model production.

It's a, it has a start type batch that means all the data that goes through it. Now I've got two of these, what we call payload or personalized journeys. One for the sales business customer and one for the marketing business customer. That's their data. Um, and not related to each other. And actually we can see a problem that the, the, the sales won't work, but something happened with the marketing business.

The, the journey sequence was not met. This Python model didn't run for some reason, just for the marketing customer. And that, that's great to know because here you've taken this complicated world, you've peeled it apart by who needs the data. Now, this, this could be because it's a dashboard that's specific to the, the, um, the marketing customer.

So maybe you're getting HubSpot data, um, or maybe you're getting some other data set who, who knows what the problem is. But it gives you a start to be able to find out, well, why didn't this segmentation run? Um, to be able to go in and actually look at what's happening here and, and looking at the log files and digging into it, and this idea of a personalized journey, seeing a journey not just for the entire system, but for a per customer view, and then having them kind of dig in, whether a marketing or sales customer.

Um, and, and why is this important? Again, because of we're trying to get your team to not have waste and not spend hours digging into data when it happens by easily instrumenting your system to, to make sure that it can produce this kind of insight and this kind of observability. So, um, uh, but been talking here for a bit.

I, I can see that there's, um, some questions, but just to, just to conclude. So just watching your database periodically pulling your database and checking isn't enough, right? You need to actually look beyond anomalies. You need to look at the breadth and depth actively, you need to put smoke alarms everywhere. You just don't need, uh, a blueprint.

Likewise how your teams organize their data systems. All those little boxes structurally need to be covered. And there you need sort of monitoring. You need hub and spoke. You need data. Uh, you need end-to-end. You need all those design patterns and a way to look through them to get at the air quickly. And then lastly, you have to look at the data from your customer's eyes, and maybe they want to find out what ex exactly happening with your, your customers.

And so all these cases are, um, valid for observability. And I think, um, as I talk to analysts, I think we're, they're learning, uh, I think what and, uh, the industry sort of learning more and more what true observability means for data and analytic systems. Um, and so if you wanna find out more, um, and we've got, uh, as I've shown, you've got two tools to do this, that do all these, uh, use cases, all three of them, our DataOps observability and TestGen tool, um, find out more.

Um, and I'd be happy to talk to you about it. So thanks again for taking the time to talk and let me see, uh, go back to our chat window and see if there's any questions.

So again, there's that, there's that question about will the slides and the recording be, um, uh, be sent? And yes, of course, I'll put it up. We'll put 'em up on a website. Um, we'll send you an email with, with the link. Um, and, uh, all right, I think that's it. I don't see any more comments or, or questions.

Um, just one last, uh, one last time. Um, I know we've got a special guest, uh, here. We've got a, uh, uh, an analyst from a regarded firm listening in. Um, we've got some other people, um, who, who, uh, who are new to DataKitchen. Uh, we had a couple of companies have half a dozen people register, so hopefully this met your needs.

Um, feel free to, to, um, uh, give us feedback and look for an email from DataKitchen. And again, thank you for your time. Have a great rest of your day.