Introducing Open Source Data Observability

00:00:00

Hello, everyone. My name is Chris Bergh, and I run a company called DataKitchen. And, um, what we're gonna talk today is about two pieces of open source software that we have just released in the last week or two. And, um, so we're gonna go through sort of background of why it exists, uh, a short discussion of why we've open sourced it, and I'll talk through four use cases, um, in data and analytics where it's useful, um, from a logistics standpoint.

I'll be the speaker. Um, the slides and the recording will be a mail will be, uh, made available at the end. Um, feel free to type your questions in the question box. I'll answer at the end. So without, uh, any more introduction, I'll start. And so, you know, our, our company's about 10 years old.

Um, for the last 10 years we've been talking about an idea called DataOps. And, and really the motivation to do DataOps is waste in that lots of data and analytic projects fail. Lots of projects have too many production errors, lots of, uh, people who receive data and insight, they don't trust the data, and lots of data teams are unhappy and, and want therapy.

And so for us, we think the reason for that is just this little box problem. There's jot, there's lots of little boxes, and like if you go to, uh, Azure or Amazon or data architecture, you see these diagrams on the left. It's just lots of boxes everywhere. And then also our teams are siloed.

Sometimes you have hub and spoke, sometimes you have centralized teams, some they're siloed. And then of course, we just have lots of data and lots of diverse data. And so the, the result is that most data and analytic teams suffer from getting bad raw data, a complicated tool chain, having too much to do and having customer visible errors.

And so the, I think the root cause for that really is a focus on day one. And that means focus on immediate tasks. What do I have to do? What do I have to build? Um, how do I get it done? Sort of how do I get through the day? Given all that stress, um, all that lack of success, it's not unreasonable.

And the concept of DataOps is really about, not day one, but day two or day three, metaphorically. It's about optimizing the systems of people, their tools, their data, and their work process and how they deliver value. And so for us, we think the value in, uh, taking DataOps has three parts. One is focused on decreasing errors during production.

Um, and then the second is really increasing the cycle time at which you can make changes to production. And if you do both those, um, you get a huge improvement in your team productivity, um, you have less waste and you have more trust. And so, um, where to start this idea? It sounds good.

I think where we're gonna focus in this discussion is on day two, kind of decreasing production errors, decreasing customer visible errors in anything that you, uh, create for them. Data models, dashboards, extracts. And so we, we recommend that you start with open source data observability. And so I'm gonna talk through that. And so, uh, kind of similar to this in that you build a data and analytic system and maybe you build the cloud or on-prem, you've assembled your tools, you have your diagrams, but stuff goes wrong.

Um, and that the problems in that are manifold. They could be in the raw data, they could be in the integrated data, they could be in the tools that are acting upon data. Um, they could be in the duration that things are taking,

00:05:00

they could be in the server. There's just lots of places where things could go wrong. And so for us, over the past few years, we've been building these two products on the left, DataOps, observability and DataOps TestGen. And we've used them with a bunch of customers we've been working on them for, for years.

And so I'm gonna talk a lot about what those products do today. Um, but I'm also gonna talk about why we chose the past to actually make them, um, uh, Apache two oh full open source. Um, and we also have, uh, and so why is that? And I think first of all, from a

a where our company is at, we've always had our North star in getting teams to adopt DataOps. And so, uh, as being a profitable company, we've always tried to sell things like software and services. Um, but what we've noticed is that, uh, selling to leaders, uh, selling to people with gray hair, um, uh, is a problem because there's a big churn in leadership and data and analytic teams.

The average tenure of A CDO is like two years, maybe even less. And so how do you make it, uh, a change, uh, getting people to work in a more automated, a more tested, a more observed a more DataOps method. And so we think that there's, uh, we should focus on the bottoms up approach, and that's sort of why we've done the open source, um, version.

And also we, we built these tools, um, and we've used them with success at lots of customers. And conceptually, there's kind of three things that we think are really sort of intellectually interesting on them, and I'll talk about this. The idea of a data journey, um, which is separate from data lineage, uh, we think must exist.

And, and second is that data engineers are sort of too busy to understand their data and actually test it or find anomalies. So that has to be all automated. And then you have to integrate. And so our tools are full featured. You can go use them today. They have front ends, they have backends, they have use UIs, but we focus on making them full featured for an individual user.

And since we are still capitalists, we sell an enterprise version. And the whole goal of open source is for us to accelerate the adoption of DataOps in the industry. And so what's the start in on these things? So I think there's a, a, a number of data observability companies out there. Um, I think the message here is one does not simply check for anomalies.

There's a number of different use cases for observability that you need to cover in order to, to make sure that things are right. And so if we go from the top to the bottom, the first case is, let's say you get a new data file, um, that you have to integrate Well, is it correct?

Can you read it? Um, does it have any problems? Should I patch it? Should I push back to the provider? That's one certain case. Um, the, the second case is maybe you're bringing in things, maybe you're loading CRM data every 10 minutes. Maybe you're doing change data capture. Maybe you're doing your weekly data load, maybe some new data magically appears.

And all those cases are your updates to really existing data sources. And that happens kind of all the time. Um, and then the third case is really you're, you're building your production deliverables, sort of monitoring of multi-tool, multi-data set, multi hop analytic processes, and that's like dashboard production, warehouse building. And then the last is during your development process, how do you actually do things like regression and impact, or more specifically, how do you add new code or new tools or new new configuration, uh, into your system?

And do that quickly and do that with, uh, a low, uh, low problems. So, uh, I'm gonna talk through this first case data evaluation and uh, I'm gonna kind of use these pictures here to talk about it. And so these pictures are kind of all the same. They have a functional view, a data view, and a scheduler view.

So in this case, and, and these are just examples, they don't have to be this way, right? You could pick your database and pick your, uh, file, you know, pick your raw data store, et cetera, et cetera. But you're bringing in a new file, maybe it gets into an S3 bucket and you load it into a, a Redshift table.

Um, and so what could go wrong with that? Well, I've got a new data source. Maybe there's double quotes on every column or double quotes on some columns. Maybe the concept of null is in three different ways. Maybe it's blank, maybe it says null. Uh, maybe it has a dash in, maybe some things have leading spaces in.

Um, and how do you actually look at this data file? And number one, understand it. And then number two, find out if you should patch it or push back on it. And so typically, like what I've done is you sort of load it into Excel or write some queries against it. And so, um, that's time consuming and problematic.

So we built some a tool to be able to go in and actually scan the data and profile it, um, with 51 profiling characteristics,

00:10:00

and then basically have some suggestions on how to, how to, how to find out where the problems are. Like, and these are meant that term suggestions is important. And so, um, what I'm gonna do is actually kinda show you how that works here. So I'm gonna share, uh, this is, uh, our open source version of, uh, of DataOps TestGen, which is part of our data observability products.

And so in it, this is a tool that, um, uh, connects directly to your database, executes sql, and the first thing that it does is, is profiles your data. So it scans the tables with 51 characteristics and, and profiling's, you know, interesting. But like, um, you know, it tells you, uh, the different states that it has, it, it can actually break out the income level.

And there's a whole bunch of characteristics here that you can look at, and we can kind of group it by table name. And it actually has some things like the suggested data types that it can go through to see if it's, uh, actually right. And so this is interesting for you to understand your data.

Um, and this is not unique. Like there's 57 tools that, that do data profiling. Um, I think what is unique is that once you profile the data, we find, um, these sort of highest on it. And here's some examples on it. So, uh, if you look at this, for instance, this is a four tables, it's sort of a fake bike, um, e-bike set.

And so, uh, what that shows is, um, you know, for instance, the frame size. And so if we dug into it and we found some non-standard blank values, and we can kind of drill into the source data and say, well, there's some things that say missing, and some say that na and some say that are just blank.

And so that's an another case. And so here's a case where the column's empty, and you know, why is that? Well, I don't know, it's just empty. Maybe that's fine, maybe it's not. And that, that's why these are suggestions. Um, and

here's a case where there's quoted values. So we go in and find some things that are quoted, and these are like really kind of annoying, right? When someone goes to do a report and they put in a string and then they forget the quoted string. And so this is kind of data cleansing, standardizing your data, being able to look at it.

And, and then we also can disposition these. We can go in and click on one and say, I wanna con conform it. I want to, um, uh, deactivate it for future runs. Uh, I want to, I, I wanna redo it. And because I think sharing this sometimes with people who know more about the data as a data engineer, maybe you, uh, a business analyst, maybe someone in data governance.

So all those things we think, and kind of going back to the slides here, um, are about the first use case data evaluation. Now, the, the second use case is stuff's happening to my data. I'm getting new data. And that could be from ingestion, right? I'm, I'm, maybe I'm bringing some new things in, uh, every 10 minutes, maybe somebody's mucking around with database tables, I don't know.

But let's look at this from a, another picture that talks about the functional view, the data view, and the scheduler view. So for instance, here, I've got an Airflow job that puts some that, that updates database tables. And so maybe it gets it from, you know, pick, pick, pick your place that it loads it in, and it goes similar to goes, goes in S3 file, goes into some raw tables and maybe into some, uh, you know, L two, uh, tables.

And that runs every kind of 10 minutes. And so what, what's some problems? Well, maybe, uh, there was problems with S3 for some reason it didn't load. Or maybe the data load didn't refresh. Maybe you're expecting it and you didn't get new rows. Um, or maybe there, there's some sense of the schema has changed or that the data has drifted in some ways, or that the recurring job failed three times in a row.

And all those things, I think are something that you want to know about, because you may have users using these tables directly, you may be preparing those for doing your weekly production. And so the way we handle that is, is two cases. We have a set of tests that actually do the anomaly detection, and I'm gonna talk, uh, on this about how we generate tests and, and how these are automatically created.

And then we actually go in and, and, and in our observability product, remember I'm talking about two open source products, DataOps TestGen, and DataOps observability. It actually goes in and, and, and allows you to collect all those. So let me go into observability and, and, and share this tab. So I'm gonna talk a little bit about the concept of a data journey, which is a sort of a grouping of all the pieces and data sets that you have.

I have one data journey here called a data bot, uh, uh, database watcher. And when a journey's running, it creates an instance. And an instance is a, uh, kind of the as run view of it.

00:15:00

And this one has got a bunch of tests that run. You can see there's 901 tests, uh, a bunch passed, a bunch failed, et cetera. And so looking at these, I can actually kind of filter them by what I see past versions. And so it's important to have these things are sort of past or warning or failed.

And so I can go look at my warnings or look at my failures and see what happens. So I can, for instance, see that there's, um,

you know, that there's column values not expected in the list. And I can actually go in and kind of see the change. Like, it was fine. But now we're getting data with some new data in a column. And this actually goes into our, and you can see the case here, the baseline view is blue, silver, red.

So we've got some new, you know, a new color showed up basically in my bike list. And yeah, that's, maybe that's fine, maybe it's not. Um, but it's something that you can control and understand, and then also build rules on how to get alerts on it. And so, um, we have, uh, as part of our system, a rules engine, and we can kind of create rules, and I'll, I'll, I'll talk about this in a little bit, but basically you can look at it on data errors, run errors, metric errors, et cetera.

Um, and so how do those actually get created? Well, in those, we actually go into our TestGen, and I'll share this again. And, and to do that, we actually first start off with profiling, but profiling is run infrequently maybe every couple of days, every month. And from that, we actually generate a set of data quality tests.

And so here you can see the runs of these tests. And so I can kind of drill into the last run, and I can see the test results against it. And so I see a list of failures and warnings here. And so this engine can go and I can kind of find and filter on which ones have failed and which, uh, which ones are different.

And so I can kind of look at each one of these and, and kind of understand where the problems are. And I can go into this color case, and if I scroll up a bit, I can kind of see the same problem, but I also can change it. And here's the test configuration. So I can set the threshold, I can tell whether it's gonna be performed, I can lock it, so it's not changed.

I can actually go in and, and sort of review the source data from here, um, and look in it. Um, and then I can actually go in and edit the test if I want. And so there's a bunch of characteristics here on editing the test. So for instance, if I wanna add that new color in, I can add it in here and, and say, okay, there's a new color.

And so this is, again, making these data, these data anomaly detection tests, having them generate automatically, having them be customer visible, having them be alerted is a, a, a great step towards finding problems when, um, you have,

you know, when during data ingestion. Um, now let's talk about the third case. Now, this is production. So let's say you're building a warehouse, so you're integrating a bunch of data files. Maybe you're producing a dashboard or an extract, or a model, or all those things, right? And so you've got this assembly line.

And so assembly line of multi tools, multi-data sets, multi hop, um, and, you know, you wanna find out, did it run, did it run a time? Is it producing the right stuff? Did my tools work? Is my data wrong? And I wanna find that pretty quickly, and I wanna know exactly where it is in these complicated systems.

And so if you look at that, these things are, you know, and, and as an example here, maybe you've got Airflow and DVT and Databricks and Power BI, um, you know, pick your, pick your tool here. Um, and then the data view, you can sort of follow the bouncing ball, right? It could go from FTP to S3, to raw tables, to process tables, to maybe some ephemeral tables and Databricks, and finally back to an extract.

And so where is the error in the data? And then finally, you may or may not have multiple schedulers, right? You may have an Airflow job that runs, uh, on Tuesday at 4:00 PM and then you have a Databricks job that runs at Tuesday at, at 8:00 PM and then you have the extract on at Tuesday at 11:00 PM Again, you don't have to do this, but that's not uncommon.

And so you end up with this problem, right? I have my daily production job run, and maybe one of my DBT tests failed. Maybe I had a server error, maybe I had a model error, maybe I'm missing some data. Or maybe there was a column pattern mismatch, or maybe the Airflow schedule never ran, or maybe it's just late.

And so all these things you think you need to become aware of. So it's the, the challenge here is like, how do you understand this big complicated system? And so to do that, we're gonna talk here a lot about the idea of a data journey. And it's also to do that, you need to not only check the tools, uh, but you need to check the data as well.

And so, kind of going into this idea of a data journey, so I'm gonna share this tab, actually, I'm gonna share, um, I will share this tab. I'm gonna show something called the dashboard and model production. And so what does this view mean? So in this system, this is, uh, sort of running on Azure, not on on Amazon.

So there's an Azure Data Factory job run.

00:20:00

After that runs a Python segmentation model must run, and some, uh, uh, a notebook runs after those two run, then the Power BI dashboard is extra and loaded. And so all these things have to happen in order. So one level of checking things is thinking of it from servers process, is it running? And so did these guys run, did they run on time?

Did they run in order? Um, and are they late? And that's a certain level o of checking things. And so another level is checking the data itself, did the raw data, the integrated data, the data that's put into the dashboard, uh, work. And so to do that, we have, we collect the ordering information on this.

And so we're actually going in and observing what's happening in, um, A DF. So we build what's called an observe graph. So we observe all the steps and, and intuit, uh, what's in, in the graph, and then we capture events for it. So here's just the events for Observe Data Factory, and I'm gonna select all the events here.

And so what we're getting, and we're getting a bunch of different events. So one of the event types is something called Run Status. And this is why it's hard to actually observe data and analytics systems with your favorite, um, it observability tool because it has to run, and then each substep runs and it has to run on time and complete on time.

And then if there's a problem, which we're not having here, you want to be able to sort of, and I'm gonna click on this box and it can take me to directly into the tool, uh, if I've got the appropriate credentials. Um, and so, um, there are other things that this system collects. One is it collects messages.

So maybe Azure Data Factory is sending you some log messages, or maybe it's sending you some metrics. It can also collect that. And then also it collects, um, what we call test outcomes. And that's what we showed before these tests that were brought in. And maybe those tests are done by our tool TestGen, or maybe they come directly from DBT and our DBT integration.

We don't particularly care how you're checking and validating your data. We just care that you get it done. And so we bring all these things back together, um, kind of into, um, this idea of a data journey. And so this data journey doesn't have one part, it has multiple parts, and we're collecting events across all those tools, not just as our data factory.

And so we can kind of see the events, what's happening, here's the run status event, here's for instance, uh, what the test outcomes are across each, or here's various messages across all these tools. And likewise, we can kind of drill into each one of these tools to see what the problem is. Like I could sort of drill into, um, like what's happening in Databricks.

And so as these run, we collect all this information together and, um,

Oops,

We collect all this, all this information together into, uh, observability. And so there's not, usually organizations don't have just one journey. They have journeys and instances of journeys. And so in this case, we've got a number of running. We've got, for instance, the daily data load happens one time at a day. We've got this one that runs again, uh, maybe it runs every day or every, every three hours, the dashboard and model production.

We've got a nightly export that's happening, and all these things are collected, and you can kind of start to see their status over time and see what's going on. And in this demo system, uh, that if you take our, we load this demo data as part of the, um, building of the observability. And so that, that's the, the, the third case that we talk about, um, that we support.

And really it's, we've gone through sort of, I got some new data, I gotta check it. I've got some data changing all the time. I don't know where it is. I wanna find out problems right away when it's ingested. And then I wanna, uh, monitor my whole data and analytic system for errors no matter where they happen.

And so the last case is really development. And this is not a production case. This is, I, I wanna make a change. And so, um, in, in, in some ways DataOps is about three problems. Two 23-year-old problems and 1 46 year problem. Now, the two 23-year-old problems are, can you hire a 23-year-old and have he or she really smart, uh, with a CS degree?

And can their first week, can they find a problem in your data and analytics system exactly where it is in production? And I think our observability and, and, uh, tools handle that. The second case is, can you take another 23-year-old and have 'em make a, have 'em make a change, make a small change to, to something in development, one line of sql, change one table,

00:25:00

and then have it auto, have them be able to tell if something, if they affected something. And so we build development systems, sometimes more than one, but we end up with this problem. I've had that 23-year-old, they changed some new SQL code, did it work? Did the SQL code work, but somehow the report's empty, you know, like they changed the schema, um, or did it produce empty columns?

Or is the extract row row count? Or did it not work in CI and cd? And all these things are to get the goal of can I have, uh, a person who doesn't have the whole view of the system in their head and make a change and have the system have some automation next to them to be able to say, look, it worked or it didn't work, and not have to get manual checks, not have to have someone, um, go in and, and, and dig in and find the problem.

And, and to do that, you, this idea of a data journey is also really important because it's needed to do regression testing. The data, uh, journey becomes, uh, sort of a harness that you can, IM, that you can integrate into, um, your CI and CD system. And so, um, uh, that's, so what we did is we went through kind of these sort of, um, going back these four use cases on, uh, bringing a new data set in watch as it's ingested or changed to find out errors right away, monitoring, production, and then development.

And so, um, the last thing I wanna bring up is really kind of how it all works. And so oftentimes people are like, well, what is this thing? So first of all, there's a product called DataOps Observability, and that has really an API and that API brings in things like messages or metrics or run status or schedules or test results.

And you can actually see the API. So you, um, if I go in and I share this tab, we have, um, what's called the event ingestion, API. And so this is the API, it's Swagger doc, it's open, uh, it's documented, um, in fact, even the, uh, API internal to the product, the, uh, actual API that's used to run the, the UI is, is available there.

So that's sort of the way, uh, this works, is that there's agents which are small pieces of software, which connects to the tool. So there's a Redshift, there's a, uh, uh, Databricks agent or a Power BI agent, and that sort of marshals log messages and metrics messages and run status and sort of sends it to the API automatically, excuse me.

Um, and so another source is DataOps TestGen, which in itself is really checking the data in a database. DataOps TestGen is a tool that talks directly to the database, talks in sql, um, supports, uh, right now four databases, Redshift, Postgres, uh, Snowflake and Synapse as, uh, as SQL Dialects. And, um, it sends things into the API so functionally.

Um, and, and so the way that works is just let me, um, kind of share observability again, is that there's integrations in the ui. And so here I've got a list of integrations in there. These are all the ones that this is, uh, sort of faked data that you, you can, uh, do, but it, it's running against it, and it's sort of pinging against each one of these saying it's alive.

And so we have a bunch of available integration agents, and really these agents are kind of Docker connectors or, or Docker. And so you can go in and sort of pull them. And we have, from the open source perspective, all these, almost all these are available in open source. And then there's a framework that we use to write the agents and, and do the heartbeats.

And you can extend that and add your own system. Um, and the way it works is there's a sort of an API key that you go in and, and, uh, uh, if you wanna add a new API key, you can do it through, uh, this ui. So kind of going back to this, uh, this ui, um, so this is architecturally how it works kind of functionally.

The way it works is that there's an API and you get data tests, you get tools with observability agents, or maybe you're just writing it against the raw API in our product. It makes something called a component. And those components are sort of bound in, uh, in an ordered or un our journey. And that goes in the UI and dashboards and alerts.

And it's kind of going back to, um,

going back to the product. You can kind of see this in that there's a bunch of components in the ui. And so, for instance, I can go and look at my, uh, my, uh, Azure data factory components. So kind of going here, I've got one agent that produces two components, and the components are kind of equal to the jobs that are running in each one of these tools.

And so in there, I can kind of go in and see the events and sort of drill into each one of those.

00:30:00

I could also manually add a new component, but they're auto created. The agents actually sort of auto-create and build new components for us. And then each one of these components themselves is really collecting events. So I can kind of see the events from that as a data factory one or, or another one. And so each one of these is, uh, sort of an event collector, and that's what the purpose of this diagram is.

Uh, and so some people, um, uh, are interested sort of architecturally and functionally how it works. Um, and so, you know, it has a rest, API, we have, uh, tool data tool specific agents. Um, there's multiple component types. So let me, um, let me stop a second and see if there's any questions so far.

Okay, no, no questions. So, um, let, let's, uh, in the, the next, uh, 15 minutes, I just wanna go through, uh, two, two or three ideas. So the first idea is this concept of the data journey and why it's the missing piece. And so if you think of what you have in the world, you have data in place, in a data, in a bucket or a database, but you also have data in use as we saw it, follow the bouncing ball from data, data representation.

And there could be all these errors in the data. You could have the wrong row count, you could have some business rule validation. Sales are down 50%. The model could not predict, the dashboard could be empty. Uh, your extract could be low, and those are all data errors. But you could also have sort of process errors or the tools acting upon it.

Maybe something didn't start, maybe you got an error message, maybe you got a scheduling error or there's an order of operations. Um, and then lastly, you could have things like basic IT monitoring. You could have a log file error, or your disc is, is, is going too much, and you wanna be able to be notified about all these and, and, and understand what's happening.

And so for us, we think the most important thing is be able to sort of look across and down the data and analytics stack, being able to say where it is, have that 23-year-old, very quickly point to where the problem is, and then be able to go quickly to the system. And you can do that, uh, in our tool, like I said, either by and I'll kind of looking at observability here, sharing this tab can either do it by sort of management, by inspection, by looking at the UI or management by kind of drilling in, uh, to each one of the tools.

And so, um, that way we can, uh, be able to keep going. And so we bring this idea, this across and down idea, and we call that the data journey. And I, I think that's a really interesting and missing piece of every data architecture. And I, and I think it's actually conceptually separate from data lineage.

Um, and data lineage in, in my mind, is a blueprint, sort of the as is blueprint of what the tables are, what their relationships are, um, with a data catalog, what's the meaning? Um, and those are good, right? But, uh, I think a, a building blueprint's an amazing thing. Uh, however, if my building's on fire, I want the fire alarm control panel.

I wanna know in which room that the problem is so I can go to it quickly. And, and that's what we, that's the purpose of the data journey. It's sort of the connection of all the pieces. Um, and then allows you to drill in and find the right piece at the right time, at the right level to be able to get at it.

And maybe you're gonna use, uh, lineage to diagnose it. Maybe you're just gonna go open the source code to figure it out. So that's one thing that's conceptually interesting. And then the other thing I think is really about TestGen and, and, uh, what we try to do here and, and we, we've built data quality validation tests, at least my team and I for 15, 18 years.

It's, it's, it's not new. Um, and so the challenge is like knowing what tests to do. And, um, what TestGen does is it baseline profiles your data, and then it does these 27 kind of automatic profiling hygiene tests, and then it does 32 auto-generated data quality tests, and it can run those, uh, really quickly since it's sql.

So, uh, in 20 minutes you can get 15,000 pests run. Um, and so it allows you to sort of drill down into, to relevant data track issues. It allows you to tweak and sort of makes these things visible. And also coming, we have sort of eight more sort of multi column fill in the blank business rule tests that are, that are coming.

And, and we've been using these, uh, uh, the real way we built TestGen is we built all the SQL first over years. And we've been using that our customers and our consulting engagements for years and years and years. And so what we've done is really productized it. And so we're gonna productize these last sort of fill in the blank things several months.

And so in the use case of data evaluation, we've got these 27 data hygiene detectors. You can go into our documentation and you can, when you walk through the, um, uh, the installation of the product, uh, you'll be able to go in. And so if you wanna look at, for instance, um,

uh, our DataKitchen, uh, documentation.

00:35:00

So there's a page here that kind of brings you into how to start doing it. And so it, uh, you can go right into the, uh, uh, installer. Um, we did a lot of work to make the installer easy, uh, on multiple systems, but installing software is often challenging. And then there's some use cases where you can walk what will go that actually map pretty much to what we're talking about here.

Um, uh, data, pre-production, anomaly detection and production. And, and we, you can walk through the demo, uh, and we can set up, uh, each one of those so that that's maybe a tip for you to kind of, uh, get going. So kind of going back to the slides, um, you know, we've got these, uh, uh, data evaluation phase tests.

We've got ingestion tests and anomaly detection tests. We've also got sort of these automated data quality tests. 32 of them. We've got two custom tests and, and eight business rule tests. And, uh, you can kind of see the breakout between one, one things that are automatically generated and, and ones that have to be configured.

Um, and then lastly, you know, the question is, is this magic? And, um, how, how do you know how to do this? Well, I think there's a bunch of algorithms behind it, I think. Is it ai? Is it machine learning? Well, technically, maybe. Um, but it's auto, it's automated and it's intelligent. And so I'll, I'll, I'll stand with that.

And so, kind of walking through a couple things. Here's another view of our architecture, um, uh, similar to the one I showed before. Uh, we have a list of our agent availability and, and, and, uh, ones that are available. And again, you can kind of go to GitHub and see, uh, lastly, we've got a breakout on features.

And, and really open source observability is meant for a single data engineer. Um, you can do everything with it. All the tests are there, all the features are there. Um, the only things that are, are missing are really having more than one user, or in the case of TestGen, more than one project in a project is a way to group journeys or, uh, things together.

Um, and in our cloud version, we have some analytics, um, that go with it. And so, uh, there is no sort of functional hobbling on it, uh, and to be able to get done. Um, and lastly, like how does this compare? And this is a bit of an eye chart here. Um, but like, if we look at follows the rows here from the sort of new data cleansing, uh, I think we're much stronger from the sort of polling and anomaly detection.

I think we're comparable. Um, I think there are probably other tools that do better sort of, uh, anomaly detection than we do. But that's an area that we're, we're gonna work on and see what the community wants. I think we're much stronger in production due to the data journey and due to automated data, uh, tests.

And the same thing with, um, development because of the data journey. Um, and, uh, so lastly, why, why should you care? What's unique? Well, the data journey is a unique concept, right? It's end-to-end visibility across and down your data estate, um, auto generation of data quality validation tests. It's so no coding, no yaml, it's all done.

And it's two full featured with UI open source products, and they're complete for an individual. And so lastly, aware of this has been successful. Um, I want, uh, um, you know, we've used this in, in a bunch of different industries. Um, the idea here as an example is that, uh, here we're being able to walk through each step, um, and here it's sort of S3 Redshift, um, some sql, some Python in, uh, code, some more sql, some Tableau.

And how do you find the problems in this? And so it's about testing at each step. It's about testing and production. And, and this is interesting because there, this is refreshed every 30 minutes in a system that integrated 70 data sets built by a very small team, team. On the data side, it was two-ish on the, uh, business side, it was one-ish.

And this is very productive and very, um, like 10 x productive from a, a pharma standpoint. But this degree of automation, all the tests that run against every level, um, and this is all sort of production, uh, monitoring. 'cause it, it's refreshed every 30 minutes. Um, and, you know, uh, it's really hands off.

Um, it's low errors, fast changes, and in some ways we've taken the visible and made it, uh, the invisible and made it visible. So lastly, kind of concluding, um, DataOps. So, you know, we've been talking about DataOps, we've written a bunch of books on it, uh, training programs, et cetera. But focusing on production errors, you know, production errors, embarrassing errors, people finding out problems that leads to shame, that leads to low pro productivity.

So by focusing on that, you gain a big chunk of productivity, but you also can gain productivity by focusing on, uh, increasing your deployment speed and risk. You can also gain productivity by automating things, parameterizing, libraries, all these things actually end up kind of going into sort of the 10 x benefit. And it's not just us saying it, even though we've experienced it.

It's, it's, um, there's several reports, uh, by Gartner that validated. And so it's better, it's faster, it's happier, and yeah, you have to do some work to automate it. But, you know, we think that, uh, our, our open source tools can help.

00:40:00

And so we've got two open source tools, as I talked about. We also have an automation tool. Um, and so, uh, another thing we're having a a four-part sort of certification series, uh, on, on sort of how to do data observability and data quality testing. And, uh, my co-founder Eric and I are, are walking through it and actually uses the open source tools.

So we're gonna go into detail about how you profile data, how you find anomalies in data, how to do data testing, different data testing techniques. Um, you know, Eric's just been doing data testing forever and it's got a huge background on it. And so this is free. Uh, you'll get a certificate at the end, um, which, you know, you may or may not find valuable, but I think, uh, uh, this'll be very helpful.

'cause I, I, I've had so many people say, well, exactly how do I test? And like, how do I do this? And so this is meant to, um, do this and, and we're gonna use our, our open source software as a way to do some tests and answers part of the questions. Um, and then lastly, there all is a whole bunch of other free resources, right?

We've written a manifesto on DataOps and one on the data journey. We've written a cookbook that's had 30,000 copies delivered. We had a second book on how to, uh, transform your organization. We do have a DataOps concept certification. And then of course now the, um, the open source software that you can be able to go, uh, and, and get at.

And, um, you know, you can kind of go in and, and click directly into GitHub and you can actually see the installer. And, uh, it's pretty short. It has, you know, required Docker, composed mini cube helm to be able to go do it. And it's not too bad actually to go in and, and, and get it going.

But like I said, we, we, uh, it's still, we only released it a few weeks ago. We're gonna run into some in installation issues. Uh, get on Slack if you have problems, or our team's gonna be on it to help you. So, um, that's really it in terms of my, uh, overview of, uh, open source.

Uh, I'm gonna stop sharing and see if there's any questions. So anyone have any questions?

So I urge you to give it a try. Um, it's pretty easy to go get going. Uh, if things, uh, if, if, uh, we're here to support you. And again, thank you for your time. Uh, I look forward to seeing some of you in our, uh, our certification series. Thank you much.