Data Quality in a Medallion Architecture

00:05:00

Hello, everyone. My name is, uh, Chris Bergh. Uh, we'll be starting the webinar now. Um, so, uh, in, in case you're confused, you're here for a discussion of data quality in the medallion data architecture. My name is Chris Bergh. I have a long background in technology, data engineering, um, and DataOps. And I'll be your host and, uh, leader of the discussion today.

So, uh, a few housekeeping things. So our slides and recording will be shared, uh, to everyone who joined this week. Uh, I wanna apologize for, uh, uh, including everyone's email and letting everyone see everyone else. I, I fixed that someone sent me an email. Um, if you have questions, put them in the chat window, which is the third button from the right on the bottom, and I'll answer them at the end, or we'll answer them during the discussion.

And we've targeted 45 minutes, uh, with our presentation, um, again, with questions at the end. And, uh, just a trigger warning, I will make some bad Olympic based dad jokes, so I'll be worried about that. Uh, if, if that sets you off, I, so what, what are we gonna talk about today? Uh, so we're gonna talk about the, um, really about the medallion architecture first, why it's interesting, what it means, um, and then we're gonna talk about the data quality challenges within that architecture.

Um, and then we're gonna go through each layer, the bronze layer, the silver layer, and the gold layer, and talk about how to do data quality in each. And to do that, we're gonna use the perspective of our open source, uh, data quality tools as a way to, to, uh, move the discussion forward.

And then we're gonna talk about regression testing, which is the ability to make a change to the code that's acting upon the data in that architecture. Then we're gonna talk about data quality scoring within the medallion. Um, and then we're just gonna, uh, talk at the end about, um, kind of do, uh, three or four slides that talk about, uh, what we do and, and how you can win.

So I hope that's what you expect, but that's what we're gonna go through today. Again, if you've got questions, put them in the, the chat window on the right.

So what is the medallion data architecture? Well, I guess the, the way I think of it is, if you went to a data conference circa seven or eight years ago, everyone talked about the idea of a data lake, and they wanted to get their Twitter fire hose and all the data in a data lake, and it was on Hadoop.

Um, and so that is in, in some ways, uh, what a, what the first layer of a data architecture, uh, of a medallion architecture is, it's a data lake. Um, and so it captures all the data, all the raw data and, and, and sort of unprocessed or original form. Now, the next layer is an intermediate layer, which kind of goes some cleaning, some entity resolution, uh, kind of to say what, what a, what is sort of the unified enterprise view of your core business entities, um, and, and data.

So it's, it's sort of a, a, a, a clear copied version of, of what's in the bronze. And then really the gold layer is, is kind of the analyst ready layer or the data that's useful for your customer. And so those could be, you know, one big table. Those could be facts and dimensions.

They could be various schema designs that are available for bi and advanced analytics. And, and sometimes there's a platinum layer that people use after the gold. Um, I, you know, I've talked to customers who their bronze, silver, and gold layer are all in bucket stores. And then their platinum layer is actually in a, in a warehouse.

Now, sometimes the bronze and silver and gold are called L one, L two, L three after level one, level two, level three.

00:10:00

And the medallion data, uh, architecture has been, uh, I think primarily by Databricks, but also Snowflake and other big cloud vendors as, as called a data lakehouse or a data lakehouse architecture. Um, and so, um, there are other data architectures out there, and it's not the only one, right? Um, so like the one I grew up with is you've got a raw and staging files somewhere.

You have something, uh, a data warehouse, and then, which is, uh, and then you build marts off of it, which are sort of, uh, function specific.

Um, so there's a, uh, a question posted in qa just so everyone knows. I will share the recording and we'll share the slides to everyone in an email.

And so, um, uh, so, uh, so that's one. There's, uh, streaming architectures, the so-called Kappa and Lambda architecture, where the streaming to the, the, uh, data landing or a data warehouse zone has a differs. There's called, there's a, a data mesher domain driven design. Um, there's a, just the traditional data lake, but all these data architectures in some ways are, have the same challenge, right?

They, they are bringing data in from a source, they're transforming it in some ways to be useful. And that means you, you, you need data quality testing. So what does a, a, a data lakehouse look like? And so, um, so if you, uh, I, I've used a lot of, uh, diagrams from a tool called xra in this presentation.

So if we look at kind of the further for left column, the blue column, these, think of these as files that are coming in, maybe from FTP or some source that's coming in, and those files are sort of landed somewhere. Um, I think of them as, you know, uh, maybe they're a parquet file, maybe they're a CSV, maybe they're in a bucket store in a disc somewhere, and then they're loaded into some table that's in a database.

And that, that's really your bronze layer. And I, I think of it as the true representation of your data. Um, if it's a CSV file, it's a CSV file. Um, I, I, I, I think of it treating it as immutable. That means not changing. You don't do any work on it. It's just sort of there.

And, and, and so why is it good to have all this data? May maybe some of it's useful, maybe some of it's not. Well, there's always, um, the case of your data scientist who wants to get at the finest grain and the rawest data, so at least you have it. Um, and I'm a big fan of, uh, immutability based data architectures, and having an immutable layer is, is important.

So that's the sort of bronze layer that's in the second column on the left. And then the middle column here is called the silver layer. And that's clean data. So imagine I've got some transactions and I remove the duplicates out. That could be a cleaning, or I've got some, uh, maybe I've got some Salesforce data.

And again, I've, I, I've cleaned it and I've, you know, I, I've loaded some zip codes and maybe I've noticed that the, the, the leading zero isn't there and the raw data. So I added in. So all the di zip codes are now five digits. Um, you know, maybe I have A-A-A-A-C-S-V file and I can't parse the last row.

And so in the clean table, I actually have, you know, I have 50 rows in the partition table, but 49 in the clean table, and then the, the, um, and so it's not really aggregated in the silver layer. It's sort of cleaned up, uh, in it. And, and so, uh, in a lot of ways, maybe that would be where the data scientists would, it depends upon their, their use.

Now, now the gold layer is actually fit for consumption or, or analyst ready. One way to do that is with a, a, a, you know, a set of facts and dimensions. Uh, another way is sort of one big table. Another way is you build a bunch of function specific, uh, report specific tables. Um, uh, you know, some people call these models in the DBT world, but, but think of it as ready for consumption.

That means your report, that means, uh, uh, your, your consumers all should point at this. And so this is just an example of why you would do this, mainly because you have an immutable landing layer, you've cleaned up your data, um, and then you have, uh, uh, a, a gold layer for consumption that is targeted at kind of mostly all your, all your use.

So I hope that makes sense. Um, and, uh, uh, I'll keep going here.

So what are some of the data quality challenges, uh, in a medallion arch architecture? So the bronze layer, I think of it as the place that an a data ingestion team works, or in some bigger organizations, data engineering teams are solely responsible for ingesting data. Um, and so I'm bringing in data

00:15:00

and how do I know I've got enough of it? So how do I know it's the right shape? How do I know, um, that I've got a whole host of particular column issues? Like do I have the right zip code? Um, or is it a zip code? Are they, you know, character strings, ABCs in a zip code column?

And so those, I think of as typical ingestion, um, quality issues, you know, I've got, um, uh, sort of freshness volume schema, and then sort of a whole set of data quality issues. 'cause that's really your landing data and your continually landing data from, uh, from, uh, your data source. And so this is a very common, uh, issue in every data engineering practice is that your data providers, they kind of don't care that you exist.

They will drop new data, they will forget that you're giving data. They will change data schema at the last minute and not tell you, um, and all those fun things that we have to deal with as, uh, as, as a data team. Now, um, if I go to the silver layer, um, and that's where I've got clean data, well, I, I'm copying data into another layer.

So there's always the case that my copy didn't go. So does the number of records, uh, match between a customer master in the bronze layer and a customer master in the clean? It may or it may not, right? You may have duplicates, uh, in your customer rack master that, that aren't fit into the silver layer, but, um, it really sort of depends on how you work this.

But it's important to make sure that it have some way to match the customer ma master records between each. Likewise, you've got a customer dimension in the gold layer. How do you tell if you've got the same number of records or the same number of customers, or that if one layer is delayed versus the other?

Um, so there, uh, uh, and then finally, the set of challenges around gold. And so if I look at the gold layer, that's where I filled out the facts and dimensions in this, in this one. Like, how do I prove that I've actually got the right number of sales transactions, right? I'm my in, in this example.

The bottom row is, I'm, I'm getting sort of, uh, it's a really simple example. I've got my MDM system in, and then I'm getting a customer sales transaction. So customers, products, and, and sales data. And so, you know, a very simple way to lay that out is you've got a customer dimension, a product dimension, some sales facts and time.

And so how do I know that I've actually got the right number of sales transactions that your customer and product data is actually correct? Um, again, because you're copying data from one to the other. And so, um, all these are very typical data quality challenges in this architecture that have to do with the fact that you're kind of, uh, uh, duplicating data.

Um, and also the fact that you're, you're transforming data into analyst ready for, uh, format. And so very common, uh, not unusual, but, uh, there's some things that we have to think about. So what are overall, like in every data architecture, what are data quality best practices? So the first thing is find data errors before your customers.

Um, and that means, you know, I think all of us have had the misfortune of having, uh, a data customer say, this looks weird, or your data's wrong. And then having to spend all day or all weekend fixing it. And then when you find errors, find them as soon as possible in the processing of the data.

And that's where a medallion can help, right? Because you have a bronze layer. Well, you should find problems there for as, as opposed to, uh, doing it into the final gold layer. That way you don't have to, um, rebuild or, or reprocess data and then be, uh, don't be reactive, be proactive. And you do that by automation.

You automate data quality tasks, you automate tool observability. Um, you try to build a mission control on top of it. It's really about not waiting for your customers to find problems, um, but you find them before your customers automatically and not manually. And then in the case that you do find problems, always putting in a check.

And that, that really goes to the, the third bucket is a reflect and improve mindset. You know, you wanna set up a, a learning kind of a love your errors culture. Like don't shame and blame people when something goes wrong. It's kind of an opportunity to improve. Um, and, and I think the first step that we've talked about is, is, um, it's really set up if you, if you're doing none of this, uh, in, in quality, set up a quality circle.

Get your data engineers, maybe your governance people, maybe a customer, um, and keep a list of your errors and try to focus on fixing one of those errors. Um, and then be able to react, react quickly, right? If you want to be, not wanna be reactive

00:20:00

and be proactive, be able to kind of think of, I wanna have sort of a mission control and look at my errors and count my errors over time. And then lastly, there's this term of safety culture, which comes from, uh, you know, comes from manufacturing, right? Be allow people to say this, allow people on your team to raise issues like trust your teams and, and work on your process that your team works in.

And kind of don't trust the sort of vendor or analyst BS that like, Hey, I've got this cool tool, it's gonna solve everything. Like, it, it really is about improving your, your how your team works. And I think these sort of five categories are in variant of data architecture. In fact, they're kind of, um, they're, they're really the, uh, I think the way that team should work in order to get on the, um, to, to just drive error reduction down.

And, and I think, uh, we've written quite a bit about the, the reduction in errors in production leads to a, a, a rather large step change in productivity of the team and in customer data trust.

So how do you do this data quality in the bronze layer? Um, and so, uh, and so what I'm gonna do in the next three sections, kind of talk about how you do data quality in each layer. And so again, the bronze layer is the ingestion layer. And so to do that, I, I have to think in the framework of, of how I understand the world and how I understand the world through the point of view of the tools that we've built.

Luckily for everyone, they're, they're open source, Apache two oh and freeze. We've got two tools that we're gonna talk about today and, and, and use to explain it. One is our, um, uh, completely fully functional DataOps, data quality TestGen, and the other is DataOps observability. And so, um, if, if you wanna know about those, uh, you can download them and try them.

We also have a seven hour sort of data quality and, um, observability training and certification that you may want to, uh, uh, check out. We've had, um, uh, close to six, 700 people take it since we've released it a few months ago. And so how do you do data quality in bronze? Well, again, the challenge in bronze is I'm getting new data all the time from all my suppliers.

So I have maybe 10 data sources, hundreds of files, hundreds of sources, right? This is a very simple example. Um, and so how do I know what to do, right? How do I know what to check? And so we think of it in two ways, and this goes into our training and our point of view with the tool one is that I'm going to get new data sets into my bronze layer, and to do that, I have to profile them.

And when I profile 'em, I have to prove whether they're worthy of being put into my architecture. I have to say that I can pass this new file on, I, I, I, maybe I should push back on it or patch it up myself, and that, that really means data hygiene screening. And then I'm, I've done that.

I've said, I've got a new file, it's in production, now I'm getting incremental updates to it. Maybe I'm getting 10 rows, maybe I'm getting a full replacement of that file. And to do that, you've kind of gotta do these four buckets. Um, like, check, did I get something new? You know, and, and did I get enough? Right?

And does it fit in how I think? And then does it make sense kind of conceptually did it drift from it? And so, um, this is the way that we think of how to do data quality in the bronze, because there's really these, these two parts, right? And, and for us, our, our point of view of our TestGen tool, and this is, um, gonna kind of inform how we think about testing in each one of these, is that to look at sort of your database data time one.

Um, and so our TestGen takes it and profiles the data and then builds a baseline set of data quality tests automatically. We, we do something called automatic data quality test generation. And it's kind of smart way of doing it. And it's based on that database tests at time one. And then, uh, when time changes and there's more rows in, well, you can then run those tests against it.

You can say, well, I've got, you know, I, at, at time two, I've got a hundred more rows. Does that make sense? Or, I've gotten a bunch of fields that look like zip codes and they're not zip codes, so that's probably a problem. Or, I've got a new product, I have six product names, and now I have a new product name.

And so all those things are, are based on this idea that you can learn from what happens with, uh, the change by profiling the data and generating automatic data quality tests. And so our workflow in this is sort of profiling, screening for hygiene gotchas on new files, generating data quality tests, and then executing the tests.

And so you can learn more about this in our, um, in our, um, um, in how we do this in our certification program and with our open source tools. But again, the, the most, uh, the, you know, every data person does this. I've got a new data set, is it worthy for me to do?

00:25:00

And and that's really about profiling it. We've got 51 characteristics. We're not the first company to do profiling. There's lots of tools. And then we've got this automatic sort of hygiene suggestion, like maybe you have this term, like if you can look here, there's, there's a term for e-bike and we've got six different ways to write e-bike.

Maybe you should standardize it, maybe you should not. Maybe you should tell your data provider to do it. Maybe you should patch it. Um, and this is a case where, um, patching the data. So for instance, if you look at, um, in this slide, we've got the word e-bike here, written six times in six different ways.

This would be a great case for you to fix in your, uh, uh, in your silver layer to be able to go in and, and sort of standardize it there. Um, and then likewise, when you, you're continually ingesting new data, you've gotta sort of look at anomalies in that data. And we have a bunch of automated tests that do that.

Um, and then being able to alert on it, and we'll talk about that. So, so that's sort of the background and, and how we think and how we're gonna talk about this. And so in, in general, um, uh, you know, doing the bronze layer is very important, finding problems there first before they get into the silver layer.

But there is, uh, interesting challenges in the silver layer itself. And, and, and one I think is like, how do you actually know since you're copying the data that you've got the right data? Um, and so I've seen a lot, I I've talked to a number of different people who've, who've done this. And I, I I guess most people, first of all don't do a lot of data quality tests.

They kind of build it and hope that it works. Um, and so like one way to do it is sort of table by table count matching, where you say, I've got 50 rows in the customer master, and then I should have 50 rows in the silver layer. Now that's problematic, right? 'cause you could have duplicates.

So I think there, there's table counts and perhaps could be smarter with entity counts. Um, but the way that we think about it is that you, you should be able to go in and, and run the same version of the data quality tests that you ran in bronze. Now, it depends on, on how you've done your silver layer.

Um, I don't think the, the market has really laid out exactly how you want to do the silver layer. What's an exact schema copy of what's in bronze, or it's, the schema is modified slightly and some of the entities are, are changed and renamed. I, I, I don't think there's guidance in what people have written.

I, I, I guess I've heard it done both ways. And so, for instance, you could have the customer name be abbreviated, um, or the, the name of the column that is the customer name be abbreviated in the bronze layer, and you're gonna fix it up and make it long in, you know, uh, uh, in the silver layer.

And I think that would be perfectly acceptable. So you have in, in essence two, two similar, but different schemas in each one of these zones. And I think that's a challenge, right? In, in, in a lot of cases. Um, one sort of obvious way to do it is to write a big old extract and, and do sort of a row by row count comparison.

I, I, you know, I, I find that incredibly inefficient. There are tools to do that. Um, I, I'm not sure that that is the right way to do it. I've, I talked to a, a company, uh, six, eight weeks ago, and the fact that they were sort of theoral or taking the data out from each layer and doing row by row comparisons meant their QA process was actually taking too long.

It was fa it was slowing down their production process. And I don't think that that's necessary here. I think you can actually build a whole set of data quality validation tests and run them on the silver layer. And if you've got anomalies, um, and you sort of anomaly check each layer based on, uh, its own schema and look for variations on that.

So it's a very similar case of what you've done with the bronze layer. It's just doing it again in the silver layer. And by running this whole series of data quality checks, you sort of remove the need to actually copy data from one system, copy it from another, and do a data diff. Um, you know, I, I, I think that that's a more efficient way of doing it.

But, uh, I've still talked to customers who want to do data diffs. So kind of going on to the gold layer now, if you remember, the gold layer is put in a customer specific form. Um, and so how do you actually make sure that it's right? Well, I think in, in general, you've got the same idea.

If I can generate data quality tests at every layer, and then I can look for variations from that, I've got a set of tests that run. Um, and so I, uh, what's in, what's interesting here is you've got something that's, um, a way that you're putting the data together so you can start to look at building tests that represents kind of your business view of the customers.

So we talked in our, our discussions and trainings about something called a historical balance test. And that way I'm looking at kind of sales per product per period and saying, I've got my top products,

00:30:00

I wanna look at their sales for the last six months, and then I want to compare them between what was given before, kind of looking at it through the lens, or I wanna look at my top customers and see what their sales are. Um, and that's the way your business people look at the data, right?

And, and sometimes you've, um, can have them look at the data and they can in sub-second say, the data's wrong. And, and what they really have done is kind of looking at it from these important entities and trying to look at kind of the amounts and metrics based on those. So doing these business metric desks where you're sort of joining across multiple tables is a really, uh, important test that's in the, in the gold layer and can, can inform you, um, where the problems are.

And so, again, you could also do things like entity tests, like I can find my number of customers here and my gold layer as well as my number of customers in, in the master layer. So there's really kind of two buckets here. One is sort of this business metric tests where you're joining across multiple tables.

Um, and then there are really your baseline sort of gold data tests, the, the drift tests, um, and then there's count matching. So these three types, I think are, are sufficient for you to be able to, to do gold testing. And so, um, you know, I, I have in my past taken and completely transformed a gold layer into a, a de-normalized table, and then mapped it back into a customer, uh, uh, raw table, um, and done a big old SQL accept statement on it.

Um, uh, you know, that sort of row by row comparison, that's another way to do it. I just don't think with a, like a technology like TestGen, you know, I didn't have it back, back seven, eight years ago. Um, and so we sort of remove the need to do that.

Um, so kind of keeping going. So, so I've got in production, I've got checks at the bronze layer, I've got checks at the silver layer, I've got checks at the gold layer, you know, bronze is ingestion, silver is cleaned up, and gold is, is analyst and, and customer ready. Um, you know, primarily the gold layer, we're, we're doing things like looking at multiple tables and, and the joins against them and the silver.

Um, we're trying to make sure that we've, we, we've, uh, captured all the cleaning, right? And then the bronze is sort of basic ingestion testing. So that's running. So production's running right now. Now, how do I make a change to production? How do I change my pipeline acting upon it? How do I change a model that's tools that are acting upon the data?

And so what, you know, what that means is I've got a development environment, I'm developing something, and then I'm putting it into production. And that's really what I'm talking about. So one way to think about this is, is sort of the simplest way to think of it, is that is your production's like a bunch of your production environment's a tube, right?

You're, you're taking production data and through a bunch of tools and you're getting a result at the end. It's, uh, getting transformations, modeling, visualizing, and think of it as a factory, right? You're working through these factories, and in these tubes are pipelines and, and there's tools acting upon data and code in those tools.

And what we're trying to do here, and, and any system is give you a big red light, green light, all my tasks and monitors, everything's running. And, and that's really kind of the, the goal of, of, of, um, any sort of data quality and observability system is that red or green light. Now, in development, you want a very similar case, right?

You've got, instead of the production environment, you've got a development environment here. And instead of production data, you've got test data, but you've still got this same kind of, think of it as a big two running through. And what you want is sort of, I wanna be able to run a very close sets of tests and monitors that I have in production, in development.

Uh, and really what you're doing here is saying, I'm, I'm keeping in production. Your data varies, but the code, the data varies, but the code that's in each one of these tools, um, is the same. But in development, your data's fixed, but your code, that varies. And so what you're trying to do is say, look, I want to be able to make a change, and I wanna see if there's problems in developments based on that test data based on that version of code that I'm running.

Um, and so I can see all the impact and yay, I haven't broke my, my code change hasn't broken. I can samely safely push it to production. And that's the idea here. And, and, and with Medallion, it's a very similar case, right? I've got, um, on the left side, I have maybe a development environment with one developer.

I have a QA and test environment where multiple developers are, I have a production environment, I have a method to move artifacts, code configuration, um, from one environment to the other. That's, uh, normally called CI and cd, continuous integration deployment. But really it means I can copy things from one environment

00:35:00

to the other safely, um, and repeatedly. And so if I'm in this QA and test environment, that means I have test data. And since I am in a medallion architecture, I want to be able to say, if I change something here in my customer master, like take a really simple case, I changed the schema in the bronze layer, I renamed something.

Now did that because I renamed a column in the master file, did that break this customer master? Did it break the customer file? You wanna be able to find that out in development and not sort of wait to production. And so there's a lot of coupling in these systems, and you need to be able to sort of run data in it to actually see that coupling using your tools and running the data quality tests and observability.

So you're kind of doing very similar to what you do in production, but you're doing it in a development environment. And why, because it's combinatorial complex, um, and sort of just looking at a blueprint, you have to, does it work? You have to pour data into the whole system to run it and test data.

And that's really the idea of regression testing. And the benefit here is, um, you know, and again, Doug, we, we do some, uh, DataOps consulting with people. And often it comes down to a symptom of problematic deployment is you have the two or three most experienced people have to look at every change and sort of bless it, and they become the bottleneck.

They end up spending all their times in meetings because you've hired five new people, they don't understand the system, they maybe only understand one table or two tables, um, and they don't understand the full concept of how everything works together. And now your smartest, best people, he or she has to sit and review every change, make sure every works, say if there's an impact, and then suddenly your team bogs down.

Um, and, uh, what really has bogged down is you've created a bottleneck out of your most experienced people. And to remove that bottleneck, you need to be able to do, uh, regression, end-to-end functional testing and development with good testing data in an environment with a whole bunch of data quality tests. And that will remove that bottleneck, it'll enable your team to push smaller changes faster with less risk into production, and therefore they'll be able to be much more productive.

And again, this is the same idea as what we talked about. Work on your process as a team, remove the bottlenecks. Um, and I've, uh, in our consulting engagements, I've talked to so many people who have this problem of, of the one or two heroes who have the whole system in their head and everything has to go through them.

You don't want that. You want to be able to take in a 23-year-old, have 'em make a small change, and in this environment, see if there's a problem, um, that will improve your team productivity. And again, it'd be similar data quality, observability problem as in development just done in a different way for a different purpose, uh, in, in as production.

Now, uh, kind of skipping to another aspect of this. So, um, everyone talks about data quality, and then there there's kind of a second aspect of data quality, which is really kind of the, the, the data itself. Is it good, can it be improved? And now there's one way to improve data, which is like, you take that raw data and bronze, and then you sort of patch it or fix it up, uh, in your silver layer, that's perfectly fine.

Um, the best way to do it is to actually have the source system fixed. You know, if they've got five different ways to write the term e-bike, well maybe they should change their source system. And so how do you get leverage on people to do that? Um, so you don't have to patch the data or, or keep finding it.

So a, a technique to do that is, is what are called data quality scores. And so we've, um, you know, you can get a data quality score at each one of the levels. Um, and so we're actually building a feature in our product to do that. And the point of the feature is to actually give data teams leverage to be able to force their, uh, their source providers to fix the data so they don't have to fix it themselves.

And so the way that we think about this, again, is informed on our, our sort of agile iterative way is by profiling the data, generating tests, generating scores, building packages to influence the data o owners, and then iterating quickly. Um, instead of doing long analysis, start generating scores quickly, and we've got a new feature coming out where we're gonna have a score dashboard, um, and we're gonna be able to score in lots of different ways.

And so partly it's scoring based on DMA data quality. It could be based on source provider, could be based on CDEs, could be based on a dataset, and then be able to drill into those scores and look at the issues, then be able to send those issues very quickly to the data source provider so they can fix it.

And so the biggest, we did some market research,

00:40:00

had a webinar a while ago where the biggest challenge is there's a lot of people who wanna improve their, their company's data quality, but it's sort of like an ecology term, the tragedy of the commons. Um, the people who are creating the data kind of don't care about it 'cause it's good enough for them, and they don't care about what everyone else has to use.

And so, um, the way to get success is to give them, uh, is to be able to very quickly generate relevant scores, uh, that are relevant to what they do, uh, get their management to buy in, and then give them sort of packages that they can actually fix it quickly. And that's the goal of our, our new feature that we're releasing.

So lastly, how do we, how do we think about this, uh, in terms of our software? Well, we, we've sort of, um, put the idea of DataOps and data observability into five pillars and sort of thinking about how do you check data at rest in, in the tool data at use in your tools across all the steps, setting expectations and going up and down the stack to find the problems.

And so we've written a a little bit about this on, on from the five pillars of data observability, which is sort of, uh, being able to check every tool, acting upon the data, being able to drill in. Is it the tool, is it the software, is it the server? Uh, being able to look at the data at rest, the raw data, being able to look at the data at use.

This is really your, your goal level data. And then kind of setting expectations is, is the data late as the, what, what's our SLA or SLO? Um, and of course, uh, our tools support that. And then from a further reading standpoint, we've got some new white papers, uh, data quality, the DataOps way, we've got the data journey manifesto, we have our trainings.

Um, and to sum, sum up, um, having your customers uncover a data error and whatever data architecture you have is problematic. It's embarrassing, it's damaging, it hurts the cause of, uh, your company being data-driven. It causes productivity problems with your team. So the only way to fix that is sort of rigorous quality testing within the scope of a medallion architecture or whatever architecture you've done, automated data quality testing.

Um, and so, uh, and quality testing at every stage, uh, is essential, right? 'cause it catches issues in that stage. Um, and it minimizes the risk of data inconsistencies and, and, and customer facing problems. And then, um, you can by a, a lot of times teams are very, very busy. They have a lot to do.

And so it really becomes, how do you get this from a good idea to, Hey, we should absolutely do this. And I think what it means is that the idea that you're, you're always gonna be behind the ball, you're always gonna have data problems, you're gonna have customer features. The only way to get ahead is to work on the processes that you have with your team.

And that is testing, deployment, observability quality, all these things that, um, are important for your team. But if you can leverage them, you can, you can make it forward. And so, uh, uh, lastly, there's just a bunch of free resources, open source software, uh, uh, uh, 10 hours of training programs, um, two books, lots of resources that are free for you to use to, to learn more about this.

Um, and so I'm gonna, uh, stop and have, have questions. Again, I will send out a copy of the stack and a copy of the slides, uh, with some other things at the end. Um, and so, uh, if you have any questions, either put them in the chat or put them in the q and a and I'll look at both.

Um, so we do have a, a question from Daniella that she asked. Um, uh, and terms of question of, uh, kind of design of medallion architecture. So where, where would you say, uh, these things should happen? Sort of renaming source specific field names to business consumer friendly conform names. Um, and so sort of a field name question.

And so, um, you know, my view is bronze is immutable, so you just keep the crappy names that you have in there. And so it becomes just a design choice about where you want to do it in, uh, whether you do it in the silver layer or in the gold layer. Um, you know, I think, uh, in, in general, uh, uh, it should actually be in the gold layer.

You should have business friendly names in the gold layer because your analysts are gonna look at it. So, uh, uh, and it, in my view, it depends on the quality of your gold layer. Uh, if you have a really good analyst ready data that then they end up just using that and they kind of, silver ends up being kind of an intermediate product that you don't, people don't use very often.

So fixing it in the gold layer is much more important than the salt, the silver layer, because, but if your gold layer's not very good, it's not very complete and people are having to kind of bounce to other layers, then perhaps you should do it in the silver layer. So the second question is merging of multiple sources, for example, product master, uh, where different attributes might come from,

00:45:00

from, uh, different systems. And same for customer entity. Um, so like, where do you merge multiple sources? Um, you know, I, I, I guess I think of the silver layer as, um, kind of, I want to have my customer, I wanna have all my customer data, all my sales transactions in one place. Now if I've got two transaction files, well, uh, and maybe they have a slightly different schema in, in my bronze layer, I wanna have one transaction table, uh, in my silver layer.

So I would merge it in the silver layer because you may take those transactions and sort of break them apart into facts or dimensions or other tables, uh, in your analyst ready gold layer. So, um, I think of kind of, that's one of the perces of the silver layer kind of catching, joining, fixing, uh, these entities and, and, and these, uh, actions together in one place.

So I would do it in the silver layer.

So any other questions that I can answer? Um, all right, so we've reached,

so if we merge in, so, uh, new questions in q and a, Danielle again. So if we merge in silver, do we need to merge and conform the names as well? You know, Danielle, I, I, I, I, I'm not sure I, I can give you a a good answer. It's Curtis. I, I think it's informed by how good your goal layer is and how, how much are your users.

'cause like I, I tend to think about let's do enough to make ourselves successful. And if your users are getting, if 98% of their time they're getting information from the gold layer, then why does it really matter what's in the silver layer? Uh, you could, uh, you could conform the names in the silver layer or not conform the names if they're hitting your silver layer a lot.

Uh, I don't think it quite matters. Um, so, uh, I, I, I tend to think having a really good high quality, uh, simplified analyst ready schema in the gold layer is, is a success. And, and the bronze and the silver, um, tend to be used much, much less frequently. Um, uh, uh, but it all depends on, on your domain.

Uh, and so, um, and how your organization is split up. Sometimes the data team just does the bronze layer. You have an analytic engineering team that does the silver or gold layer. There are different teams with different management structures. And so it might be better, um, to have that, uh, you know, to have that fixed earlier in the process.

So it's more consistent because you have, uh, different teams working in a way, uh, different places. So, uh, maybe Danielle, I'm not giving the perfect, uh, answer, but this is, uh, the best I got.

Alright, well thank you. I'll send out the slides. I appreciate you. Uh, and we also, uh, if you wanna have any more information, uh, give us a call here at DataKitchen. Thank you so much for attending and have a great rest of your day.