On-Demand Webinar · 34 min

You're Massively Overpaying for Data Observability

Data observability is a commodity anomaly-detection algorithm with a six-figure price tag on it. Where that price comes from, and a demo of TestGen's monitors doing the same job for $100 a month per user and per database connection.

Presented by Chris Bergh

What you'll learn 6 points
  • Time series anomaly detection is not new technology. The models are commodity open-source libraries — ARMA, z-scores, random forest, isolation forest, Facebook's Prophet — and the implementation on top of one is about half a page of code.
  • A survey of publicly available data observability pricing put the average cost of monitoring a thousand tables at over $170,000 a year.
  • Over $500 million of investment capital has gone into the data observability space. The pressure to return 10x on it is what sets the price, so your data quality budget becomes someone else's exit strategy.
  • Most vendors charge more as you test more — per table, per credit, per run. TestGen Enterprise is $100 a month per user and $100 a month per database connection, invariant of how much data you test, and open source is free for one user and one connection with unlimited tables.
  • The new monitors feature learns when data should arrive, how much should arrive, and when the schema changes, and lets you add custom drift tests from a SQL expression. Freshness, volume, and schema are automatic after a baseline of about 30 runs.
  • Automatic test generation turns roughly seven months of senior data engineering work into minutes: around 2,500 data quality tests generated against a set of tables.

Prefer to read it? The written version is in Webinar: You’re Massively Overpaying For Data Observability.

Slides

42 slides

Transcript

Chris Bergh

Show chapters and dialogue 31 chapters · 5,693 words
  1. 0:00 Why we surveyed data observability pricing
  2. 4:02 What time series anomaly detection is
  3. 5:12 Where anomaly detection is already used
  4. 6:21 Half a page of code on a commodity model
  5. 7:22 $170,000 a year to monitor a thousand tables
  6. 8:32 What data engineers say about the cost
  7. 9:35 $500M of venture capital sets the price
  8. 10:42 Pricing that taxes you for testing more
  9. 11:50 A month of an engineer's salary, not a year
  10. 12:49 TestGen pricing: $100 per user, $100 per connection
  11. 13:47 No per-table tax and no row-count fees
  12. 14:50 How monitors baseline your data
  13. 15:42 The four things a monitor learns
  14. 16:47 Demo: profiling, hygiene issues, and the new monitors
  15. 17:45 Q: how is security and data encryption handled?
  16. 18:58 Reading a volume drop and a schema change
  17. 19:55 Table trends and the prediction cone
  18. 20:56 Custom metrics, scheduling, and sensitivity
  19. 21:57 Prediction model, historic calculation, or static threshold
  20. 22:54 Q: install complexity, and which databases are supported
  21. 23:46 Monitoring by exception when tables are cattle, not pets
  22. 24:44 Four DataOps processes that run on data testing
  23. 25:48 Pets, cattle, the assembly line, and the code
  24. 26:52 Reactive quality work versus proactive tripwires
  25. 27:47 Which feature covers which process
  26. 28:58 2,500 generated tests instead of seven months of work
  27. 29:52 Closing, and a comment on data quality and AI
  28. 30:55 Patching data, influence, and support
  29. 32:11 The connection wizard, and fast first value
  30. 33:09 Q: the gateway approach, keeping bad data out
  31. 34:13 Test coverage everywhere, and making testing cheap

00:00:00 Why we surveyed data observability pricing

Chris Bergh: Hi everyone. I'm Christopher. We'll start in about one minute. All right, we'll give it about 30 seconds and then we'll start. All right, everyone. My name is Chris Bergh. Welcome to the webinar today. As always in our web in our webinars, we'll share slides. We'll share the recording and the transcription. Probably later today or early tomorrow on Google chat. There's a a comment box in the lower right hand corner. Put your questions in there. And I'll go ahead and send those out later in the week. So, what's this all about? So, I I think we're really about releasing a new product feature, but I'm going to talk a little bit about the background of why we've released this product feature. And I had Claude go off and do a survey of publicly available data observability vendors trying to find out what it would cost to have a thousand tables monitored. And I was very surprised at the results.

00:04:02 What time series anomaly detection is

Chris Bergh: Mainly because it's really expensive, right? It's hundreds of thousands of dollars of years to actually monitor a thousand tables. And so that sets the context for kind of what we're going to talk about is sort of what is data observability and what is time series anomaly detection and why is it so expensive and then I'll introduce a new feature that we have that we call monitors. And then I'll talk about sort of the perspective on why we built TestGen and why its open source model and lowerc cost enterprise model are so exciting. So time series anomaly detection is not a new technology. It's been going on for a while and it really means I've got a some variable over time and I'm trying to see if something interesting happens, some outlier. And so there's different types of time series data, but at the core they're like, I've got some expected range and something has peaked out of that range. And so it could be I've got rows and something's peaked out of that rows. It could be a whole lot of things.

00:05:12 Where anomaly detection is already used

Chris Bergh: And there's techniques like ARMA and z-scores. And it has a lot of applications. So for instance in IT monitoring looking at CPU metrics or machine resources in fraud detection like for instance in unusually large deposit in a bank account for instance in healthcare looking at people's heart rates or in you know usage of manufacturing like oh all of a sudden I have a pump that's going out of whack. So time series anomaly detection is very much a useful tool and there's just lots of open- source libraries that that has done it and and they're really commodity right there's a number of Python libraries there's one called Prophet that Facebook has done there's some that are based on neural networks some that are based on different algorithms but it it's pretty much known what their characteristics are some are marginally better than others. Some are better for others use cases. So instance, we use in our own tool, we use SARIMAX for volume and metrics test and and something else for for custom tests.

00:06:21 Half a page of code on a commodity model

Chris Bergh: And so really if you look at it from a implementation, it's actually sort of half a page of code, right? I download the model, I put in the data and train the model, and I give it a new I give it a new score, and it tells me if it's in or out of the range. And that's pretty straightforward. It's not that much for someone to do in terms of implementation and and so there are these kind of commodity models and in fact most of them are pretty good at predicting things that are anomalous and so you've got random forest or isolation forest you've got ARMA you've got Facebook's Prophet and so they're all sort of telling you and and it's still pretty it's still an active area of research but in general they're all pretty good and so It's kind of a commodity algorithm that that is being used. And so at the core really what data observability is is is pointing this kind of commodity algorithm and pulling your database and doing time series anomaly detection on it.

00:07:22 $170,000 a year to monitor a thousand tables

Chris Bergh: That's really what it is at its core. And so it's not to say it's not useful, right? Is that when you look at data, especially data that has been recently ingested, you're going to find problems in it. And that finding it sooner, in fact, as soon as it lands, is really a good thing. And so, and if you don't do it, in fact, all all the parts of of testing data have the same characteristic. You waste time, you hurt your reputation. And you know, it also can mean that you you are taking data and and recanting it or re doing it over again. So it's problematic. So really this idea of data observability, it's kind of helping you be more reactive to problems, find problems before your customer sees them. And so very useful tool but also very expensive. So let's talk about well why is that? Why is this sort of commodity algorithm wrapped in software end up being so expensive? An average of over $170,000 a year for a thousand tables according to Claude.

00:08:32 What data engineers say about the cost

Chris Bergh: And it's not just me saying this, right? I went in and looked at did a Google search on data observability and too expensive and you know I got a bunch of Reddit quotes obviously and you know you could take Reddit with a grain of salt but like you know what the data engineer said in our data engineering is like it's expensive right and it's prohibitively expensive to cover your entire infrastructure and when you do do it you get a huge spike in monthly costs and following pricing changes and So why is what what's happening here? So the I think the first thing is that why is kind of data quality or data observability become so expensive? Why is it kind of a luxury good and you have to only monitor certain tables at certain times? And so it's kind of crazy like why, you know, why are you going to this really expensive restaurant that's serving you kind of this commodity food wrapped up in a nice wrapper? And and let's look at it that way.

00:09:35 $500M of venture capital sets the price

Chris Bergh: Like I mean really that what it is is there's these boxes of algorithms that everyone uses that's pulling your database and it takes a nice open source algorithm it wraps it up in a box and charges you basically an engineer's salary a year sometimes two or three engineers salary a year and that just seems to not make sense to me right and and I think you have to look at it from another perspective a sort of business perspective and that almost all these companies and there there are dozens of them have large investor bases. In fact, over $500 million in investment capital has gone into the data observability space. And so on that they have pressure to return the fund. They have pressure to have give a 10x return. And so that means that they've got to make that their own revenue number go way up. And so that creates this need to have high prices. And so in some ways their budget is really trying to you know help their exit strategy.

00:10:42 Pricing that taxes you for testing more

Chris Bergh: And we've also seen this in in other data vendors like dbt's pricing in increases. Fivetran is is notorious for this being very expensive for such a tool that moves data from A to B. And so that's the kind of world that we live in. And so what that means is you've kind of got a tax, right? As the number of tables or as your usage go up, your cost goes up. And so sometimes it's priced on per table, sometimes it's priced on credits, sometimes it's a very custom price, but the more idea is the more you test, the more you pay. And so to me that doesn't make sense. That's prohibitive because testing is a really important thing that we've talked about in many webinars for data quality, for data observability, for reducing production errors, for finding regressions. Doing lots of tests is a really important resource for you to make your job tenable and make your team more productive. So, you know, fortunately for our company, we don't have any investors.

00:11:50 A month of an engineer's salary, not a year

Chris Bergh: We've built it profitably now for a dozen years. So, we don't have growth at all cost. We don't have a board that's answerable to VCs or answer answerable to limited partners. We're not pricing to have a for our exit strategy. And so, you know, we've been profitable for 12 years. We're a bunch of engineers who are trying to make a difference in the data world. And so we don't think a year of observability data quality should cost an engine US engineer salary or two US engineers salary. We think it just costs a month of an engineer salary for an enterprise product. In fact, we think if you're you know you're one or two people, you should just be able to use it for free. And that's why we have fully functioning open source. And so that's kind of our principle really is that a month of an engineer salary should equal about a year of enterprise license. And so for a small team it it shouldn't be you shouldn't have to say do I have to hire or fire someone in order to just have find errors before your customers.

00:12:49 TestGen pricing: $100 per user, $100 per connection

Chris Bergh: That is to me it seems crazy. It seems like you should be able to it's just much too expensive. And so also it shouldn't mean that like if your data doubles then you have to let one another engineer go. That doesn't make any sense either because you're already too busy. So it's we've designed pricing that's sort of invariant of the amount of data that you have. And so what is our pricing on our enterprise TestGen? It really means that it's $100 a month for every user in the enterprise version and $100 a month for every database connection. And so open source of course is one user database connection, unlimited tables, Apache 2.0 forever free. And so you get a very predictable price. And so, again, looking at other vendors, you can every month they're charging you 5,000,10,000, 30,000 a month for it. And, and we're talking about a $100 a month. And so, one of our customers has got sort of 10 users and three database connections.

00:13:47 No per-table tax and no row-count fees

Chris Bergh: That's about $15,000 a year for the enterprise user. And so, no credit calculations, no row counting fee. Point at as many databases as as many tables as you want, use it as much as you want. And and that really points to our philosophy that testing is so important that you shouldn't ration it. It should be able to be used widely. And so and that's a very reasonable cost. So, what we've done is is we have a pricing model that doesn't tax per table. It allows you to use it for one user, one connection, unlimited tables. And we're a sustainable partner. We're not going to jack up your price. Because we're want you to stay as a customer. And so I want to talk about this new feature, the monitors feature now that really implements time series anomaly detection on top of data. And so what it does is it provides this way of doing it without sort of requiring you to write SQL or Python.

00:14:50 How monitors baseline your data

Chris Bergh: It does it based on implementing time series anomaly detection. So it learns from your data. And what it does is it allows you to work very easily. So you select a bunch of tables, you set a schedule and automatically monitor. There's really not a lot of work involved. Sort of zero friction. It takes does take some time to to baseline your data. But that's actually a good thing, right? It takes sort of 30 runs of you to make sure it happens. And so but we also allow you to set manual thresholds if if you want to do that or percentage thresholds but and and this is what you see. We have a training period in our software and then when the model has got enough information you can actually start to make predictions and learn from it. And then as your data changes that sort of prediction cone can be made wider or smaller based on configuration. And so what does it do? Well it does four things.

00:15:42 The four things a monitor learns

Chris Bergh: It learns when your data should arrive and tells you if it's late or early. It learns how much should be sent and tells you if there's too much or too little. It learns if the structure changes columns and it also allows you to build very custom data drift tests. And so we have freshness volume schema are all automatic. But if you want to attract sales over time or sales in a region in time or manufacturing per region, you can very easily build time series anomaly detection tests on top of it. And that fits our pattern. We want you to have 80% of the stuff work out of the box with in a few minutes and then being able to have a very easy UI for you or others to manage the custom tests. And so I'm going to walk through this today in the demo. But really what happens in here in the freshness test we compute a fingerprint of your tables and then compare it. And similarly with the monitor test, the schema variation and the custom metric tests.

00:16:47 Demo: profiling, hygiene issues, and the new monitors

Chris Bergh: And here's kind of some a little bit more information about about why that's how it works and what it does. And so let me and finally you'll see this in the UI. We we show what happens and we show it and we also present it. So I'm going to jump into the demo right now and and show this new feature. And so here's our product. It's TestGen. This is the open source version what you get when you or this is the enterprise version. But you get a very similar case when when you install it. And so when you log in you see a project dashboard. And the first thing that is what works on TestGen is that we we learn your data. We profile your data and that's at the bottom here. This profiling at the bottom. So we'll we'll you build a connection to a database. You select the tables and then we start profiling and we develop these things called hygiene issues which are ways to tell and then you can build a suite of tests that run against it and that can also be done automatically.

00:17:45 Q: how is security and data encryption handled?

Chris Bergh: But what's new here are these three buttons these three dots freshness volume schema and metrics. And so I just see a question how's the security and data encryption handled? Well, we don't encrypt the the we don't move or touch your data at all. So the data is all done from an architecture pattern. We shoot SQL into a database and then execute SQL against it. And so all the data itself and all the values are actually kept on prem. So this isn't a cloud product at all. And and we'll share I can share the security thing. So this is a completely self-hosted product. So, you don't have to worry about any data exfil exfiltration at all. And, so, and we've done there's a whole bunch of other good security stuff on here. So, it's a it's a perfectly safe product for you to use on any any data set. So, let's drill into this this monitor dashboard. And so if I look at this, it clicks on and I can see in our demo data, it's got some five tables kind of made up of three dimensions and two fact tables and a column on freshness and a column on volume and a column on schema and then a column on metrics and

00:18:58 Reading a volume drop and a schema change

Chris Bergh: It says what are the anomalies in the last 28 runs. But let's actually kind of drill in and show the changes here. And so I'm going to go in and make my lefthand side a little bit smaller. And so what we can see here is that oh what happened? There's a volume. So something happened in the row count. I can see that the previous count was 503. The latest count was 502. So that that's an interesting change. And also I can look into the schema changes. And if I click on this, I can actually see that I added two columns and subtracted two columns. Again, that that's that's interesting, right? Going in and and looking at the column F eBike sales, I can see that the row count went up and it had a 44% change. And this sort of tells me what's happening across each one of these. And if I drill in, I can actually start to see what happens. So let's look at the first debike customers.

00:19:55 Table trends and the prediction cone

Chris Bergh: And so volume schema and if I click on this one view table trends I can actually start to see the trends. And so one of the things is that since this is an algorithmic solution there's a learning period. And so I I showed the learning period here but after it learns it can actually start to make predictions. And so here it's updated updated updated all correct. So there's no problems. But here I see the volume drop. It's suddenly dropped and then I can start to see the cone of prediction in the future and likewise I can start to see these schema changes that show up. So in one view I'm getting kind of details about what happened to my table over time. Likewise I can go in and look at a case where everything's great. So in de by products everything's working fine. And if I look at the trends, I can see that I've got two custom metrics, max discount and average price that are all fitting perfectly in the bounds.

00:20:56 Custom metrics, scheduling, and sensitivity

Chris Bergh: And these are important. So for instance, we have pharma customers who want to track for instance the 12 month TRx sum over time and make sure that that for a specific product that doesn't break break bounds. And so I can look at another one which is here I've got F eBike sales. I see one volume anomaly. So I could click in and see the available trends. So you can go in and just see one volume discount here. And so these are also very configurable. So there's one thing I can edit the monitor setup. So I can look at the number of loop back runs. I can set up the scheduling of this to here. It runs every 12 hours, but I can configure the scheduling and also allows you to to kind of configure the sensitivity of the model, the look back and exclude sort of weekends and holidays based on holiday codes from different countries. And so that that's kind of the a minimum set of configuration that that we thought was was necessary.

00:21:57 Prediction model, historic calculation, or static threshold

Chris Bergh: Likewise there there are notifications so you can set up emails if you get an alert. And if I go in and look at these custom metrics and hit monitors, I could go in and here's how I configure my VA my volume monitor. I can use a prediction model or I can do a historic calculation kind of minimum, maximum, sum or average. Very sort of simple way to do this. Or I could have static thresholds. And so we allow you to kind of set these by kind of based on your data. But we start automatically with a prediction model. And then likewise, you can go in and and have metrics here that are custom. So here I have a SQL expression that says the average of the discount amount. And I'm looking to see if that happens. And so you can input little bits of SQL to track metrics that are important to your business. And so I see a question about how about installation complexity if it's self-hosted?

00:22:54 Q: install complexity, and which databases are supported

Chris Bergh: Well, you should try it. It takes, I don't know, 10 minutes, five minutes to actually do it. It sets up Docker Compose, running on a Mac. So, it's it's pretty, if you want to get into production, you probably need your own database and running it in Kubernetes and the enterprise version. That may take a little bit of time, but I I think you just try it and see how how how it works for you. We we spent a lot of time trying to make the installation work on three different platforms. And then Attendee has a question about can it be used on MDM databases? Absolutely. We support kind of half a dozen well maybe eight or 10 different databases out there. We're adding new ones all the time. I can point you to our our documentation and it's really doesn't really matter what type of database it is. But you know we do support of a a set number of them.

00:23:46 Monitoring by exception when tables are cattle, not pets

Chris Bergh: So, all this stuff is meant to be, you know, one of the things that we've we've talked about here is trying to make sure we have this sort of term about trying to think I've got a lot of tables to monitor, right? I'm not here's five is interesting, but a lot of people I don't know what's going on in my data. I don't really care. You know I'm only interested in in kind of looking at I've got hundreds of tables to monitor. So you can go look at the UI. You can monitor by exception. And we are able to actually help you handle that case where you don't have tables that are pets but t tables that are more cattle that you want. You got hundreds or thousands of tables that you don't know about. And I think that that's really an important part of our wanting to build this feature was that you know it takes testing is is a very important part of of what you do and in fact that's why we built this.

00:24:44 Four DataOps processes that run on data testing

Chris Bergh: You know we have a profitable company. We've been testing for decades. And we just think it's really at the core of kind of four critical DataOps processes. And the first thing is really data quality. You can't prove that your source data is good or not good, that it's not fit for purpose without actually checking on it. And trying to improve that is tends to be a really important activity for data teams. Likewise, when you're building a warehouse or an analysis infrastructure, you want to know that it's the data that you're getting is good as fast as possible. And that's really observability. Has it arrived on time? And then kind of production. I'm have a set of tools that are operating on data. I'm building star schemas or gold medallion layers. Is the data good as it goes through that whole process? And then finally, I'm not just doing data. I'm actually making code that's acting upon data. And so all these cases, data testing is a core piece in this trying to make sure it all happens.

00:25:48 Pets, cattle, the assembly line, and the code

Chris Bergh: And sometimes we talk about data quality, sometimes we use the word data observability, sometimes we use data journey production. But these metaphors of like, hey, data quality is really about making sure I've got certain tables are good. They're pets. Whereas data observability is I've got lots of tables going on. So we think of them as cattle. Data production is really the assembly line. I'm taking data through a step-by-step process, multiple layers in a database, multiple tools, and I want to make sure it works perfectly for my customer. And then lastly, these assembly lines are really governed by code. I want to make sure that my code is good. And so all these processes are incredibly important for data testing. And data testing makes them so much better. And so that's why we've built this tool and priced it because we think these processes are really important for you to perfect and work on. And so again thinking about this right if I'm a data quality purpose I'm trying to make my data fit for purpose and my scope is really perhaps just my source data.

00:26:52 Reactive quality work versus proactive tripwires

Chris Bergh: And so having good tests is really important for that. We've got data tasks, we've got data hygiene. And then anomalies in production. And so these top two, I think, are a little bit more reactive, right? Because you've noticed data quality, you're trying to improve it, or I've got some data that's arrived in my system, so I've got to ping the people downstream if there's a problem and maybe talk to my data providers. But the bottoms one, I think, are a little bit more proactive. Where I'm stopping production while producing data. Think of it as tripwires or stopping the assembly line. And so we have a you know orchestrators more are starting to have these features. We've had it in our orchestrator for 11 years. But the idea of building an assembly line and stopping the line if there's a problem and then a lot of companies are deploying code. They're starting to do CI/CD but they eye it up. They hope it works.

00:27:47 Which feature covers which process

Chris Bergh: And so we think having a whole suite of regression testing and thinking of this in terms of test coverage is also a way we've talked about that. So lastly, we've got a bunch of features that cover that, right? So data quality is about dashboards. It's about hygiene tests. We've got that covered. Data observability is with our new feature about monitors, automatic thinking of cattles and pets. Data production. We've got this idea of a data journey in our open source another open source product or integrating to orchestrators and then finally deploying to production and doing CI/CD. So we handle all these cases and that that's really important and we handle them at a reasonable price and so our advantage here and so is really the time it takes to do this. And so if you've got a code test and let's say it takes 30 minutes per test or maybe it takes five minutes per test or 10 minutes when you're having to to to use Claude Code to do it, but it still takes time to develop the test to know what tests to write to write the tests or prompt the test but you're still spending months of time to build a suite of tests to cover your tables.

00:28:58 2,500 generated tests instead of seven months of work

Chris Bergh: And so for us, we think, you know, every table should have these frequency volume schema tests. Every table column should have a test on it. And so we have this engine that can build thousands of data quality tests automatically. And so you can generate 2500 tests pretty quickly and you can take that seven months of B senior data engineering work and actually make it into minutes. And so that itself has an ROI. And so, what we encourage you to do is sort of start today. Download the tool. It's really easy to install. Like I just did an upgrade and install. Took like three minutes for the installer to work. We've got great support u for you. Just join our Slack and and do it. Put it in production on some tables. See if it works for you. If it doesn't, tell us why. But our whole point is for you to start using it in production.

00:29:52 Closing, and a comment on data quality and AI

Chris Bergh: And then when you want to have you found value, you want to start connecting to different databases. You could either have another instance of the open source to do it or start paying a reasonable enterprise $100 a month per database connection per user. And so that's really it for my my talk today. And so let me look at some questions here. And so, one question, without data, there's a comment here, without data quality, you have worse than nothing. Additionally, you can never have effective AI. And so, I I I'd agree with that, Attendee. I think, you know, having good data quality is really important. And so, however, in my experience, your data providers don't care about data quality. And so, you've got to influence them to improve it. And so, sometimes you've got a job to do with per data quality. And so the worst thing for a data and analytics team is just to throw up their hands. And so I've spent a lot of time in my day patching data and trying to fix it.

00:30:55 Patching data, influence, and support

Chris Bergh: And and if you look at sort of the medallion architecture, the middle layer of medallion architecture is all about patching data to make it analytic ready. And so I think the I've I've sort of come to accept that our data providers don't care about us on a data and analytic team, data engineering perspective. And so you've got to get leverage and that's our that's tests and as well as you've got to get leverage on the people to improve data quality. And so certainly I think from an AI perspective both the quality of the data and the quality of the context are also important. And so we've haven't talked too much about the sort of context that TestGen provides the test results the the profiling but that's also really important for AI models that that that becomes effective. And so and then a question around a support and so from a open source standpoint we we have a slack channel we do that sort of reasonable US hours plus to help you and then we have a similar support model for for enterprise with options for more support but if you find you work with DataKitchen because we are a profitable company we love our customers and you know We want you to stay.

00:32:11 The connection wizard, and fast first value

Chris Bergh: We we actually give very great support for both our open source and enterprise versions. But I encourage you now to to give it a try. It's super easy to install, super easy to try, configure it with some tables. We've even done something very simple in our our software to help you set up a connection. And so if I want to be able to make a connection and add a new one, we've actually built a wizard for you to that will say I want to make a connection to a to a database. Let's say Redshift. It'll walk walk you through it'll help you set up the set of tables that you want to talk to. It'll say, "Oh, I've got to do some profiling." It'll generate the tests for you and set up the monitors all in the simple wizard process. So, really easy to just get going and get this thing living and running against your data. And another very simple way that we've tried to mature the product to make it fast value so you can start using it and and because it's open source, you can try it for free.

00:33:09 Q: the gateway approach, keeping bad data out

Chris Bergh: If if it works for you, great. If it doesn't, move on to the next tool. And so there's another one. Attendee, I believe in the gateway approach today. Bad data doesn't get in. Yeah, I love that approach as well, Attendee. You know, we've done cases where we find bad data, we take it out of production, make ancillary tables and and resolve it. You know, stop. We're a big believer in in tripwires in production, being able to go in and stop the production line to make sure that things happen this and and so I like I like that idea. It's not always possible given customer demands. Sometimes having to get things in production or patch data as best as you can or cut data out. But it it's always contextual with data, right? You know, generally finding problems in data as soon as you can. Sometimes it's setting up data quality monitors and and sometimes it's doing things like observability, pulling your data upon ingest.

00:34:13 Test coverage everywhere, and making testing cheap

Chris Bergh: Also sometimes it works putting tripwires in production having it on gold, silver, medallion, medallion plus everywhere that you touch the data, every form of the data having test coverage and then utilizing some of those tests in deployment. So as you add more data at as you change the code, how do you know you haven't broken anything? And so that's where this data testing as a central resource is so important and why I think a lot of observability vendors have it wrong in that they're trying to make it expensive to test. We're trying to make it as cheap as possible to test. Make it a central resource across all your teams and then therefore you will test better and if you test more your life will be better and you will get more done. And that's really the we've learned that after decades of working with data and that's why we tried to make it as easy as possible for you to try our software and go and so I think I've answered all the questions. So we have a rather short webinar today but give us a call and again give us give us a trial. We'd love to have you use the software. Thank you again for attending today and and that's that's all I have.

Machine-generated transcript, lightly edited: filler words removed, product and speaker names corrected, and audience members anonymised. Chapter times are scaled from the meeting clock onto the recording, which is shorter than the meeting. Speaker attribution is as captured on the call.

Questions from this session

Why is data observability tooling so expensive?

Why data observability costs what it does, and what DataKitchen built instead. Chris Bergh walks through time series anomaly detection as a commodity algorithm, the $170,000-a-year average price to monitor a thousand tables, the venture-capital economics behind that price, and TestGen's pricing. He then demonstrates the new monitors feature detecting a volume drop and a schema change on live tables.

Why is data observability so expensive?

Not because the technology is hard. The anomaly-detection models are commodity open-source libraries, and the code that wraps one is short. Over $500 million of investment capital has gone into the space, and returning 10x on that requires high prices and fast revenue growth. The result is pricing that rises with tables, credits, or usage, so you pay more precisely when you test more.

What does TestGen cost, and what do the monitors add?

TestGen Enterprise is $100 a month per user and $100 a month per database connection, with unlimited tables and no credit calculations or row-count fees — one customer with ten users and three connections pays about $15,000 a year. The open-source version is free for one user and one connection. Monitors add time series anomaly detection on freshness, volume, and schema automatically, plus custom drift tests written as a SQL expression.

How is security and data encryption handled?

TestGen does not move or touch your data. It is a self-hosted product that pushes SQL into your database and executes it there, so values stay where they already live and there is nothing to exfiltrate. It is not a cloud product.

How complex is installation if it's self-hosted, and which databases are supported?

Installing locally takes five to ten minutes using Docker Compose; a production deployment with its own database on Kubernetes takes longer. Chris said DataKitchen supports somewhere around eight to ten databases and adds new ones regularly, and that a wizard walks you from a new connection through table selection, profiling, test generation, and monitor setup.

What support do you get with the open source version?

A Slack channel staffed during reasonable US hours, for the open-source product as well as the enterprise one. The enterprise support model is similar with options for more coverage. Chris framed it as a consequence of being profitable rather than a paid add-on.

Where to go next