On-Demand Webinar · 39 min

The FITT Way to Data Products: A New Data Architecture

FITT — functional, idempotent, tested, two-stage — is the architecture DataKitchen has delivered customer work on for over a decade. What each word buys you, how to make SQL behave when it is neither functional nor idempotent, and why full builds from raw data cost less than they look like they do.

Presented by Chris Bergh

What you'll learn 6 points
  • A data product is a how, not a what. Chris Bergh's objection to the component-list debate is that a product is defined by iterative delivery of value to a customer, measured in cycle time and deployment success — not by an agreed list of parts.
  • FITT stands for functional, idempotent, tested, and two-stage, and none of the four ideas is new. DataKitchen has run customer delivery on it for over a decade; the point of naming it is to make it easy to copy.
  • Idempotency's real payoff is psychological. If a rerun always produces the same result, recovery is trivial: rerun after a break, rerun to backfill, rerun on yesterday's data to try a new feature.
  • Two-stage means the whole thing is built from immutable raw data to final deliverables in one pass, so anything in between can be thrown away. A sub-terabyte full build takes about two hours and ten or twelve dollars, which is money well spent.
  • SQL is neither functional nor idempotent, so the unit of work is a FITT chunk: create new then replace, cleanup at both ends, tests always inside. A build is just a set of chunks run together, and you can iterate on one chunk without rebuilding the pipeline.
  • There is no dev and no prod, only parameters — git branch, data version, credentials, infrastructure, historic test results. A canary, an A/B production environment, or running a customer's change in parallel for two weeks is a parameter change.

Prefer to read it? The written version is in Webinar: The FITT Way To Data Products: A New Data Architecture For A Product-Centric World.

Slides

56 slides

Transcript

Chris Bergh

Show chapters and dialogue 35 chapters · 6,575 words
  1. 0:00 Welcome and housekeeping
  2. 3:39 Agenda, and what ChatGPT says a data product is
  3. 4:37 A data product is a method, not a component list
  4. 5:39 Project versus product: work brought to the people
  5. 6:41 Flow of value, and how to measure it
  6. 7:34 Background: the value pipeline and the innovation pipeline
  7. 8:24 Fewer production errors, faster cycle time
  8. 9:19 Why publish this architecture now
  9. 10:22 People cost far more than compute
  10. 11:35 The process acting on data matters more than the data
  11. 12:53 Choose productivity over one more feature
  12. 13:54 What FITT stands for, and why none of it is new
  13. 15:00 Where FITT applies, and where it hasn't been tried
  14. 16:03 Functional: clear inputs and outputs, no hidden state
  15. 16:58 Idempotent: run it again and get the same answer
  16. 17:57 Tested: automated tests as living proof of the system
  17. 18:57 Tests are the gift you give your future self
  18. 19:56 Stop on errors, shift left and shift down
  19. 21:07 An aside: open source that writes 80% of the tests
  20. 22:06 Two-stage: build everything from immutable raw data
  21. 23:08 How two-stage looks at a Fortune 100 customer
  22. 24:11 Parameter-driven builds, and how to get FITT
  23. 25:08 Parameterise the pipeline as one orchestrable unit
  24. 26:03 Which parameters a build actually needs
  25. 27:02 There is no dev and no prod, only parameters
  26. 27:58 SQL is neither functional nor idempotent
  27. 29:12 Create new then replace: the FITT chunk pattern
  28. 30:28 Iterating on one chunk instead of the whole pipeline
  29. 31:17 Every step carries tests, so a failure stops the line
  30. 32:02 Full builds from raw: two hours and about ten dollars
  31. 33:06 Write-audit-publish, and why it isn't idempotent by itself
  32. 34:09 Rules of thumb, and the four metrics to manage
  33. 35:19 Don't over-optimise compute costs
  34. 36:24 Why a 23-year-old should be able to rename a column
  35. 37:21 Automation software, free resources, and closing

00:00:00 Welcome and housekeeping

Chris Bergh: Hello everyone. My name is Chris Bergh. I run DataKitchen here in Massachusetts. I'd like to welcome you to today's webinar. so just a few housekeeping things before the meeting. So those of you who are actual real persons and not AI bots, thank you for joining us. I will share the recording and slides after the meeting. put your questions in the chat window during the meeting. I'll answer your questions when I can. or I'll certainly answer them at the end. And we've targeted about 45 minutes. It turns out I have a lot of slides, so it may take a while. and that's our that's what we'll that's what we'll do. So, you'll get the slides at the end, ask some questions. I'll try to answer them when I go. And for those of you in person, thanks for being here in person. Not that there's anything wrong with AI notetakers.

00:03:39 Agenda, and what ChatGPT says a data product is

Chris Bergh: So, we've got a a five-part agenda. So, we're going to talk a little bit about data products and a little bit about DataKitchen, our sort of background and motivation on this topic and some perspectives on data engineering. Then, we're going to talk about this data architecture pattern that's that we've used for over a decade. So, I guess it's it's not new. is called functional, idempotent, tested, and two-stage and then we'll talk about some patterns on how to get FITT. so starting with data products I and you know I I asked ChatGPT what a data product was and it gave this answer that it's a thing it's a reusable trusted well- definfined data set that's designed and built and managed like a product and so I think a lot of the discussions around the concept of a data product is its thingness what is it what's it made of but I I don't believe that I actually think data products are a how and not a what they're a method and not sort of a list of components.

00:04:37 A data product is a method, not a component list

Chris Bergh: I went to a conference and heard two people debate for an hour what the right component list is of data products and I just think that's absurd. the whole point of a product is to to deliver value to your customer in an iterative way. and it's a sort of the products about the outcome not the method. and it really can be anything. And so I think the idea of data products is that it's not a project. So it doesn't have a start and an end. It doesn't have a waterfall. It's focused on the flow of work to the customer. And so this instead of being focused internally on what your steps are, it's focused externally and delivering focusing on the flow of value. And I think that's really important both for the architecture talk that we're going to have today and and and data products and and another aspect of of a project or data projects is that you know you've got tasks and they're sort of assigned to you right people are sort of brought to work.

00:05:39 Project versus product: work brought to the people

Chris Bergh: But on the other hand you're trying to have work brought to the people like your customers are asking for these things. and of course a project tends to be waterfall. A product tends to be iterative and have lots of sort of irons in the fire. And you know one way to think of a data product and if you look at this diagram it's pretty typical. It's got sources on the left process technology and organization and customers on the right. And then there's this idea of like I have some work and I'm going to do a whole bunch of development modeling, testing, building and then I'm going to deliver a sort of monolithic and but I'm following my project things. I've got my documentation and I think data products are much more iterative. They are focused on delivering little bits of value and continually improving. and so they're really never done, right? It's sort of about releasing insight quicker. And I think the other thing about data products that's incredibly important and I'm going to mention this again is that it helps you realize exactly what your customers want.

00:06:41 Flow of value, and how to measure it

Chris Bergh: And so this notion of flow is really important and it's the flow of value. And and what is value in a data and analytic system? Well, it's could be new data. It could be improved data. It could be a new model, a new visualization. And that's all sort of measured as code and configuration, right? And and I think it's also trust. you know, you can have a new data set, but if your customer doesn't trust it and how do you measure this flow? Well, it's cycle time, which is how long it takes to bring something from the brain of your data engineer scientist into production or into your customer's hand. It's how fast you can or what the success rates of deployment. And I think I would add to that the success rates of quality since the data is always changing and then the work done. And so if you read that Mik Kersten book about projects of products, I think that enco encompasses my view of what data products are.

00:07:34 Background: the value pipeline and the innovation pipeline

Chris Bergh: And so now I'm going to switch to a slightly different topic than this idea of data products. I'm going to talk about sort of our company and our background and motivation. So I I've been working in I've been working in technical fields for 35 years and and data fields for 20 both as an individual contributor and manage lots of data engineers and data scientists etc. And I've learned kind of three things. you're going to get crappy data and stuff's going to break. your customers don't know what they want until they see it. and you always have too much to do. And for the last dozen years, my co-founders and employees and I have really focused on trying to fix this problem called DataOps. And we've written books on it and talked on it sort of through the roof. and you know DataOps focuses on this the idea that there's sort of two processes that you have to focus on. One is your assembly line.

00:08:24 Fewer production errors, faster cycle time

Chris Bergh: Your data product has to be manufactured. That's what we call it the value pipeline. And then the other one is that your data product has to be changed the innovation pipeline. And so, you know, we think that the goals of these things are that when you have an assembly line, you want to decrease production errors. So, in production and then you want to also be able to increase the cycle time and and the success rate of putting things into production. And the result of that is you just get a lot more productivity. So, that sort of thinking in a managerial way to think about DataOps, but also you know, we are a profitable company. were owned by our sort of founders and employees. and so we've been an independent business for about 12 years. And partly what makes us independent is we sell our software. but we also deliver data products following DataOps principles using the FITT architecture with all our customers and we've got these great results.

00:09:19 Why publish this architecture now

Chris Bergh: And so to me, one of the reasons I'm talking about this architecture now is, you know, we've written a lot about DataOps. we've written a lot about data quality. we have open source but we haven't talked too much about this architecture pattern and and so one of the things I constantly wonder is why it's not common for people to follow these principles and perhaps the FITT data architecture is a way that can make it easy so that's why we're talking about it so lastly what are given all these perspectives or our background what's our perspective on data engineering so the first thing is that data in. So if you look at the entire cost of delivering value to your customer, you add up all your costs over a year for a data engineering team or this I think applies also to data engineering team that have analysts any any sort of data and analytics team. The cost of the people is much greater than the cost of the stuff the people work on. And lately that's shifted to you know more towards compute than storage.

00:10:22 People cost far more than compute

Chris Bergh: At other times it would have been more storage than compute given there's a lot of Snowflake and Databricks customers out there. But I think this is really true and in in my experience this ratio is off. It's often like 80 or 90% people and and less compute. But you know Snowflake's getting expensive. but unfortunately if you look at that the amount of time that people spend the amount of is really unproductive and that means they're doing work that they don't need to be done they're have they're fixing and delivering errors they're firefighting they're doing wasteful tasks and this is backed up by Gartner other analysts research that we've done and and this sort of I think as a consequence makes I think everybody who works in data sort of knows that we're doing a lot of stuff that really doesn't matter. and so the second idea is that from our background is that maximizing flow, trying to focus on the productivity of the people, trying to make it easy for them to do work, lowering their cognitive overhead, making things really apparent and simple is a very important principle.

00:11:35 The process acting on data matters more than the data

Chris Bergh: And then every data process whether it looks like this one is complicated and errors can happen everywhere like they could be in the raw data they could be in the integrated data they could show up that that could all be perfect and your Tableau server could fall over or your export could could break all these things could happen and then the last idea or the second to last idea is that the processes acting on data. So this is things like the code, the transformation process, the predictions, the reporting are actually much more important than the data. And so this is a very almost philosophical argument, but it but it's sort of bounded in in the sense that like we really work in a system as a team and that making that system better, the tools, the techniques actually drives productivity. And so in some ways the process is much more important than the data which may be upsetting to a lot of data people but it's kind of bounded in this sense that you know it comes there's a a person who changed my management and my leadership about I read about 15 16 years ago Deming who talked about that it's when you have a problem 98% of the time it's the system and not the person and that system Elon Musk calls the machine that makes the machine.

00:12:53 Choose productivity over one more feature

Chris Bergh: It's a factory. and there's a quote from Satya Nadella, if you've got a choice between a developer working on a feature or working on productivity, always choose productivity. And so trying to emphasize the productivity of your of your team. And that productivity I have a very contrarian view. It's not the amount of time there since there's so much waste in the system. So much time here with errors and operational. This is the stuff that you have to focus on. using your LLM to create new code again isn't going to get you the lift of focusing on things like errors and operational tasks. And so in some ways this idea of a FITT architecture is in response to that our view of data products, our background and our perspective on data engineering. And so let me stop for stop there. And that's all sort of a you know a 12minute introduction to how we think about the world. and so now we're going to get into what is the FITT architecture.

00:13:54 What FITT stands for, and why none of it is new

Chris Bergh: And so the FITT acronym stands for functional, idempotent, tested, and two-stage. And we're going to talk in this presentation about what what those mean. and so the first thing FITT is is not new. It's in some ways functional programming as opposed to object-oriented programming is been around for decades. idempotency has been around forever, right? It's it's just a fancy name to you do it again, you get the same thing. and and testing obviously is been around and two-stage data architecture. So this is nothing new. it's just taking some of these ideas and mapping them onto data and analytic systems. In fact, mostly everything in data and analytics isn't isn't new. It's showed up somewhere else. You know, you can sort of pick your pick your favorite buzzword. It's got a heritage in in in manufacturing or or software engineering these days. And and FITT is really designed to maximize flow and minimize errors. And and for flow, you could use the term productivity.

00:15:00 Where FITT applies, and where it hasn't been tried

Chris Bergh: You could talk about what it specifically means is the ideas that originate in your team's head getting into the hands of your customer. That's really the flow that we're talking about. and so how fast and quick and then minimizing errors both in deployment and running in production with with crappy data. And so we've used this is where we've used it. I I'm it could work in other cases, but this is our experience. So, I'm I'm sort of caveating what we talk about. So, your data size isn't too massive, like under a 100 terabytes-ish with kind of updates that are smaller than that. so, I haven't worked on petascale databases. I would imagine this works, but haven't tried it. So, your data updates and refreshes kind of happen every hour or every day. it's not it's sort of not streaming and and you're doing kind of serious data integration work. There's a lot of work that has to be done in the data to make it useful for your customers.

00:16:03 Functional: clear inputs and outputs, no hidden state

Chris Bergh: You know, you could build schemas, joins, one big table, predictive models, all that stuff. And then it really shines when your business stakeholders, when your customers, they keep asking for changes because of course they're always going to do that. And of course, your customers really want to trust the data. And this has just come out of our history of of who we've, you know, the kind of customers that that we've wanted we've we've got in our consulting practice. So I'm going to go through now sort of the the the four parts to it. And so functions, everybody knows what a function is. You took algebra when you were a kid and you talked about the function machine. You got some stuff that goes in, you get some stuff that comes out, right? And so the idea here is there's no hidden state or no mysterious dependencies in it. They're they're kind of pure. Stuff comes in, stuff comes out, but there's nothing happens way over somewhere else. and it's sort of clearly definable inputs and outputs.

00:16:58 Idempotent: run it again and get the same answer

Chris Bergh: And then second, idempotency. Think of it as running a data pipeline multiple times where the same input will always produce the same output. And so that's sort of hard, right? If you've got a pipeline that adds two to an integer column, well, you run it again, you're going to you're going to add two again, and so it's not idempotent. It's always it's not going to produce the same output. and that's the point of of a database in some ways. The I but when you run things that are idempotent, your recovery becomes trivial, right? If the pipeline broke, well, you can just rerun it. If you get weird results, you can rerun it. If you want to try a new feature, just rerun it perhaps on yesterday's data. You need to backfill, well, just rerun it. And so that psychological impact of being able to rerun things is really important from a development standpoint and a recovery standpoint. And the idea of building a little, testing a little, learned a lot that I learned from my branch chief at NASA in 1990 working in aviation fits here.

00:17:57 Tested: automated tests as living proof of the system

Chris Bergh: It iterating on in individual parts during development without having to rebuild entire pipelines over and over again is just a much better way to work. I think because that way you're kind of have you have you can build a little make a test build a little make a test build a little make a test and pretty soon you you you're you've got something pretty rock solid. And and that goes to the other part the the third part of the FITT is tested. you know for years we've been decorating you know the last 15 years I've been in data you know I've written and my teams have written hundreds of thousands of data quality tests and our customers love it right because for instance seven eight years ago we had one of our customers say he loved this he said our 10,000 tests didn't catch it. Are you sure you found a problem? and it's a, you know, it's a great way to both advertise that you know what you're doing and kind of put it put the heels back on your business customers when they they say this looks weird.

00:18:57 Tests are the gift you give your future self

Chris Bergh: And you know, perhaps it is weird and and perhaps the test you need a 10,001st test to catch it, but a lot of times it just proves how good your system is. And so the idea of FITT is that tests and are first class citizens and we mean all automated tests, no manual testing. And think of test as the living proof of what the system should do. and you are testing the data and you're using tests to monitor the tools and the code that are acting upon data. And another one of my favorite phrases that came from Gil, my co-founder, is tests are the gift you give to the future self. Or if you write a bunch of tests, you don't have to own the stuff that you work on for the rest of your career. You can hand it off. And then lastly, another metaphor from an assembly line, the idea of an andon cord, which comes from the Toyota production system in Deming, is that anyone on assembly line has the ability to stop it by saying, "This is wrong.

00:19:56 Stop on errors, shift left and shift down

Chris Bergh: I'm gonna pull the cord and stop the assembly line. And that's a good thing, right? You don't want to build something where you know it's going to have an error. And so stop on errors in in your production workflow is really I think an important aspect. And then what this and coverage of tests in the FITT architecture is really important and this idea that you should shift left in production i.e. When you get bad data, you should find that that data is bad as soon as you possibly can when it's loaded or the first transformation. Don't let it get to the very end. and then the same thing too, if you can find problems in the new code that you've written on your desktop or in a CI/CD process before it gets to production. Both of these have the 1:10:100 scaling issue. If I find problems earlier in production, it's takes much less time to deal with your customers after they're yelling at you because the data is wrong. And likewise, if I change some code, and I find that really quickly, I'm much faster than fixing it than having that new code get into production and then having your customers and and so I like the term shift left, shift down because of this T diagram.

00:21:07 An aside: open source that writes 80% of the tests

Chris Bergh: And so these terms are sometimes overloaded, but this is the way I think of them. and so here's an ad. We have some open source software that that since we've written so many tests over our career, we found all the patterns and we put them in a tool that writes 80% of the tests for you. Learns your data and creates all the tests and it does things like profiling data set screening. It uses some AI to generate data quality validation tests. does anomaly detection and and dashboards. And the point here is that the 20% of the tests that this doesn't do are actually the most valuable thing that you should write. They're they're unique to your c company, unique to the data set that you're using. And nobody writes enough data quality tests and so our goal is just to give you a tool that writes 80% of them and then has a nice UI so you can manage them. It's totally open source, comes with a UI, full-featured, and the enterprise version is is pretty inexpensive, $100 per user per month.

00:22:06 Two-stage: build everything from immutable raw data

Chris Bergh: So, now let's go to the last the second the last thing or the last T- stage. So, what this means is that the whole architecture is set I start from raw data and so for us raw data is immutable. that's not changing to a final data and that could be processed data or the things that are made from process data and it's sort of think of it as done in one step. Now there could be lots of substeps, there could be other things that happen but anything in between you can throw it out and so start from raw data build the whole thing at once. Now, there's some, you know, the question is, when I've talked to this before, the question is, aren't you wasting database cycles? How do you deal with small updates? and so we we'll get to those. but this is the main idea. And so, in practice with one of our customers, sort of a Fortune 100 company, it works this way. So, we've got kind of our software in the middle here.

00:23:08 How two-stage looks at a Fortune 100 customer

Chris Bergh: But if you think about it, this first step, there's a data lake that's kind of shared between group one and and the the corporate entity. And then the the two-stage does it basically builds a database that's got three layers. Raw layers, a star schema, and then sort of a bunch of extract tables that may back up to a Tableau report or may go out to a sales system or whatever. And so this is all sort of built at once in one place. And there's sort of multiple technologies here, some SQL, some Python, etc. And and again, the data is sort of immutable here in the data lake, but it's addressable by time-based parameters. So sometimes the data is a replacement of data that already exists, or sometimes it's just a an update to it. It's here's here's a week's worth of data. You've got 52 weeks in the database. I've gotten the 53rd week or the 54th week. And so we address it in it by time-based parameters. And so that's just listing sort of time and and location in an S3 bucket.

00:24:11 Parameter-driven builds, and how to get FITT

Chris Bergh: And then what's really good about this is we do parameter-driven builds. So it's super easy to take this whole immutable data set and run today's build, run last week's build, run a development build. And so this idea of parameterization of the process and the two-stage it this is where sort of making it clean is really helpful. And so and so I I'll talk a little bit about that. So my next thing is sort of what I did is I went through what what is functional, idempotent, tested, and two-stage. Now we're going to kind of talk about sort of FITT practices or how to get FITT. sorry for the joke here. So the first thing and how to get FITT is parameterization of your pipelines. Right? In our software, our automation software, we get this idea of kitchens and recipes and ingredients. There's another company I like called Bauplan, which actually uses Iceberg and a Python DSL to be able to do that. Again, this is a technique.

00:25:08 Parameterise the pipeline as one orchestrable unit

Chris Bergh: It's not sort of super proprietary rocket science, right? And and think of it this way. In development, I want to use yesterday's data in today's code, right? Where I'm have these sort of ephemeral hardware environments that I'm spinning up and spinning down. Production is today's data and yesterday's code. And so it's a little more complicated than that, right? You want to be able to sort of that the thing that builds the second step, right? The does all the work. You want to parameterize the whole thing. And so and create this sort of repeatable process as one orchestrable unit. And so the first thing is like well what code am I going to run? Well let's let's go here. The first thing you have to think about is what am I going to do? What am I going to do it on? And what kind of sort of metadata I'm going to produce or need. So that could be things like the git branch or what data version.

00:26:03 Which parameters a build actually needs

Chris Bergh: So what data am I going to point to? And that's a lot of parameters and and dates. the other parts are your credentials. You know, what infrastructure you're going to run on, what database, what tool chain you're going to use. and then you might want to keep track of your or point to historic test results because comparisons between the last time you delivered something and the current time are actually really important. And then maybe operational metadata like notifications or be able to update a data catalog. So being able to parameterize all this in one step really gives you this cool set of opportunities because making a development environment just a couple of parameter changes running something in parallel running a canary version running one in parallel again just a couple of parameter changes and so this is helpful in a lot of different ways like I have a new data set I want to run it in parallel with my customer I have a customer who's got some complicated change, they want to look at it for a week or two.

00:27:02 There is no dev and no prod, only parameters

Chris Bergh: It it gives you this nice ability to say yes to customer requests. and with the ability to have the this all done in a cloud or in a in a Kubernetes type environment, it makes it really great to say yes to a customer and and then not give the risk of things breaking. And so that's also sort of based on this idea of of having multiple environments. And so in the FITT architecture there really is no environment. There is no dev no prod. There's just a bunch of parameters that you've run in the environment. And so the environment could be a dev environment, could be a prod environment, could be a AB comparison production environment, could be whatever you want, right? Canaries are always available. And this it's gotten a lot easier. you know, seven, eight years ago, we used to do we and you still can do with things like Redshift, a backup and a restore of a database. with Snowflake, it's zero copy clones.

00:27:58 SQL is neither functional nor idempotent

Chris Bergh: You know, things like Kubernetes, the cloud, it makes it easier. The idea of infrastructure as code, sort of script driven, stateless environment creation, it's it's really important to do that. the the challenge here is that SQL itself and most of us do our work in SQL, not in Python. but like if you think of a chunk of of normal everyday SQL, you know, I'm doing an update statement or an alter table, it has side effects, right? I change the table, I update the the table, maybe I install a new SQL component in the database. you know, if my inputs are data, a SQL command relevant tables, my outputs are change tables, updated tables, and SQL components. And this is the way databases work. It's nothing wrong with that, right? but how do you get FITT techniques when SQL is so not functional and so not idempotent? So the way that we think of it is that needs an idea of a functional idempotent chunk for lack of a better word and think of it as a unit of transformation work and that work can be executed repeatedly with the same results regardless of external state and and so there's a set of patterns we follow.

00:29:12 Create new then replace: the FITT chunk pattern

Chris Bergh: Think of it as like create new then replace. And of course in our chunks — they probably should be called FITT chunks. Tests are always included. And what is a build process? It's just a way to run a whole bunch of FITT chunks together. Build the entire database and deliverables from immutable raw data. So what does that look like? So think of this as a a pattern that we follow. So when we first get the data, we clean up, we set up, maybe we'll take a database table and make a copy of it. or we'll run a delete, we'll copy or we'll delete what's there and bring and reload it from reload it again. That's another way to to do it. and tests and we'll do our SQL transformations in the in the window and in the middle and then we'll kind of do a bunch of cleanups. And so it's these sort of bags on the side of SQL that help us maintain the sort of functional and idempotency of every SQL chunk that we work in. And so this is a practice that we have and it our software helps it a little but there's also some sort of tribal knowledge that we communicate to our to our employees to make sure that this happens.

00:30:28 Iterating on one chunk instead of the whole pipeline

Chris Bergh: And so the real benefit here of this is that I can iterate on a chunk over and over and over again. I can, you know, build a little, test a little, learn as a developer quickly, quickly, quickly. I'm writing 20 lines of SQL. I write a new test. I run the whole thing again. And of course, everything works in that chunk. And so kind of conceptually, I I'd like you to think of these chunks as sort of part of a data pipeline. And so if you look at a typical data pipeline, I I set up an environment. I load some data. I test that data. Maybe I build some dimensions in a fact table. I test or publish. And you know, however the way you want to do it, data vault, it doesn't quite matter here. in this example, we're just doing a Kimball schema. and so I've got different code using the Python. Maybe I've got some Python that's loading the data.

00:31:17 Every step carries tests, so a failure stops the line

Chris Bergh: I've got some SQL. I've got some ELT. I'm doing some tests in SQL. I've got a bunch of SQL that build my dimensions, a bunch of SQL that builds my fact tables, some tests, and then I got some Python code that publishes it. And of course, we're a huge believer in tests. So, every step should have tests. And why again? Because if something goes wrong, why do why do everything else? If you can't load the data or you don't have enough data, well, don't do everything the rest and get it to your customer. And so, that's the sort of andon cord or stop the production. And so we kind of think of these five chunks kind of fitting into the pipeline like each piece is a chunk. I can iterate on loading data and so I can I don't have to affect anything else. If I have a new data set I sort of build a new FITT chunk and then I can work that way through the development pipeline.

00:32:02 Full builds from raw: two hours and about ten dollars

Chris Bergh: So this is a concept that of of how we think about doing functional idempotent work in a SQL database that's basically not doesn't support functional and there's some research work that talks about making databases more functional but it it's you can do it it just takes a little bit of a little bit of behavior change and and thinking and so there's a pattern that's become popular and and so this is this is a full Right. I want to do a full build of a database from raw. And so that's great and I think you should do it. I think sub-terabyte databases a full build can take two hours and cost 10 bucks or 12 bucks. I think that's money well spent. and so for instance, when we do work in our consulting, if someone modifies a FITT chunk, we always have them do a full build to before they actually check it in to make sure that it all runs. And again, this is yesterday's data with today's code. and again, I can hear you thinking and talking about, well, isn't that expensive in database costs?

00:33:06 Write-audit-publish, and why it isn't idempotent by itself

Chris Bergh: And I'll get to that at the very end. a pattern of of working in of doing updates, tested updates that's gaining popularity is called the write audit publish pattern. And it's exactly as I said, you're going to write some data to somewhere else to a staging table or an audit table. You're going to test it, then you're going to publish it. It's a great pattern for cases where I've got I I'm overlaying data. I've got an existing database and I'm adding a couple of hundred rows of new data. you know, I'm updating my Salesforce stuff and so that process can do that, but if you think about it, it's not necessarily functional and idempotent. So, basically, I'm saying make these the work that you do a funk in fact at even after you publish, I'd run the tests again just to make sure the publish happened. So, but you know, WAP is great. It's a great pattern, but it's not inherently functional or idempotent. So, you know, apply the FITT chunk pattern to that and, kind of going here.

00:34:09 Rules of thumb, and the four metrics to manage

Chris Bergh: Some rules of thumb on how to do FITT. So, always be able to build completely from raw. build quality throughout. Have tests at every step of the process. And, and what is done mean for everyone? Done is it's in production. And you know just a when you do have errors love them. If you have an error in production that you didn't catch say great I'm sorry we had 800,000 tests that we ran. We're going to have to add a new one to to catch that case. And from an ownership standpoint own the end toend pipeline enco encompassing both development and production like own everything. and test coverage is very important. And from a a manager step point standpoint, I think you want to drive two metrics down, errors and cycle time and two metrics up, productivity and customer satisfaction. And so two conclusion slides. So we're going to focus. So your biggest cost is people. So if you want to save money or get more productive with the same amount of money, really that's this area.

00:35:19 Don't over-optimise compute costs

Chris Bergh: The people area is your biggest lever. And so I'm saying, well, this extra cost and doing some more full builds that you normally wouldn't do is is going to save so much time on your people that it's going to end up being a benefit. And so I think the other way to think about it is don't overly optimize on compute costs. Yeah, there's opportunities to save, but like doing some more full builds in a FITT architecture is just going to save you so much errors and so much time on your people. And and and lastly, I think just to conclude this our data your data architecture should focus on maximizing data engineer flow and minimizing errors. I think what I've seen with things like Medallion Architecture, they're all about honestly maximizing data vendor data vendor revenues. and while FITT isn't magic, it represents the closest approach we've discovered to making a data engineering kind of boring. Like it's predictable, it's reliable, it's stressfree. you know, and heroism is bad.

00:36:24 Why a 23-year-old should be able to rename a column

Chris Bergh: You don't have to be a hero hero to make a minor fix. You should be able to hire a 23-year-old right out of school. have them be able to change the name of a table or change the name of a column in your database and have them be able to get in production. They should be able to and this this is why this architecture is really good. They change a column. If you've you're building two-stage, so you have the complete build they make a change, they do a complete build, they're going to see the effect of that change in production. they're going to be able to say, "I changed a column name. Oh, wow. That messed up." how this field was calculated that broke this report and they can see that and I think that's a really good thing for engineers to experience and that's you know if you can have your 23-year-old make small changes on your development team in production and not have to talk to the smartest person and have it that's really a te a thing for teams to optimize and the FITT really helps with that.

00:37:21 Automation software, free resources, and closing

Chris Bergh: And then yeah, senior engineers kind of you don't have to like keep the whole system in your head, right? They can focus on high-value coding and and not have to like firefight or check everyone's simple, you know, they can just say, did you run all the tests? and we have another last add today. We we do have some software that we've been using to support this called our automation software. It's kind of built to run the FITT data architecture. It follows data data data engineers and it basically does SQL-based ELT. It handles environment creation, deployment automation, these sort of two-step pipelines and and testing. and so there's a a lot of people who've written about these ideas already around functional programming around idempotency and data around te we've written some bunch on testing and test coverage. and then we've got two articles about FITT that I'd like to reference. And, we've got a lot of free resources, including open source software, including DataOps, certifications and training, testing and certifications and trainings available for you. So, that's the, presentation today. I'll leave these resources up. like I said, I will share I'll share the, the slides and the recording after the meeting. what do what do people think? Any comments or questions I can answer? All right. Well, thank you folks. Thanks for taking the time to show. We got we got through fast today. give it a try. I think this architecture really focusing on flow, focusing on engineering productivity is is what we is what we should do. and this is a very simple way to do things.

Machine-generated transcript, lightly edited: filler words removed and product and speaker names corrected. Speaker attribution is as captured on the call; chapter times are scaled from the meeting clock onto the recording, which is shorter than the meeting.

Questions from this session

What does FITT stand for?

Functional, idempotent, tested, and two-stage. Functional means clearly defined inputs and outputs with no hidden state or mysterious dependencies. Idempotent means running the pipeline again on the same input always produces the same output. Tested means automated tests are first-class citizens — no manual testing — covering both the data and the tools acting on it. Two-stage means building from immutable raw data to the final deliverables in one pass.

Aren't full builds a waste of database cycles?

Chris Bergh raises this objection himself, twice. His answer is that people are 80 to 90% of the cost of delivering value, compute is the smaller share, and a sub-terabyte full build runs about two hours for ten or twelve dollars. The extra compute from a few more full builds saves so much engineer time that it comes out ahead, so don't over-optimize compute costs.

How do you deal with small updates rather than full rebuilds?

The pattern he points at is write-audit-publish: write the new rows to a staging or audit table, test them, then publish. It suits overlaying a couple of hundred rows onto an existing database. But he is explicit that write-audit-publish is not inherently functional or idempotent, so wrap it in the FITT chunk pattern — and run the tests again after publishing to confirm the publish actually happened.

How can SQL be functional and idempotent when update and alter statements have side effects?

It can't, by itself — an update or an alter table changes state, which is what a database is for. The workaround is the FITT chunk: a unit of transformation work that can be executed repeatedly with the same result regardless of external state. In practice that means a clean-up and set-up step at the front (copy or delete and reload), the SQL transformations in the middle, tests included, and cleanup at the end.

Where does FITT apply, and where has it not been tried?

The stated envelope: data under roughly 100 terabytes with updates smaller than that, refreshes hourly or daily rather than streaming, serious data integration work, stakeholders who keep asking for changes, and customers who need to trust the data. Chris Bergh has not worked on petascale databases and says he would imagine it works there but hasn't tried it.

Why does this architecture matter more than optimising the data itself?

Because the processes acting on data — the transformation code, the predictions, the reporting — are where the productivity is won or lost. He cites Deming's point that 98% of the time a problem is the system rather than the person, and Satya Nadella's rule that given a choice between a developer working on a feature and working on productivity, choose productivity.

Where to go next