Your VP has two tabs open, and revenue is different on each. Nobody changed anything. Both refreshed this morning. The data team now has a week of forensics ahead and a meeting at 3.
Often neither dashboard is broken. They’re answering slightly different questions under the same name.
Five causes, and how each one shows up
| Cause | What it looks like | How to check |
|---|---|---|
| The same rule written in two places | Numbers match for months, then drift after a definition change | Find every copy of the calculation; compare their SQL |
| Different sources or default filters | A steady gap that never closes | List each dashboard’s source system and filters side by side |
| Data refreshed at different times | Numbers converge by afternoon, disagree in the morning | Compare each source table’s last load time with the dashboard refresh |
| A join that duplicates rows | One dashboard is high by a few percent | Row count before and after the join; uniqueness on the join key |
| Inconsistent category values | Totals agree, but a group or segment splits differently | Distinct values in the grouping column (“NY”, “N.Y.”, “New York”) |
The same rule, written twice
This one starts quietly. A rule gets implemented in a report, copied into a spreadsheet, rebuilt in a second BI tool. When the definition changes, some copies get updated and some don’t. The cost shows up as two reports that disagree and a week spent finding which one holds the real rule.
Two systems, two honest answers
Some metrics live in more than one system. Finance reports revenue from the ERP. CRM reports it from closed-won opportunities. They’re different numbers because they measure different points in the process. The same goes for default filters that live in analyst muscle memory: internal accounts, test users, pilot markets. One dashboard excludes them and the other doesn’t.
The clock
Timing problems masquerade as data errors. A report runs cleanly and pulls yesterday’s transactions against today’s customer dimension. Every job succeeded. Row counts and schema checks passed. The numbers are still wrong for this hour.
The join
When a join matches more than one row on a key that should be unique, the measure repeats. DataKitchen’s illustration in The Seven Deadly Sins of AI-Generated Data Pipelines, a hypothetical rather than a measured case: a revenue number 4 percent high because a join fanned out on duplicate keys. Nothing failed, and the dashboard rendered.
The categories
Three spellings of the same category give each analyst a different workaround, and each workaround becomes its own version of the truth.
A worked check: split the difference layer by layer
Say dashboard A shows $4.10M for October and dashboard B shows $3.95M.
- Compare each dashboard’s filters and source table. If B excludes internal accounts and A doesn’t, that’s a definition gap. Pick one and label it.
- Same source and filters? Compare the source table’s total with each dashboard’s total. If the table says $3.95M, A is adding something.
- Check row counts on A’s path before and after each join. A jump means a fan-out. Test the join key for uniqueness.
- Counts match but a region splits differently? Check the distinct values in the grouping column.
- Everything matches by 2pm? It’s refresh timing. Compare load times with refresh times.
Make it stop happening
Specify your contested metrics once. DataKitchen’s context framework puts it at three to five metrics first: revenue, active users, churn, whatever leadership asks about most. Name the authoritative source for each and record when a definition changes.
Then test it. An Aggregate Balance test compares totals by group against a reference, such as the source or the prior delivery, so a SKU moving between product groups or a fanned-out join shows up on the load where it starts. A Unique Values test on every join key catches the fan-out before the total moves.