Census · 3 November 2026
Asking again won't catch it.
We asked Microsoft Fabric’s data agent, a Databricks Genie space and a GPT model writing SQL the same 106 business questions from nine industries, in three wordings, five times each, with and without written definitions: 18,108 graded runs. The wrong answers repeat, different agents give the same ones, and a stronger model gives them too. An answer computed from your own data catches them.
76–87%
of each agent's wrong numbers came back on another run of the same question
asking again doesn't catch them
34%
of answers all three agents agreed on were still wrong
asking another agent doesn't either
12 of 18
shared wrong numbers a stronger model gave too
and it got none of them right on most runs
17 to 38
points of pass^5 gained from written definitions, on every agent
and definitions are code that can be wrong
The platforms are named so you know what was tested. Figures are pooled or given as ranges across agents, because one day of runs is not a basis for ranking vendors.
01 · Wrong answers repeat
Most wrong numbers come back the same way
On every agent, 76–87% of wrong numbers were given again by another run of the same wording. Asking the question again would have confirmed them, not caught them.
Some tools and teams go further and show an answer only when three asks agree. That check would still have let through 42–59% of wrong answers. When three runs of one agent agreed, the agreed answer was wrong 37–44% of the time.
Agreement is still useful: it separates right from wrong better than chance. But an error the agent makes every time agrees with itself every time.
02 · Found after seeing the data
Different agents make the same mistakes
We compared every combination of one run from each of the three agents. When all three gave the same answer, it was still wrong 34% of the time. On 18 of the 106 questions, every agent's most common answer was the same wrong number.
Each of those is the plain reading of a column, where the business meant something else:
- sales as the sum of a unit-price column, dated by order rather than delivery;
- every school in a county, including the closed ones;
- every taxi trip, including the invalid ones;
- purchase spend including tax and freight;
- a simple average where the business weights by demand.
Given the company's written definitions, the agents reached the expected value on 16 of those 18. We didn't plan this comparison before the runs, so treat it as a finding to replicate, not a measured rate.
03 · Predicted before it ran
A stronger model doesn't fix them
We ran the plain agent again on a larger model, and wrote down the prediction first: it would repeat most of the shared wrong numbers. It did. On the 18 shared questions, its most common answer was the same wrong number on 12, and it was right on most runs of none of them.
It was right more often overall: pass^5 of 52% against 30% for the smaller model. But the errors it kept were more stable: 88% repeated, and 62% would pass a three-run agreement check. A better model is wrong less often, and more consistently.
04 · The biggest fix, and a new risk
Write the definitions down, then test them like code
Giving each agent the business's written definitions raised pass^5 by 17 to 38 points on every agent, with every interval above zero. Wrong answers matching a known alternative reading fell by 79–100%. With definitions, when all three agents agreed, they were wrong 1.1% of the time.
Figure 1 The chance that all five asks of a question are right, without and with written definitions
pass^5 by industry, the mean over the three agents. Industries run from largest gain to smallest.
- Without definitions
- With definitions
View the data as a table
| Industry | Without | With | Change, points |
|---|---|---|---|
| Transport | 29.6% | 98.1% | +68.5 |
| Retail | 11.1% | 62.0% | +50.9 |
| Banking | 34.3% | 71.3% | +37.0 |
| Healthcare | 42.6% | 71.3% | +28.7 |
| Manufacturing | 52.8% | 75.0% | +22.2 |
| Energy | 52.2% | 66.7% | +14.4 |
| Subscriptions | 26.9% | 35.2% | +8.3 |
| Education | 22.2% | 29.6% | +7.4 |
| Insurance | 19.4% | 25.9% | +6.5 |
Source: the census runs of 3 November 2026, five runs per wording. Across all nine industries the mean moved from 32.3% to 59.5%. Seven datasets are public and two are synthetic; we wrote the questions and the definitions.
A definition can read right and compute wrong
- A sign convention. "Claims paid: payments less recoveries." Recoveries were stored as negative amounts. Two vendors' agents both subtracted them, and gave the same wrong total.
- A word taken as a value. One agent filtered on a phrase from the definitions that no column holds, found no rows, and got worse with definitions than without.
- A definition that spread. A July-to-June financial year, defined for questions about financial years, turned a calendar "Q3 2024" into April to June. "Schools are the ones listed as active" stopped schools that have since closed from counting as opened.
What caught them
- Expected answers computed from the data. Agreement between runs, or between agents, can't see an error the agents share.
- The gate's check on single questions. Adding definitions improved every agent overall, and the gate still blocked 4 of 36 comparisons, each for a question the definitions broke.
-
Exact values. 10–15% of repeated wrong numbers were zero: a filter on a guessed
value, such as "home" for
HOME, that matched nothing.
On the one platform where we built it, a semantic model with measures did better than written definitions over three industries: +25.5 points of pass^5, 95% interval +8.8 to +42.2. It led on two industries and trailed on the third.
05 · The gate on nine industries
Few false alarms, and it caught what mattered
We ran the gate on every natural experiment in the census. On an agent asked again with nothing changed, it blocked 1 of 27 comparisons. When definitions were taken away, it blocked 34 of 36, and the two it passed were the right calls: one where definitions had made things worse, one where they had changed nothing.
06 · The vendor's own helper
Its grading is sound. The expected answer is the hard part.
Microsoft's Fabric SDK has an evaluation helper that asks the agent itself whether its answer matches the expected one. Given expected answers we had computed from the data, its verdicts matched ours on 107 of 108 asks, and the one difference was our own grader misreading a list.
So the work that matters is upstream of grading: computing each expected answer from your data, recomputing it every time the suite runs, holding it the same across platforms, and gating changes on it.
07 · What failed, and the limits
Read these before you quote a number
What didn't go as predicted
- We registered three bars before the first ask. Two held on every agent: most wrong numbers repeat, and many pass a three-run check. The third failed: only 26–39% of the repeated wrong numbers matched an alternative reading we had listed in advance.
- Many of the rest turned out to be the plain readings above, shared across agents. That was found after the fact, and it shows how hard it is to list a business's readings ahead of time.
Scope
- "Wrong" means "not the company's definition". Without definitions, an agent can't know them. The risk is that it picked a reading and gave no sign there was a choice.
- One day, one region, one version of each product, on agents and spaces we built in our own tenant. Platforms change their models without notice.
- We wrote the questions and definitions. Seven datasets are public, so models may have seen them. Two are synthetic.
- The agents take definitions in different ways, so the arms aren't identical across platforms. That is one more reason not to rank them.
What to do about it
- Compute expected answers from your own data, and recompute them every time you test.
- Write your definitions down, with sign conventions and exact values, and test the agent with and without them, question by question.
- Gate every change to instructions or semantic models, on single questions as well as overall.
Every Rekount report now says what asking again would miss on your agent, and recount agreement
shows whether a second agent would have caught it. See what the audit includes.
See it on your agent
We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.
Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.