Proof
If a tool says a change was safe, how do you know the tool is right?
We replayed 2,250 recorded live asks through the real grader and the real gate, on eight cases where we know what actually happened to the agent. The gate gave the right answer on 8 of 8. A test fails if that ever changes.
01 · Known answers
Eight cases where we know what should happen
Nothing here is simulated. Every run is a real answer from a live Microsoft Fabric data agent. Each case compares two recorded configurations with the gate's default settings. The last two use questions nothing was written against: see below.
| What happened | Should say | Said | Questions that moved |
|---|---|---|---|
| Nothing changed: the same configuration asked again five days later | Pass | Pass | T3, left out: 14 of 15 to 10 of 15 |
| Nothing changed, compared the other way round | Pass | Pass | T3, left out: 10 of 15 to 14 of 15 |
| A real improvement: definitions and example queries added | Pass | Pass | A1: 12 of 15 to 7 of 15 (not proven); D2: 0 of 15 to 12 of 15 |
| A real regression: the same definitions taken away | Blocked | Blocked | D2: 12 of 15 to 0 of 15 (blocked); A1: 7 of 15 to 12 of 15 |
| A one-line fix aimed at T2 | Pass | Pass | T2: 3 of 15 to 15 of 15; T3, left out: 10 of 15 to 1 of 15 |
| The same fix, with T3's expected answer out of date | Can't decide | Can't decide | None past the limit |
| Questions nothing was written against: nothing changed | Pass | Pass | None past the limit |
| Questions nothing was written against: a bad line saying every average is demand-weighted | Blocked | Blocked | P7: 13 of 15 to 2 of 15 (blocked); P4: 8 of 15 to 0 of 15 (blocked) |
D2 asks for South Australia's average price in 2025, and A1 for Tasmania's average demand in 2025. T2 asks for Queensland's peak demand last quarter, and T3 for Victoria's negative-price intervals last month. P4 and P7 are among the new questions below.
A fall past the limit that an exact test can't separate from chance is listed as not proven and doesn't block. One question has too few runs to block on size alone: in this comparison, it takes about eight fewer right answers out of fifteen.
02 · Questions nothing was written against
We broke the agent on purpose. The gate caught it.
A tool can look right on the questions it was built around. So on 2 November we wrote 10 new questions about the same data, in three wordings each, and checked every expected answer against the raw market files with code that shares nothing with the tool. Then we installed the tool clean, as a customer would, and asked the live agent each wording five times: once as it stands, once more unchanged, and once after adding one line that sounds reasonable and isn't. The line says every average is demand-weighted.
The gate's answers: pass for the unchanged agent, and blocked for the bad line, naming the two questions it broke as unlikely to be chance. Comparing the two runs of one of them, the tool showed the bad line's rewrites asking for a demand-weighted average, which is the cause.
| As it stands | Unchanged, again | With the bad line | |
|---|---|---|---|
| pass@1 | 82.0% | 86.7% | 66.7% |
| pass^5 | 60.0% | 60.0% | 50.0% |
| South Australia's average demand, March 2026, right of 15 | 8 | 9 | 0 |
| Queensland's weekday average demand, 2025, right of 15 | 6 | 10 | 2 |
| New South Wales' average demand, FY2026, right of 15 | 13 | 12 | 2 |
The published instructions already leak
Before the bad line, the agent's real instructions define average price as demand-weighted. On the Queensland question, which no instruction mentions, 9 of 15 runs returned the demand-weighted average demand instead of the plain one. That is the same leak the published write-up described, found again on questions nobody had tuned for.
It also shows a limit. That question started at 6 right of 15, so even a fall to none can't be told from chance on one question's fifteen runs. The gate caught the bad line through the other two questions and the time category. A change that hurt only an already-weak question would pass, marked not proven.
03 · The same agent, asked again
The overall result came back. One category moved 8.9 points on its own.
Five days after the published runs, we asked the same configuration everything again. Overall accuracy moved 0.3 points. The aggregation category moved far more, with nothing changed.
That corrects a reading of ours. The published write-up took aggregation's fall from 85.6% to 75.6% as an effect of the instruction change. This rerun doesn't support that: the same instructions scored 84.4%. It is also the best argument for repeating a measurement before believing it.
| 28 Oct, published | 2 Nov, same | 2 Nov, one line more | |
|---|---|---|---|
| pass@1 | 85.0% | 84.7% | 88.1% |
| pass^5 | 61.1% | 59.7% | 65.3% |
| Always right | 44 | 43 | 47 |
| Flipping | 15 | 17 | 13 |
| Declined | 16 | 14 | 11 |
| Aggregation | 75.6% | 84.4% | 88.9% |
| Definitional | 88.9% | 86.7% | 91.1% |
| Lookup | 94.4% | 94.4% | 94.4% |
| Time | 81.1% | 73.3% | 77.8% |
| "Last quarter", of 15 | 4 | 3 | 15 |
| "Last month", of 15 | 14 | 10 | 1 |
04 · One line
A fix helped one question and moved another. The average hid it.
The agent often read "last quarter" as the latest period in the data. We added one line saying it means the last complete calendar quarter. That question went from 3 right of 15 to 15 of 15.
A question about "last month", which the line wasn't aimed at, went from 10 of 15 to 1. Both sit in the time category, which rose 4.4 points, so a category score alone would have shown only a small gain.
We're not calling that a break, for a reason worth knowing. That question's expected answer was pinned to September, and by the rerun "last month" meant October, which the data doesn't reach. A stale expected answer looks exactly like a wrong agent. So the tool now refuses to score a question past the date its expected answer stops being right, and it still reports how the question moved. Someone has to decide what "last month" should mean before that score means anything.
05 · False alarms
A gate that blocks an unchanged agent gets switched off
We split one configuration's ten runs into two groups of five, every possible way, with nothing changed between them: 252 comparisons. In 40 of them, some question fell more than 30 points by chance alone. The question check blocked 0. The whole gate also checks each category and overall pass^k, and those checks blocked about one of these comparisons in seven. Bringing that down is the gate's main open problem.
That is why a fall only blocks on proof, by default. Size alone would have stopped an unchanged agent. The cost is real: a smaller fall on one question passes, marked not proven, until more runs settle it.
On real reruns it did better. In our census, three agents were asked nine industries' questions again with nothing changed, and the gate blocked 1 of 27 comparisons. When written definitions were taken away, it blocked 34 of 36, and the two it passed were right to pass.
What this doesn't prove
- It is our own agent. One agent, one dataset, and questions we wrote, including the new ones. No run has happened in a customer's tenant.
- One bad change, chosen by us. The gate caught it. That shows it can, not that it catches every bad change.
- It doesn't tell you how your agent behaves. It shows the method gives the right answer on these runs.
- Microsoft Fabric is the platform tested in depth. Databricks Genie has run on test spaces of ours, across the nine industries of the census. Connectors for other platforms were written from documentation and have never met a live agent.
- Only a signed-in person has been tested as the identity the agent runs under.
- Only fresh conversations. Multi-turn use isn't measured, and Power BI Copilot isn't covered.
- The gate hasn't run in a real pipeline yet. It has run on these files, on our own machine.
Check it yourself
When the harness is published, one command will replay all of this with no cloud account, and exit with an error if any answer is wrong. See Open source.
The method is on the Method page, and the first 1,080 runs on the benchmark page.
See it on your agent
We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.
Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.