Benchmark
One agent, 24 questions, 1,080 runs
A Microsoft Fabric data agent answering questions about Australian electricity market data. The results show that the gap between accuracy and dependability is real and measurable. They don't tell you how your agent behaves.
The full write-up is Fabric Data Agent Accuracy Is a Distribution, Not a Number (external site). The post writes single-ask accuracy as pass^1. It is the same number as pass@1 here. A later rerun changed one of its readings: see the aggregation fall.
01 · Limits first
What this benchmark can't tell you
Scope
- 24 questions in four categories, 6 each, written by us. Each has 3 wordings, so 72 wordings in all.
- One dataset: AEMO's published price and demand data for the National Electricity Market.
- One agent: a Microsoft Fabric data agent we built and configured ourselves, in our own tenant. It isn't a customer's agent and it isn't a verdict on Fabric data agents in general.
- A preview SDK: the agent was driven through Microsoft's Fabric data agent SDK, which is a preview. The way it exposes an agent's trace isn't a documented contract.
Reading the numbers
- The original configuration (v0) was asked each wording 10 times, 720 runs. The revised one (v1) was asked 5 times, 360 runs.
- With 6 questions in a category, a category result can move on one question. We show the intervals so that you can see it.
- The ground truth was computed from the data on 27 October 2026. A model or service update could move the results.
- We wrote the questions and the agent's instructions, so this isn't an independent test of either.
02 · Accuracy and dependability
82.4% of single asks were right. The chance a question was right ten times in a row was 38.9%.
pass@1 is the share of runs answered correctly: 82.4%. pass^k is the chance that all k asks of a question are right, averaged over questions. It falls as k grows, to 38.9% at k = 10. A user who asks the same question again and again meets that second number, not the first.
Figure 1 The chance that every ask of a question is right falls as k grows
Original configuration (v0), 720 runs, 72 wordings of 24 questions.
- pass^k, measured at k = 1, 3, 5 and 10
- Lowest and highest of ten single passes at k = 1: 73.6% to 86.1%
View the data as a table
| k | pass^k |
|---|---|
| 1 | 82.4% |
| 3 | 67.8% |
| 5 | 57.5% |
| 10 | 38.9% |
| Lowest single pass | 73.6% |
| Highest single pass | 86.1% |
Source: recorded NEM runs replayed through the harness (version 0.1.0). Lines between the measured points are straight; pass^k was not measured at the k values in between. The 95% interval for pass@1 is 72.1% to 90.4%, resampling questions.
By category
Even lookups, the easiest category at 95.0% on single asks, reached 55.6% at k = 10.
| Category | Runs | pass@1 | pass^10 |
|---|---|---|---|
| Aggregation | 180 | 86.1% | 38.9% |
| Definitional | 180 | 65.6% | 33.3% |
| Lookup | 180 | 95.0% | 55.6% |
| Time | 180 | 82.8% | 27.8% |
03 · One pass is one draw
Ten passes over the same wordings scored 73.6% to 86.1%
A pass asks every wording once. We made ten of them. A single-pass evaluation of this agent could have reported any of those ten scores, and nothing in a single pass would have said which end it was near.
28
wordings right on every run
of 72
40
wordings flaky
right on some runs and not on others, counting declined runs as not right
17
wordings flipping
right on some answered runs and wrong on others, leaving declined runs out
4
wordings never right
in 10 runs
Flaky and flipping are different counts, and neither stands in for the other. A wording right nine times and declined once is flaky but doesn't flip. 42 runs (5.8%) were declined, across 32 wordings. A decline is a safe failure, so it is tracked separately.
A question can fail the same way every time
A consistent wrong answer scores well on repeatability and badly on accuracy, which is why the two are always reported together. These six wordings were right least often in v0.
| Wording | Category | Right | The other runs |
|---|---|---|---|
| How much demand did New South Wales average on Saturdays and Sundays in 2025? | Aggregation | 0 of 10 | 9 wrong, 1 declined |
| What was the average price in SA1 in calendar year 2025? | Definitional | 0 of 10 | 9 simple average (a defensible reading), 1 declined |
| What did South Australian wholesale power cost on average in 2025? | Definitional | 0 of 10 | 10 simple average (a defensible reading) |
| sa average price 2025 | Definitional | 0 of 10 | 9 simple average (a defensible reading), 1 declined |
| What was Queensland's highest demand over the last quarter? | Time | 1 of 10 | 9 wrong |
| nem average price sep 2026 | Definitional | 2 of 10 | 8 simple average (a defensible reading) |
Right label, wrong number
"Average price" has two defensible readings: the simple mean of interval prices, and the demand-weighted price. In 22 v0 runs the answer's text named the demand-weighted definition. In 9 of them the number was another definition's. In v1 it was 7 of 59. The label mismatch metric counts these separately, because a number that sounds right is harder to catch than one that looks wrong.
04 · Changing the agent
What the instruction change moved, and what a rerun showed
v1 adds definitions and example queries to v0's instructions. To compare like with like, both configurations are scored on the first 5 runs of each wording, so v0's overall figure here (81.7%) differs from the 82.4% above, which uses all 10.
Figure 2 Accuracy before and after the change, by category
pass@1 on the first 5 runs of each wording. Categories run from largest gain to largest fall. The axis starts at 50%.
- Before: v0
- After: v1
View the data as a table
| Category | v0 | v1 | Change, points |
|---|---|---|---|
| All questions | 81.7% | 85.0% | +3.3 |
| Definitional | 66.7% | 88.9% | +22.2 |
| Time | 80.0% | 81.1% | +1.1 |
| Lookup | 94.4% | 94.4% | 0.0 |
| Aggregation | 85.6% | 75.6% | −10.0 |
Source: the same recorded runs, v0 and v1, paired by wording. On those runs pass^5 moved from 51.4% to 61.1%, and flipping wordings went from 10 to 15. A wording that was always wrong and is now sometimes right counts as flipping, so that count can rise while accuracy improves.
Definitional questions gained +22.2 points, from 66.7% to 88.9%. Aggregation questions fell by 10.0 points, from 85.6% to 75.6%. Overall, accuracy moved +3.3 points.
How much of that can we trust? Figure 3 shows two 95% intervals for each shift: one from resampling whole questions, which leads because a question's wordings tend to move together, and one from resampling wordings. We treat a shift as clear only when both exclude zero.
Figure 3 Which shifts are clear
Change in pass@1 in percentage points, with 95% intervals. The vertical line is no change.
- Both intervals exclude zero
- At least one includes zero
- Resampling questions
- Resampling wordings
View the data as a table
| Category | Change | Interval, questions | Interval, wordings | Clear? |
|---|---|---|---|---|
| All questions | +3.3 | −4.2 to +12.5 | −3.1 to +10.0 | No |
| Definitional | +22.2 | +4.4 to +46.7 | +8.9 to +38.9 | Yes |
| Time | +1.1 | −10.0 to +11.1 | −11.1 to +12.2 | No |
| Lookup | 0.0 | 0.0 to 0.0 | −4.4 to +4.4 | No |
| Aggregation | −10.0 | −21.1 to 0.0 | −23.3 to +2.2 | No |
Source: paired bootstrap over the same recorded runs. Each category has 6 questions and 18 wordings, so even a clear category result rests on few questions.
The definitional gain is clear: both intervals exclude zero. The aggregation fall is not: the question interval touches zero (−21.1 to 0.0) and the wording interval includes it (−23.3 to +2.2). The overall change of +3.3 points can't be told from no change (questions −4.2 to +12.5).
That's the argument for reporting a change by category, with intervals. A single before-and-after score would have reported a small gain and hidden the large one.
The aggregation fall was mostly chance
On 2 November we asked v1 everything again, unchanged. It scored 84.4% on aggregation, against the 75.6% recorded here and v0's 85.6%. Overall it scored 84.7% against 85.0%, so the headline came back and the category didn't.
The write-up, published before this rerun, reads the fall as an effect of the change. The rerun doesn't support that, and the figures above stay as recorded. The full comparison is on the proof page.
05 · Where the variance enters
Which step first separates right from wrong
For each flipping wording, the harness looks for the earliest stage whose values track right versus wrong among answered runs. These are leads for a person to confirm, not findings: with about ten runs and hundreds of candidate phrases, a phrase can separate outcomes by chance.
| Configuration | Flipping wordings | Decisive stage |
|---|---|---|
| v0 | 17 | rewrite 15, query 2 |
| v1 | 15 | query 8, rewrite 6, source 1 |
Check the work
The grader is deterministic: a number from the answer against a value computed from the data. We also ran an LLM judge over all 1,080 runs as an independent check. It agreed with the grader on 1,079 and marked 1 unclear. A test regrades every recorded run against the original harness and checks each published figure. If a change to the code moved a published figure, the change would be wrong. The proof page tests the tool itself on cases with known answers.
Data: aggregated price and demand data from the Australian Energy Market Operator (AEMO), used with attribution.
Want this on your agent?
We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.
Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.