Method
How we measure whether a data agent is dependable
Ask every question many times, in several wordings, each time in a fresh conversation. Grade the answers against values computed from the data. Report accuracy and dependability together.
01 · Why one pass isn't enough
An answer is a draw, not a fact
A data agent rewrites your question, picks a source, writes a query and words the result. Each step can come out differently the next time. On the NEM runs, the median wording produced 10 distinct rewrites of the question in 10 asks.
That is why a headline score moves. The same 72 wordings, scored one pass at a time, ranged from 73.6% to 86.1%. Snowflake documents the same behaviour for its own agents: its page on Cortex Agent evaluations has a section on score variance across runs, which says that repeated runs of the same agent can take different tool paths and produce different scores.
Accuracy alone misleads the other way, too. A question that fails the same way every time is perfectly consistent, so repeatability looks fine while the answer is wrong. We always report accuracy and pass^k side by side.
02 · The procedure
Six rules the numbers depend on
-
Every ask is a fresh conversation
No answer can see another. Reusing a conversation lets earlier answers shape later ones, which invalidates pass^k.
-
Several wordings of each question
People ask the same thing in different ways. Wordings are reviewed by a person before they are used; a model may propose them, but they stay marked unreviewed until someone has checked them.
-
Expected answers are recomputed from the data
Ground truth is stored as SQL or DAX and run when the suite runs, so the tests survive a data refresh.
-
Grading is deterministic
A number extracted from the answer is compared with the computed value, with tolerances. An answer that matches a defensible alternative reading is reported apart from a plain wrong one. The grader passes an answer if the right number appears anywhere in it, and reports count how often that wasn't the headline figure.
-
Headline numbers use a fixed window
Every wording is asked at least a set number of times, and headline figures score only that many runs per wording. Extra runs for flaky wordings add detail without biasing pass^k upwards, which stopping early would.
-
An expected answer can go out of date
"Last month" means something different next month, and recomputing a pinned answer gives the same number. A stale expected answer looks exactly like a wrong agent, so a question can say the last day its answer is right. After that day it isn't scored, and how it moved is still reported.
We also trace each run: the agent's rewrite of the question, the source it chose, the query it wrote and the answer. That is what lets a report say where right and wrong answers part ways.
03 · The vendor's own tool
Microsoft's SDK can repeat questions too
The Fabric data agent SDK's evaluate_data_agent takes a num_query_repeats setting, 1 by
default, and its steps table records each rewrite. We've used it. If repeats on one agent are all you need, start
there. It's a preview, and Microsoft's page on it doesn't mention repeats,
so read the SDK's own documentation for the setting. As read in SDK version 0.1.34a0, last checked 3 November 2026:
| Microsoft's evaluate_data_agent | This method | |
|---|---|---|
| Repeating a question | Yes, with a setting | Yes, each in a fresh conversation, with adaptive stopping that keeps pass^k unbiased |
| Expected answers | Text in a table, which a helper can fill once from a one-row SQL or KQL query | A SQL or DAX query, rerun on your data every time the suite runs |
| Who grades | The agent, as a second message in the same conversation | A deterministic comparison, independent of the agent. A judge is optional and scored apart |
| Several wordings | List them as separate questions | Grouped under one question and reviewed by a person |
| pass^k and flip counts | Work them out from its output | Reported together with accuracy, with intervals |
| Comparing two configurations | Not in this function | By category and by question, with intervals, and a gate that can block |
| Ambiguous terms | Not in this function | Reports which definition a number matched and whether its label agrees |
| Out-of-date expected answers | Not in this function | Refuses to score them, and says so |
| Would a second agent catch it? | Not in this function | Compares two or more agents asked the same suite, and lists the wrong answers they share |
| Who is checking | The agent's vendor | Not the vendor: your own team, or one of our engineers, who reviews every finding |
Its grading holds up. In our census, given expected answers we had computed from the data, its verdicts matched ours on 107 of 108 asks, and the one difference was our own grader misreading a list. The gap is upstream: where the expected answer comes from, whether it is recomputed, and what a change is allowed to do to it.
04 · The metrics
Every report carries these
- pass@1
-
The share of runs answered correctly. It is what one evaluation pass measures.
Only a run graded correct counts. A declined run and a run that returned nothing are not answered runs.
- pass^k
-
The chance that all k asks of a question are right, averaged over questions. It is what a user who asks the same thing repeatedly experiences, and it falls as k grows.
For each wording we use the estimator from the τ-bench paper, the number of ways to pick k correct runs divided by the number of ways to pick any k runs, which is unbiased for "k independent asks are all right". Wordings are averaged within a question, then questions are averaged.
- Flip rate
-
Two counts, always given together, because they are different numbers. Flipping is right on some answered runs and wrong on others, leaving declined runs out. Flaky is right on some runs and not on others, counting declined runs as not right.
On the published original configuration, 40 of 72 wordings were flaky and 17 flipped. A wording right nine times and declined once is flaky but doesn't flip.
- Answer spread
-
How many distinct answers a wording returned. Numbers are rounded to one decimal place first, so formatting alone doesn't count as disagreement. It finds unstable questions even where you have no ground truth.
- Label mismatch
-
A run where the answer's number matches one definition of a term and its text names another. This catches right-sounding wrong numbers. An answer that names several definitions is treated as explaining a contrast and claims none of them.
- Ambiguity exposure
-
The share of questions whose terms have more than one defensible definition. It points to where instructions or measure names need work.
- Variance stage
-
For each flipping wording, the earliest stage whose values track right versus wrong among answered runs: the agent's rewrite of the question, the source it chose, the query result, or how the same result was written up.
We don't report the first stage that differed, because the rewrite differs on nearly every run, mostly harmlessly. With few runs and many candidate phrases, a phrase can separate outcomes by chance, so reports show how many were tested and treat the result as a lead for a person to confirm.
- Decline rate
-
The share of runs where the agent produced no query and said so. A decline is a safe failure and is counted apart from wrong answers.
- What asking again would miss
-
How many wrong numbers another run of the same wording gave again, how many a check that asks three times and shows only unanimous answers would still let through, and how often a unanimous answer was wrong. Replies that say no data was found are counted apart.
On the published original configuration, 71 of 81 wrong numbers repeated, and a three-run agreement check would have let through 57% of them. Across nine industries in our census, 76–87% of each agent's wrong numbers repeated. Asking again can't catch an error the agent makes every time.
Source: The census of nine industries
- Change delta
-
What moved after a configuration change, overall and by category, pairing the two configurations by wording on the same number of runs.
Two 95% bootstrap intervals accompany it: one resampling wordings and one resampling whole questions. Reports lead with the question interval, because a question's wordings tend to move together. In a category of a few questions neither interval is dependable alone, so a shift counts as clear only when both exclude zero.
- Cost per question
-
An estimate of the AI Query capacity units a run uses, from the tokens the platform reports, at the rates Microsoft documents. It is a lower bound. The queries the agent generates are billed separately to the engine that runs them, and model calls inside the agent's tools may not be reported. It is not a statement of what an audit will cost you.
When a second agent has been asked the same suite, recount agreement compares the two:
how often they agreed, how often an agreed answer was wrong, and the questions both got wrong the same way. A plain
model writing SQL over the same tables makes a cheap second agent. In the census, when three agents agreed, they
were still wrong 34% of the time without written definitions and
1.1% with them.
05 · Outcomes and grading
What a run can be
| Outcome | Meaning |
|---|---|
| Correct | The right value appears in the answer, within tolerance. |
| Alternative | The value matches another defensible reading of the question. |
| Wrong | The answer matches nothing, or states no recognisable value. Answers with no value are also counted apart, because some of them are honest. |
| Declined | The agent produced no query and said so. |
| No answer | The ask failed or returned nothing. |
An LLM judge is optional and scored separately. On the 1,080 recorded runs, an LLM judge agreed with the deterministic grader on 1,079 and marked 1 unclear. So in these runs grading isn't the weak link. What matters is that the checking is independent of the agent and repeated.
06 · What the method doesn't do
Limits
- It is only as good as the expected answers. Someone has to write the query that says what right means.
- It measures the agent as one identity. Row-level and column-level security still apply, so answers depend on who is asking.
- Decisive stages and rewrite phrases come from small samples. They are leads to confirm, and no finding rests on them alone.
- The cost figure is an estimate and a lower bound.
- The grader restates a number in the question's unit, such as gigawatts to megawatts, but it doesn't convert currencies or interpret dates, so questions that expect those need a label question.
- Testing consumes the platform's capacity. We set limits and agree them with you before a run.
- A category score can move several points on chance alone, and one question can fall a long way on chance alone. So a fall blocks a change only on proof, which means a smaller real fall can pass, marked not proven, until more runs settle it.
Reproducing the published numbers
A test regrades all 1,080 recorded runs against the original harness and checks every verdict, every stage and each published figure. The runs will ship with the open-source harness, which replays them offline. It isn't public yet. The proof page tests the tool itself, on real runs where the right answer is known.
pass^k follows τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi and Narasimhan, 2024). The NEM results are on the benchmark page.
See it on your agent
We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.
Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.