Rekount Sample run
Menu

Reliability checks for enterprise data agents

Right once is a demo. Right k times is a result.

Rekount asks your data agent each question k times, in several wordings, each in a fresh conversation. It reports how often the answer is right every time.

Request a free sample run Read the nine-industry census

k: how many times you ask the same question

pass^1: 82.4%. The chance that a single ask of a question is right.

Out of 100 questions like these, about 82 would be right.

View the data as a table
The chance that all k asks of a question are right
kpass^k
182.4%
367.8%
557.5%
1038.9%

One Microsoft Fabric data agent, 24 questions about Australian electricity market data, 720 runs. Each cell is one in a hundred. What this does and doesn't show is on the benchmark page.

Flaky wordingsFlipping answersAmbiguous termsStale expected answersDeclined runsRight label, wrong numberThe agent's rewrite of the question

01 · The problem

Accuracy is one number. Dependability is another.

  1. 72 wordings

    24 questions, three wordings each. Every wording was asked ten times, each in a fresh conversation.

  2. 28 were right every time

    Ten asks, ten right answers. These are the wordings that a one-off evaluation and a repeat both get right.

  3. 4 were never right

    A question that fails the same way every time is perfectly consistent, so accuracy and pass^k have to be read together.

  4. 40 were flaky

    Right on some runs and not on others. This is what a single evaluation pass cannot show.

  5. 17 of those flipped

    Right on some answered runs and wrong on others. The two counts differ because of how declined runs count.

  6. 73.6% to 86.1%

    Ten passes over the same 72 wordings produced scores across that whole range. A one-off evaluation could have printed any of them.

Watch · under six minutes

See it on real runs

A Fabric data agent, a Databricks Genie space and a GPT model writing SQL, asked the same question live. What changes between asks, what the gate blocks, and what an audit gives you.

02 · Nine industries, 18,108 runs

Asking again won't catch it. Neither will another vendor, or a better model.

We asked Microsoft Fabric’s data agent, a Databricks Genie space and a GPT model writing SQL the same 106 business questions from nine industries, many times each. The wrong answers held up to every check except one: an answer computed from the data.

76–87%

of wrong numbers came back on another run of the same question

34%

of answers all three agents agreed on were still wrong

found after seeing the data

12 of 18

shared wrong numbers a stronger model gave too

predicted before it ran

17 to 38

points of pass^5 from written definitions, on every agent

Asked live: “How many taxi trips were there in March 2025?”

The same question in each product's own interface. Computed from the data, counting only trips with a fare and a distance, the answer is 3,848,642.

Microsoft Fabric data agent: 4,145,243, then 4,145,224

Databricks Genie space: 4,145,224

A GPT model writing SQL over a copy of the same table answered 4,145,224 too. The Fabric data agent's second answer came through its API minutes after the first. Recorded on 3 November 2026, on the census's own agents and tables; account and workspace names are cropped out.

Read the census, what failed and its limits

03 · Changing the agent

Your instructions fixed the question you tested. What else did they move?

We added one line to a live agent's instructions, aimed at a question about "last quarter". That question went from 3 right answers of 15 to 15. A question about "last month", which the line wasn't aimed at, went from 10 to 1. Both sit in the same category, which rose only 4.4 points.

A single before-and-after score would have called that a small win. The gate lists both questions, says which falls are unlikely to be chance, and refuses to score a question whose expected answer has gone out of date, as that one's had.

See the gate catch a bad change on questions nothing was written against

"Last quarter" question

Before 3 of 15
After 15 of 15

"Last month" question

Before 10 of 15
After 1 of 15
Their category
73.3% to 77.8%
Overall
84.7% to 88.1%

04 · What you get

Evidence a risk committee can read and a BI team can act on

Rekount Illustration

Data Agent Reliability Audit

One agent, 24 questions, 720 runs

82.4%

pass@1: single asks right

38.9%

pass^10: right all ten times

  • aggregation 86%
  • definitional 66%
  • lookup 95%
  • time 83%

Method, sample sizes and 95% intervals. Built from the published runs; not a customer report.

Fix list Wordings that broke, from the published runs
  • A5 How much demand did New South Wales average on Saturdays and Sundays in 2025? 0/10 Wrong number
  • D2 What was the average price in SA1 in calendar year 2025? 0/10 Another reading: simple average
  • D2 What did South Australian wholesale power cost on average in 2025? 0/10 Another reading: simple average
  • D2 sa average price 2025 0/10 Another reading: simple average
  • T2 What was Queensland's highest demand over the last quarter? 1/10 Wrong number

Where the answers part ways: the agent's rewrite, the source it picks, the query it writes, or how it words the answer.

The gate Recorded cases, replayed through the real gate

A real regression: the same definitions taken away

Blocked

  • D2 12 of 15 to 0 of 15 Blocked
  • A1 7 of 15 to 12 of 15 Moved

A one-line fix aimed at T2

Pass

  • T2 3 of 15 to 15 of 15 Moved
  • T3 10 of 15 to 1 of 15 Reported, not checked

  • A report for the people who sign off

    Method, sample sizes and confidence intervals, in a form a model-risk, AI-governance or internal-audit function can file. It supports a go or no-go on rollout; it doesn't make the decision for you.

  • A fix list for the BI team

    Which measures and wordings break, and where in the chain the answers part ways: the agent's rewrite of the question, the source it picks, the query it writes, or how it words the answer.

  • A before-and-after for every change

    What an instruction or semantic-model edit changed, by category and by question, with intervals, so you can tell a fix from a shift that merely looked like one.

  • A test suite you keep

    Your questions, in several wordings, with expected answers stored as queries that recompute from your data. You can rerun it after the next change without us.

05 · Why not the vendor's tool?

Try it first. Here is what we add.

Microsoft's Fabric SDK can repeat evaluation questions, and it records each rewrite. If repeats on one agent are all you need, use that.

We add what a risk committee and a BI team ask for next: expected answers recomputed from your data, grading that isn't the agent marking itself, wordings grouped by question, ambiguity checks, before-and-after with intervals, a gate that can block a change, and, if you want one, an engineer who reviews the findings and doesn't sell the agent.

See the comparison, line by line

06 · How an audit runs

Six steps over two to three weeks of work

  1. Kick-off

    Agree the business questions that matter, the definitions behind them, and the decision at stake.

  2. Agree access and where the run happens

    Who the agent is measured as, which capacity it runs on, and how much the run may consume.

  3. Run every question in three wordings, many times

    Each ask is a fresh conversation, so no answer can see another.

  4. Trace the failures

    To the agent's rewrite, its source choice, its measure choice, or an edge in the data.

  5. Propose changes, rerun, report the difference

    With confidence intervals, by category.

  6. Deliver

    The report, the fix list and the test suite. In an engineer-led audit, a real person walks you through them.

What the audit includes and what we need from you

Request a free sample run

We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.

Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.

Request a free sample run

Who does the work

AI agents run the questions and draft the analysis. Then you pick a tier: self-reported, where your team runs Rekount and reports its own results, or engineer-led, where one of our remote engineers, a real person, runs it and reviews every finding before it reaches you. The company isn't certified to anything yet, and it says so on the Trust page.

The method is published, the harness that implements it is coming as open source, and the proof says what it hasn't been tested on. No audit has been delivered yet. You shouldn't have to take our word for a number.