Audit
The Data Agent Reliability Audit
A fixed-scope engagement that shows which questions your data agent gets wrong, how often, and whether a change to its instructions made it better overall or only where you looked.
01 · What's in it
Three things you keep
A report for risk and audit
The method, the sample sizes, accuracy and pass^k side by side, and every finding with its confidence interval. Written so a model-risk, AI-governance or internal-audit function can read it without us.
A fix list for the BI team
The measures and wordings that break, the definitions that are ambiguous, and where in the chain each failure starts. Written for a BI lead to act on.
A reusable test suite
Your questions in several wordings, with expected answers stored as queries that recompute from your data. Rerun it after any change.
Where the audit tests a change, you also get a before-and-after by category and by question, with 95% intervals, so you can tell a fix from a shift that merely looked like one.
What the report shows about your agent
Built from what our census of nine industries found: wrong answers repeat, other agents share them, and definitions fix most of them but can break others.
- What asking again would miss. How many of your agent's wrong numbers came back on another run, and how many a check that asks three times would still have let through.
- What your definitions fixed, and what they broke. Your agent run with and without its instructions, compared question by question, because a definition written for one question can break another.
- A conventions review. Amounts stored as negatives, and the exact values your agent has to filter on. Agents guess these, then filter to nothing or subtract a negative twice.
- A second opinion, if you want one. A plain model writing SQL over the same tables, asked the same questions, to show whether a second agent would have caught the errors or made the same ones.
02 · The six steps
Two to three weeks of work, once access is agreed
-
Kick-off call
We agree the business questions that matter (typically fifty to a hundred and fifty), the definitions behind them, and the go-live decision the audit has to inform.
-
Agree access, and where the run happens
Who the agent is measured as, which capacity the run uses, and how much it may consume. Your team grants the access, limits it, and revokes it at the end. See what we need from you, below.
-
Run every question, many times
Each question in three wordings, five to ten times each, on the current configuration and again without its instructions. Every ask is a fresh conversation. If you want the second opinion, a plain model runs the same questions over a copy of the tables that stays in your tenant.
-
Trace the failures
To the agent's rewrite of the question, its choice of source, its choice of measure, or an edge in the data. A person confirms each finding. Automatic stage results are leads, not evidence.
-
Propose changes, rerun, report the difference
Changes to the instructions and the semantic model, rerun on the same questions, and reported by category with confidence intervals.
-
Deliver
The report, the fix list and the test suite. In an engineer-led audit, your engineer walks you through the findings.
03 · What we need from you
Inputs and decisions
- The questions. The business questions the agent has to answer, and the definitions your organisation uses for terms such as revenue, active customer or average price. We help draft them.
- The expected answers. A query, in SQL or DAX, that computes each right answer, or a person who can write it. We recompute it at run time.
- A decision on access. Which identity the agent is measured as. Row-level and column-level security still apply, so answers depend on who is asking: measure it as the person who will use it. We ask for the least access that works and tell you exactly what that is before you grant it, because some routes need more than read access to the workspace. It is revoked when the audit ends.
- A decision on capacity. Evaluation consumes capacity. The agent's model usage is billed to the capacity the agent runs on, and the queries it generates are billed to the engine that runs them (Microsoft Learn), so moving the agent to a separate capacity doesn't move those queries unless its data sources move too. We agree with you where the run happens, and set a ceiling on the number of asks and their rate before we start.
- A decision on what may leave your tenant. AI agents help with the analysis. Before one works with anything from your engagement, we check that your contract allows it to be read there. If it doesn't, we run inside your tenant or do that step without an agent.
- Someone to receive the report. The person in risk, governance or audit who will sign off, so that we write for them.
04 · Who it's for
Teams about to put an agent in front of people who act on its numbers
The audit fits best when most of these are true:
- You run Microsoft Fabric or Databricks, with data already used for reporting.
- A Fabric data agent or a Genie space is in pilot, with a go-live date in the next two quarters.
- A model-risk, AI-governance or internal-audit function has to sign off.
- A wrong number would carry regulatory or financial weight: banking, insurance, superannuation, energy and utilities, and government agencies are examples.
The people in the decision are usually a head of data platform or chief data officer who owns the go or no-go, a BI lead or semantic-model owner who acts on the fix list, and a co-signer in risk or audit who needs evidence they can file.
What it doesn't cover today. The audit is for Fabric data agents and Databricks Genie spaces. Both have run live only on agents and spaces we built: Fabric many times, and Genie across the nine industries of the census. We are building for other platforms, and we say so before taking work on any of them. Power BI Copilot and multi-turn conversations aren't measured. See Open source.
Where this stands. This is a new service. No audit has been delivered yet, and nothing has run in a customer's tenant. The proof page says what has been tested and what hasn't.
Who does the work
AI agents run the questions, trace the failures and draft the report. The rest depends on the tier you pick. Self-reported: your team runs Rekount and reports its own results. Engineer-led: one of our remote engineers, a real person, reviews every finding and runs the meetings. We say that plainly because you should know who answers for what you're given.
The scope is fixed, so you know what you're buying. For cost, talk to us.
Start with a sample run
We run ten of your questions, five times each, on your data agent, and send you a one-page result: how often it was right every time, which wordings it got wrong, and how many of those wrong answers asking again would have missed. You decide whether it's worth going further.
Fabric data agents and Databricks Genie spaces. We agree access with you first, and you can limit it and revoke it.