Cloud
The audit, kept running
An audit measures one configuration on one day. An instruction edit, a semantic-model change, a platform update or a data refresh can move the numbers afterwards. Rekount Cloud is meant to keep measuring.
Where it stands
It's new, and no customer has used it. A first version exists and has run on our own machine, on recorded runs, as a container with a Postgres database. It isn't deployed anywhere, and it has no billing. We're describing it so that we can talk to teams about whether it's the right shape. The details below may change after we've heard what teams need.
01 · What it's meant to do
If Microsoft changes the model next month, will you know whether your numbers moved?
- Scheduled re-benchmarks
- Rerun your question set on a schedule, so a change in the agent, the model behind it or the data shows up as a change in the numbers rather than as a surprise.
- A gate in your pipeline
- A check for GitHub Actions and Azure DevOps that compares the runs before and after an instruction or semantic-model change. It can block a change when the evidence shows pass^k, a category or a single question fell further than your policy allows. A smaller fall, or one the evidence doesn't prove, passes and is marked "not proven". A question whose expected answer has gone out of date gets no verdict.
- Alerts on drift
- A notice when the distribution of a question's answers moves, so a shift is seen when it happens.
- Evidence packs
- A quarterly pack for risk and audit: what was tested, what changed, and how the numbers moved.
- Comparison across platforms
- The same question set run against more than one platform. This depends on connectors that have not yet run against a live agent.
- A runner inside your tenant
- Queries and answers stay in your cloud. Only scores, run metadata and anonymised traces reach us.
A first version of each exists. None has run for a customer, and alerts and evidence packs have only seen recorded runs.
02 · How it would work
The runner lives with your data
The design keeps the runner in your cloud. It asks the agent each question many times, on the capacity you choose, and sends only scores, run metadata and anonymised traces to the service, which schedules runs, checks changes and sends reports. Questions, answers and query results stay in your tenant.
For each benchmark, the service receives question IDs, wording numbers and category names, the configuration label, the connector's name and the suite hash, random IDs for the run, its schedule and its salt, a one-word verdict for each ask (correct, wrong, defensible, declined or no answer), counts, timings and cost estimates, the stage that decided each wording with counts of its distinct rewrites, sources, queries, results and answers, salted one-way hashes of answers, sources and queries, whether a question's expected answer was out of date when it was asked, and the ambiguity probe's counts. It never receives the wording of a question, an answer, a rewrite, a query, a result, a source, table, measure or term name, or a definition. Question IDs, category names and configuration labels are sent as you write them, so keep anything sensitive out of them.
Evaluation consumes capacity, as it does in an audit. The same limits apply: a ceiling on asks and their rate, agreed with you, and a clear view of where the generated queries land.
It uses the same open-source harness as the audit, so the numbers are the ones you can check.
Early access means shaping it with us
If you have an agent in production, or about to be, and you'd want to see its numbers after every change, we'd like to hear what you'd need. Early access is a conversation first. You tell us what you'd expect from it, and we tell you plainly what exists and what doesn't.
For cost, talk to us.
Talk to us about early access
The usual start is an audit or a free sample run, because Cloud is most useful once a question set exists. If you already have one, tell us.