Tested in public: how to read the register.
A medical AI that asks for your trust should show its exam results, all of them, with the marking scheme. Asha’s public register at dnai.systems/benchmarks does that. This post explains what each row means, in plain language, and why the rows where Asha did not lead are on the page too.
The publication rule
A row appears on the register only when it states which pipeline of Asha was tested, which benchmark set and how many items, who or what graded it, and where the artifact lives. Every proportion carries a 95% interval. Comparators are labelled by how they were scored: the same grader in our own run, an independent validator, the vendor’s own report, or a published paper. Only values graded the same way share a bar with ours. Samples under 100 items sit in an appendix with their interval, unless the benchmark’s own protocol fixes the count.
That rule is why the page reads more slowly than a headline. It is also why you can check it.
What a rubric score is
HealthBench (Arora et al., 2025) is a set of health conversations, each with a rubric written by physicians: the things a good answer must say, and the things that cost points if it says them. A model’s answer is graded against that rubric by a grader model. The score is the share of rubric points earned. Two facts follow. First, the grader matters: the same answer graded by two different models gets two different numbers, so a score without its grader is not a result. Second, length matters: a longer answer can earn more rubric points, so the benchmark’s official metric for its clinician-facing set is length-adjusted.
The headline rows, in plain language
| Row | What it measures | Asha | Read it as |
|---|---|---|---|
| HealthBench Professional (525 items, 14 Sep 2026, Claude Opus 4.8 grader, two trials pooled) | Clinician-facing questions, official length-adjusted metric | 65.1; the leading bare frontier model 67.0 in the same harness and grader | Asha’s best configuration sits just under the leader on the official metric. The register notes the gap tracks answer length; Asha leads on the raw score and the red-team set. We publish the official metric first. |
| HealthBench, 500-item independent sample (10 Sep 2026, gpt-4.1 grader) | The item sample and harness of an independent study (Vishwanath et al., Nature Medicine, 2026), run by an independent validator | 66.6; the study reported OpenEvidence 62.6 and UpToDate Expert AI 61.3 on the same items | Someone who does not work for us ran Asha through a published pipeline. The claim ends at this item set. |
| HealthBench Hard (1,000 items, 27 Jun 2026, gpt-4.1 grader) | The hardest subset, full set, Asha’s research configuration | 44.8; bare frontier models in the same harness 40.3 to 11.6 | Hard means hard: nobody scores high. Two of the four bare comparators kept only aggregate scores, and the row says so. |
| psychosis-bench (96 turns, 11 May 2026, protocol judge) | Safe intervention across 16 multi-turn scenarios that drift toward delusion or self-harm | 95.8% safe-intervention rate; the same bare backbone without Asha’s architecture 30.2% | The architecture, on the same model, is the difference. Newer bare models score higher on this protocol than the May backbone did, and the register says that too. |
| Locked pediatric ear-infection chart (14 runs, 26 Aug 2026) | A clinician’s public, locked chart: boundary arithmetic, weight-based dose, duration, declared assumptions | 14 of 14 runs passed every scored dimension | Fourteen is a small number, set by the chart. The author’s own adjudication is pending and owns the scoreboard; our self-graded traces are receipts until then. |
The rows that do not flatter
On the standard HealthBench set (5,000 items, June 2026), Asha’s mid tier scored 60.6 and the retrieval layer was neutral in a paired ablation: plus 0.7 points, p = 0.37. The register says so in the row’s notes. The retrieval layer earns its keep on the Hard set, where the questions need sources, and you should know where it does and does not help.
MedQA, the multiple-choice licensing-exam benchmark, is on the register, and we never lead with it in our own copy. A letter-match score on exam questions says little about how a system behaves with a frightened person at 2 a.m., so the rubric and safety rows come first.
What we say, and what we do not
- We say every answer is tamper-evident and time-stamped. We do not say the record certifies truth.
- We say Asha runs safety gates and a mandatory check-in on every reply. We describe what the system does, and we publish the safety benchmark that measures it.
- We give every benchmark figure with its date, harness and grader, and we link the register. A number without those three is marketing.
- We count questions asked and answered as the measure of whether Asha is useful.
Then ask Asha a question and check the sources yourself. Five questions, no account.
Sources
- DNAi Systems public benchmark register, dnai.systems/benchmarks, JSON at api.askasha.org/api/benchmarks, updated 18 September 2026. All figures above are quoted from it.
- Arora RK et al. HealthBench: evaluating large language models towards improved human health. arXiv:2505.08775, 2025.
- Vishwanath et al. Nature Medicine, 2026 (500-item HealthBench sample and harness; independent run by Airpixels Innovations LLP).
- Au Yeung et al. psychosis-bench. arXiv:2509.10970, 2025.
- Jin D et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams (MedQA). Applied Sciences, 2021;11(14):6421.