MedAgentBench
Electronic health record — read and write
Whether the agent can operate a patient chart over a standard health-records API: read the right resource for a clinical question, and write a correct, conformant one back. Graded deterministically against chart state.
Ran on
claude-haiku-4-5
the cheapest and fastest model in its family
provider Anthropic
The harness pinned nothing — no model, no provider, no reasoning-depth or speed setting. Those request controls did not exist on the benchmark endpoint when these runs were made, so every arm inherited the production deployment default.
Read from the deployment default in force on each run date, not from the run's own record: the endpoint did not report which model served a turn until after these runs, and no run artifact names one. Automatic failover to the secondary provider (Gemini 2.5 Flash) is logged but not recorded per turn, so we cannot rule out that a small fraction of turns was served by it.
No setup-matched baseline yet — 11 published figures shown for context, under a different setup
Run 8 August 2026.
Every metric this run produced
| Metric | Value | Sample | Interval |
|---|---|---|---|
| Overall success rateHeadlineDeterministically graded against live chart state. 95% interval as reported by the harness in report.json, transcribed rather than recomputed. | 54.0% | 54 of 100 tasks | 95% CI 44.3–63.4% |
| Query success rateReading the right resource for a clinical question. 95% interval as reported by the harness in report.json, transcribed rather than recomputed. | 68.0% | 34 of 50 tasks | 95% CI 54.2–79.2% |
| Action success rateWriting a correct, conformant resource back. 95% interval as reported by the harness in report.json, transcribed rather than recomputed. | 40.0% | 20 of 50 tasks | 95% CI 27.6–53.8% |
Run this yourself
You will need: Node 18+, uv, and Docker to run the benchmark's FHIR sandbox locally. Plus a Whissle workspace key (Settings → API keys on whissle.ai). There is no simulated user in this benchmark and no grader model, so there is no third-party key to obtain.
git clone https://github.com/WhissleAI/tau2-bench-w && cd tau2-bench-w
uv sync --extra dev
docker run -p 8090:8080 <fhir-sandbox-image> # the benchmark's own sandbox
export WHISSLE_BASE=https://aws-gateway-backend.whissle.ai/bot WHISSLE_API_KEY=... FHIR_API_BASE=http://localhost:8090/fhir/
python -m tau2.health.medagent.run --mode brain-parity --limit 100 --write-check execute
python -m tau2.reporting.cli build results/whissle/medagentbench/brain-parity_mab_100The subset is the leading 100 tasks of the published set and reproduces exactly. Research measurement only — MedAgentBench, NEJM AI 2025.
Running a benchmark straight from the Whissle CLI is on the roadmap and does not exist yet — the CLI manages agents, calls and records today, and we are not going to print a command on this page that would fail when you paste it. Until it lands, the harness above is the path.
What we measured against
Our arm in every row below
Whissle ran on
claude-haiku-4-5
the cheapest and fastest model in its family
provider Anthropic
The harness pinned nothing — no model, no provider, no reasoning-depth or speed setting. Those request controls did not exist on the benchmark endpoint when these runs were made, so every arm inherited the production deployment default.
Read from the deployment default in force on each run date, not from the run's own record: the endpoint did not report which model served a turn until after these runs, and no run artifact names one. Automatic failover to the secondary provider (Gemini 2.5 Flash) is logged but not recorded per turn, so we cannot rule out that a small fraction of turns was served by it.
Scoring 54.0% over 100 scored tasks. Where a row below names a different model, that difference is listed with the others.
No setup-matched baseline yet
We have not yet run another model through this exact harness — same domain, same task set, same user simulator, same pass^1, same concurrency — so there is no head-to-head to show. Published figures for other systems appear below for context, greyed, with the reasons they are not directly comparable. When a matched run lands it will appear here as a real comparison.
Published figures, for context
These were produced by other people under other conditions. They are not subtracted from our number and no margin is claimed against them. The public τ² ledgers describe themselves as a record of what was reported rather than a controlled cross-provider ranking, and the rows below differ from ours on the dimensions listed.
| System | Published score | Why it is not a head-to-head | Source |
|---|---|---|---|
| Claude 3.5 Sonnet v2Published — we quoted it | 69.7% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| GPT-4oPublished — we quoted it | 64.0% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| DeepSeek-V3Published — we quoted it | 62.7% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Gemini-1.5 ProPublished — we quoted it | 62.0% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| GPT-4o-miniPublished — we quoted it | 56.3% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| o3-miniPublished — we quoted it | 51.7% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Qwen2.5Published — we quoted it | 51.3% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Llama 3.3Published — we quoted it | 46.3% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Gemini 2.0 FlashPublished — we quoted it | 38.3% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Gemma2Published — we quoted it | 19.3% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
| Mistral v0.3Published — we quoted it | 4.0% | Same action grammar and the same deterministic grader on both sides, which is what makes this a real comparison. The difference is the denominator: 300 published tasks against our 100, and ours is the head of the set rather than a random draw.
| MedAgentBench, NEJM AI 2025 (Table 2), published 2025 |
For orientation only, repeated here so nobody has to scroll to hold both in mind: our figure on this benchmark is 54.0% over 100 scored tasks, on claude-haiku-4-5 (the cheapest and fastest model in its family). Not a ranking against the rows above.
What was thrown away
Attempted
100
Scored
100
Excluded
0
None excluded — the sample is the whole attempt.
Nothing was dropped. A task that had not emitted a finish marker within eight rounds was scored incorrect, not retried and not excluded.
What this number can and cannot be put beside
The most directly comparable number on this page, and the least favourable. Same action grammar and the same deterministic grader as the published table, with no judge model on either side — so the gap is a real gap and not a difference of instrument. Two caveats bound it in our favour and are stated for that reason: 100 of 300 tasks were scored, head-of-set rather than random; and writes go to a sandbox that accepts resources a production record system would reject — 7 of 35 emitted writes were non-conformant and still scored correct, which makes this an upper bound on what the same agent would achieve against a validating server.
Comparable with
- Other runs of this same benchmark on this page, at the same modality (Text).
- Runs of this benchmark on the same arm — claude-haiku-4-5 (the cheapest and fastest model in its family). A run on a different model is a different experiment.
- Other Pass^1 figures — one attempt per task, no best-of-n.
Not comparable with
- The published figures shown for Claude 3.5 Sonnet v2, GPT-4o, DeepSeek-V3, Gemini-1.5 Pro, GPT-4o-mini, o3-mini, Qwen2.5, Llama 3.3, Gemini 2.0 Flash, Gemma2, Mistral v0.3. They are on the page for context, under a different or unstated setup — task set, scaffold, user simulator and pass^k all vary between published rows, and the public ledgers describe themselves as a record of what people reported rather than a controlled cross-provider ranking.
What passing and failing look like
No published cases for this run
The same benchmark over time
One run on record
| Run date | Score | Scored | Excluded | Ran on | Status |
|---|---|---|---|---|---|
| 8 Aug 2026 | 54.0%(54 of 100) | 100 | 0 | claude-haiku-4-5 | Published |
How this run was produced
- Agent under test
- The deployed Whissle agent brain, unmodified, with the deployed persona suppressed so the benchmark's instructions are the only instructions — which is what makes the number comparable to the published table.
- How it was run
- Whether an agent can operate a real electronic health record: read the right resource for a clinical question, and write a correct, conformant resource back when the task calls for it. Grading is deterministic against live chart state — no rubric, no grader model, no partial credit. The agent supplies reasoning only; the protocol is textual read/write/finish strings the harness parses and executes against a sandbox. Eight rounds per task, then it is scored incorrect.
- Model config
- Model claude-haiku-4-5Provider Anthropicthe cheapest and fastest model in its family, in its own vendor's lineup.The harness pinned nothing — no model, no provider, no reasoning-depth or speed setting. Those request controls did not exist on the benchmark endpoint when these runs were made, so every arm inherited the production deployment default.Read from the deployment default in force on each run date, not from the run's own record: the endpoint did not report which model served a turn until after these runs, and no run artifact names one. Automatic failover to the secondary provider (Gemini 2.5 Flash) is logged but not recorded per turn, so we cannot rule out that a small fraction of turns was served by it.
- Judge
- Independent judge — the benchmark's own deterministic grader. Ships with the benchmark and decides by inspecting resulting state against the task's requirement. No rubric, no grader model, no partial credit — and nothing we supply, operate or could tune in our favour.
- Sampling
- Head-of-set subset — the leading 100 tasks, balanced 10 per category by construction Population: All 300 tasks in the published MedAgentBench set. No seed — the selection was not randomised. 1 attempt per task.
- Run date
- 8 August 2026
- Harness commit
86b4475- Cost
- Not recorded for this run.