Grand Rounds for Machines

1. Passing the boards is not practising medicine

The first wave of medical benchmarks were exam questions. MedQA, for example, is built from USMLE-style multiple choice, and the best models now clear it comfortably. That tells us they know a lot of medicine. It doesn’t tell us whether they can be trusted with a patient, because real clinical work is open-ended, spread across a messy record, and full of moments where the obvious answer is the wrong one.

OpenAI’s HealthBench (Arora et al., 2025) was a big step. It has 5,000 realistic health conversations, and for each one, physicians wrote a rubric: specific things a good reply should do, each worth points, and things it must not do, worth negative points. In total, 262 physicians from 60 countries wrote 48,562 criteria. A model grader checks each criterion, and conversation $i$ scores the points it earned over the points it could have earned:

Because the harmful criteria carry negative points, a single conversation can score below zero. The benchmark score is the average, clipped to the unit interval:

I like this design a lot, and all three of my ideas reuse it. Try it below: tick what an answer did and watch the score move. Notice how the harmful answer goes negative even though it gets one thing right.

What HealthBench doesn’t cover is a patient’s history. Its conversations are mostly self-contained and text-only. Nobody hands the model three years of labs, a CT report and a medication list and asks it to work something out. That’s the gap I care about.

2. Chartwalk: read the whole chart

The first idea is HealthBench for agents, with the patient’s record attached. The closest work is LifeAgentBench (Tian et al., 2026): 22,573 questions over long stretches of lifestyle data, with answers computed by executable queries so they can be checked exactly. Its baseline agent, LifeAgent, breaks a question into steps, retrieves evidence over several rounds, and uses tools to aggregate it, rather than asking the language model to do arithmetic in its head. That is the right shape of agent. But the data is sleep, steps and meals, not clinical records, images or treatment history.

Chartwalk keeps the shape and swaps in the clinic. We simulate patients with full histories, so the agent has to reason across every kind of record a doctor would.

2.1 A patient is a stream of events

A patient record is a set of timestamped events, each in some modality:

Here $t_k$ is when it happened and $x_k$ is the content: a paragraph, a lab value, a dose, a CT slice, or a stretch of a vitals chart. Open tools like Synthea can already generate realistic synthetic histories, which we’d extend with notes and images. Because the patients aren’t real, there’s no privacy problem, and we control exactly what’s in each chart.

2.2 Questions with an answer key

Each question $q$ comes with the evidence a careful clinician would look at, $E^{\ast }(q)\subseteq \mathcal{P}$, and a small program $f_q$ that turns that evidence into the answer:

That borrows LifeAgentBench’s best trick. “Did her creatinine rise after lisinopril?” has a right answer you can compute, so we don’t need a judge for the number, only for the explanation. We can make questions harder in three ways: more events, more modalities, and a longer stretch of time between them.

2.3 Grading the walk

The agent works in steps, calling tools to search the chart, and ends up citing a set of events $\hat{E}$ and an answer $\hat{a}$. We grade the retrieval separately from the answer, because a right answer from the wrong evidence is luck:

Then the full score mixes three things: whether the answer matches the program’s, how good the retrieval was, and a physician rubric for the reasoning and how it’s explained:

I’d start with equal thirds. The figure below has one synthetic patient and three questions. Notice how the agent on Q1 gets close but still lands on the wrong number, because it skipped one baseline lab. That failure is invisible if you only grade the final answer by eye.

3. U-Turn: the case where the first plan was wrong

Every hospital has morbidity and mortality rounds, where clinicians go through cases that went wrong and ask why. Often the story is the same: an early, reasonable-looking decision, new information that should have changed it, and a delay before anyone did. Doctors get anchored. The question for U-Turn is whether an AI can do better: notice its plan is wrong, break out of a flawed line of reasoning, and recover.

3.1 Cases from M&M rounds

The data source is AHRQ’s Patient Safety Network, whose WebM&M series publishes cases of medical errors reported anonymously by healthcare workers, each with commentary by an expert on what went wrong and what should have happened. That commentary is exactly what we need for a rubric: it says what the right decision was, and when.

3.2 Inherit the mistake

The design choice I’m most excited about: we don’t wait for the agent to make the mistake. We prefill it. The episode starts with the real case’s history up to the wrong decision, written as if the agent had made it. At step $t^{\ast }$ the new evidence arrives, and the agent takes over:

That isolates the skill we care about. If the agent makes its own decisions from the start, a model that happens to avoid the original error never gets tested on recovery at all. We can still run that version separately, and count how often a model repeats the clinician’s first mistake.

3.3 Scoring a recovery

Let $t_r$ be the first step where the agent’s decision is consistent with the corrected plan. The recovery delay is $\Delta =t_r-t^{\ast }$, and the recovery score decays with it:

With $\gamma =0.9$, turning around two steps after the evidence scores 0.81 and four steps scores 0.66. Looping back to the old plan scores nothing. Switch between the three agents below:

Speed alone would reward an agent that changes its mind at every new lab value, so U-Turn also needs control cases, where the first plan was right and the new result is a red herring. There, turning is the mistake. The full score has three rubric parts, matching the three things M&M commentary tends to judge:

4. Yes, Doctor: hallucinations hide in the chart

Most hallucination benchmarks, like Med-HALT (Pal et al., 2023), test whether a model invents facts. In a clinic, the more dangerous failures are quieter: repeating a common misconception, agreeing with a senior colleague who is wrong, or applying the textbook protocol to the one patient it doesn’t fit. Yes, Doctor has three tracks, and every item comes with a patient record so the model has the evidence to get it right.

4.1 Myths at the bedside

Each item pairs a case with a misconception and the expert explanation. Some classics: antibiotics for acute bronchitis (it’s usually viral, so they rarely help); a normal ECG ruling out a heart attack (it doesn’t, which is why we measure troponin); and a penicillin allergy on the chart being permanent (most people with that label turn out not to be allergic when tested). The rubric rewards the expert reasoning and penalises the myth, even when the myth is stated confidently in the prompt.

4.2 The agreeable assistant

Language models are known to tilt their answers toward what the user seems to believe (Sharma et al., 2023). In medicine that’s dangerous, because the user is often a clinician with authority. We measure it with paired prompts: the same case $x$, with and without a clinician’s wrong opinion $o$. The flip rate is how much the opinion alone moves the model:

A model that ignores clinicians entirely would also have $\Phi =0$, so we mix in cases where the clinician is right, and the model should agree. The record itself can also be wrong. We plant contradictions $F^{\ast }$ in the charts, like an allergy list that disagrees with an old note, and score how many the model flags without being asked:

4.3 Protocol versus patient

Guidelines are written for the typical patient. Take an acute gout flare: anti-inflammatory painkillers (NSAIDs) are a standard first choice. But if the record shows stage 4 kidney disease and warfarin, they’re a poor fit, and a short course of steroids is the usual alternative. Track C items are built so the protocol answer is wrong for the patient in front of you. Controls where the protocol is right stop a model from scoring well by always deviating, so we report balanced accuracy:

5. One rubric to grade them all

All three share the same skeleton: a patient record, a task, a program or expert answer where one exists, and a physician-written rubric for everything else. Side by side:

A rubric is only as good as its grader. Before trusting a model grader at scale, we’d have physicians grade a sample themselves and measure agreement with Cohen’s kappa, which corrects raw agreement $p_o$ for the agreement $p_e$ you’d get by chance:

6. What could go wrong

• Synthetic patients are too clean. Real charts have copy-pasted notes, missing values and typos. If the simulator doesn’t add that noise, Chartwalk will be easier than real life.
• Graders that are fooled by fluent text. A confident, well-written wrong answer can satisfy a careless grader. The kappa check in section 5 is there for that.
• Contamination. PSNet cases are public, so models may have read them. Paraphrasing the cases and changing surface details helps, and so does a held-out set written fresh.
• One right answer is rare in medicine. Rubrics need room for more than one reasonable plan, which is another reason physicians, not templates, should write them.