Building an RL Environment for Medical QA

1. What an environment is

When people say “RL for language models”, they usually mean the training algorithm. But the part that decides what the model actually learns is the environment: where the questions come from, how an answer gets produced, and how it gets scored. Get the scoring wrong and the model will happily learn the wrong thing.

Formally, the model is a policy ${\pi }_{\theta }(y\mid x)$ that writes an answer $y$ to a prompt $x$. An environment gives you prompts $x\sim \mathcal{D}$ and a reward function $r(x,y)\in [0,1]$, and training tries to push up

Prime Intellect’s verifiers library packages all of that as a small Python module with a load_environment function: a dataset, the rollout logic, parsers for the model’s output, and a rubric made of one or more reward functions. The same module runs as an evaluation with vf-eval or feeds rewards to a trainer like prime-rl. The Environments Hub, launched in August 2025, is where people publish them.

2. The data: K-QA

K-QA (Manes et al., 2024) comes from 1,212 real questions people asked on K Health. That’s what makes it interesting: these aren’t exam questions, they’re the messy things people actually worry about. For 201 of them, physicians wrote full answers and broke each one into statements, split into two sets:

Must-have statements are things a good answer has to say. Nice-to-have statements are helpful but optional. The environment uses those 201 physician-answered questions.

3. Grading free text

Grading a multiple-choice answer is trivial. Grading a paragraph of medical advice is not. The environment is single-turn (one question, one answer) and grades in two passes.

3.1 Break the answer into claims

An extractor model splits the answer into short, standalone facts, in the style of FActScore (Min et al., 2023):

“Ibuprofen can irritate the stomach, so take it with food” becomes two claims, and each can be checked on its own.

3.2 Check them against the doctors

A second judge does natural-language inference: for each gold statement $s\in \mathcal{S}$, does the answer entail it, contradict it, or not mention it? Splitting first makes the judge’s job smaller. It decides one claim at a time instead of eyeballing a whole paragraph.

4. The reward

K-QA measures two things, and the reward uses both. Comprehensiveness is a kind of recall over must-haves:

The contradiction rate counts how many of the doctors’ statements the answer goes against:

where $y\models s$ means the answer supports $s$ and $y\perp s$ means it contradicts it. The reward weights them equally:

I like this balance because it closes off the two easy ways to game a medical grader. An answer that dumps every fact it knows will cover the must-haves but probably contradict something. An answer that just says “see a doctor” contradicts nothing, but covers nothing either. To score well you have to be complete and right.

5. From a reward to training

GRPO (Shao et al., 2024) is the usual way to turn a reward like this into learning. For each question it samples a group of $G$ answers $y_1,\dots ,y_G$, scores them all, and compares each one to its siblings instead of training a separate critic:

5.1 A worked example

Say four answers to the same question score $r=(0.82,0.41,0.67,0.18)$. The mean is $0.52$ and the standard deviation is about $0.245$, so the advantages come out as

The first and third answers get reinforced, the other two get pushed down. And because the comparison is only ever within one question, it doesn’t matter that some questions are much harder than others: a 0.4 on a brutal question can still be the best answer in its group.

5.2 The objective

Those advantages plug into a clipped, PPO-style objective with a KL penalty that keeps the model near a reference ${\pi }_{\text{ref}}$:

where the ratio compares the new policy to the one that sampled the answers:

Every token in an answer shares that answer’s advantage.