Two Minds, One Model

1. Reading and drawing are different jobs

Medical AI has split into two camps. Comprehension models look at an image and answer questions about it: is there a fracture, what organ is this, what does this slide show. Generation models make images: turn a CT into an MRI, sharpen a blurry scan, fill in a reconstruction. Over the past few years each camp has produced strong models, but almost never one that does both.

Putting both in one model runs into two problems:

1. Data. Medical data is scarce because of privacy, and there was no large dataset covering both kinds of task.
2. Conflict. Even with the data, the two tasks pull the model in different directions, and each gets worse.

The paper takes on both. The rest of these notes follow its answers.

2. Feeding both: VL-Health

For the data problem, the authors assembled VL-Health from existing public medical datasets: about 765K comprehension samples and 783K generation samples, so the two halves are almost exactly balanced. The comprehension half spans eleven named imaging modalities, from CT and MRI to fundus photos and histology slides. The generation half covers five tasks, with reconstruction the largest.

3. The tug of war

The second problem is subtler. In the authors’ words, comprehension tasks “strip away visual details to focus on abstraction, while generation tasks require detailed preservation”. To say “this is a fractured radius”, a model can throw away texture. To draw the MRI version of a CT slice, it needs every edge.

One way to picture it: with shared weights $\theta$, each training step follows the sum of both tasks’ gradients,

and whenever $⟨\nabla {\mathcal{L}}_{\text{comp}},\nabla {\mathcal{L}}_{\text{gen}}⟩<0$, part of every step that helps one task undoes progress on the other. The paper measures the effect directly. Hold one task’s data fixed, add more of the other, and watch the fixed task decline:

With shared weights (solid lines), adding generation data pulls comprehension down from about 62 to 48, and adding comprehension data pulls generation down from about 54 to 47. With HealthGPT (dashed), more data of either kind helps or does nothing. That is exactly what you want from a unified model.

4. Plugins, not one brain

The fix starts from LoRA. Instead of fine-tuning a big weight matrix $W_0$, LoRA freezes it and learns a small low-rank update:

A LoRA is cheap and detachable, which makes it a natural “plugin”. MoELoRA goes further and keeps $k$ such experts, mixed by a router $\omega (x)$:

That sum is the catch: each expert is a separate pair of matrix multiplies, so training time grows with $k$. HealthGPT’s H-LoRA concatenates the experts into one wide pair,

stretches the router’s weights $\mathcal{W}\in {\mathbb{R}}^{\text{tokens}\times k}$ so each one covers its expert’s $r$ columns, and applies them inside a single multiply:

Mathematically it’s the same mixture. Computationally it costs about the same as one LoRA. With four experts, the paper reports H-LoRA needs about 67% of MoELoRA’s training time, and the gap widens as $k$ grows:

The second half of the trick is a hard router at the task level. Comprehension and generation each get their own H-LoRA plugin, ${\theta }^{\text{comp}}$ and ${\theta }^{\text{gen}}$, and the task type $T$ decides which one is plugged in. The two never share low-rank weights, so they can’t fight over them.

5. Seeing at two depths

The same split happens on the vision side. A vision transformer (here CLIP-L/14) builds up its picture in layers: early layers keep edges and textures, later layers hold abstract concepts. HealthGPT calls this hierarchical visual perception and picks the layers to match the job:

In practice that’s the second layer’s hidden states for drawing (concrete detail) and the penultimate layer’s for reading (abstract meaning), each through its own small adapter. It’s a neat idea: rather than forcing one set of visual features to serve both tasks, use the fact that a ViT already computes both kinds.

6. Pictures as words

To draw with a language model, images have to become tokens. HealthGPT uses a VQGAN (the f8-8192 version): it compresses an image 8× in each direction and replaces each patch with one of 8,192 codebook entries. A 256-pixel-square image, for example, becomes a 32 × 32 grid, or 1,024 tokens. Those codes join the language model’s vocabulary, so generating an image is just next-token prediction:

where $\mathcal{U}$ is the instruction (and any input image) and $i_j$ are the image’s code indices. Training minimises the usual cross-entropy, $\mathcal{L}=-{\sum }_j\log P_{\theta }(i_j\mid \mathcal{U},i_{<j})$, and a VQ decoder turns the codes back into pixels. Put all of this together and you get the full architecture. Switch tasks to see the path the router picks:

7. Teaching it in three stages

The last piece is the training schedule. The order matters as much as the architecture: each stage freezes whatever the previous one got right.

Stage 2 is the clever one. After stage 1, the two plugins were trained separately, so they don’t quite agree on the shared parts of the model. A small mixed dataset, with the plugins frozen, fits the shared embeddings and output head to both at once. Then stage 3 can tune each plugin for real tasks without disturbing the other.

8. The report card

8.1 Reading

On medical visual question answering, HealthGPT beats both the medical models and the general unified ones. Closed-question accuracy on three benchmarks, plus MMMU-Med, OmniMedVQA and the paper’s overall average (which also includes the open questions I’ve left out):

The detail I like most: the smallest HealthGPT, at 3.8B parameters, beats every 7B medical model in the table by more than ten points on average.

8.2 Drawing

Image quality is measured against the real target image. Mean squared error and peak signal-to-noise ratio work pixel by pixel:

SSIM compares local brightness, contrast and structure, which is closer to what a radiologist notices:

Higher is better for SSIM and PSNR, lower for MSE and LPIPS (a perceptual distance computed from deep features). On modality conversion, here is SSIM across the four tasks:

HealthGPT wins three of the four, with the biggest margin on CT to MRI in the pelvis (about nine points over the next best). The exception is honest and interesting: on MRI to CT in the brain, the diffusion model BBDM still has the best SSIM, though HealthGPT has the best PSNR and MSE there. On super-resolution it wins every metric, and its LPIPS is about 37% lower than the next best:

9. What I took from it

The lesson I keep coming back to is that “unified” doesn’t have to mean “shared everything”. HealthGPT shares the expensive part, a frozen language model, and keeps the parts where the tasks disagree separate: which visual layers to look at, and which low-rank plugin to use. The interference curves in Figure 2 are the best argument for that design I’ve seen.