Teaching a Model to Read Brain Scans

1. The problem

Here’s the thing that surprised me when I started: fMRI doesn’t measure neurons at all. It measures blood. When a patch of cortex gets busy, fresh oxygenated blood rushes in a few seconds later, and the scanner picks up that change as the BOLD signal. It’s small, around 1–2% of the raw signal, and slow. A common way to write it down is as neural activity $s(t)$ smeared out by a haemodynamic response $h(t)$, plus noise:

That double-gamma $h$ peaks around five to six seconds after an event and dips below zero after that. So whatever a model sees in one frame is a blurry echo of what the brain did a few seconds earlier.

Most fMRI studies are small and built for one question, so every project starts from scratch. A foundation model is the obvious fix: pretrain on a lot of unlabelled scans once, then reuse the representation for decoding states, predicting traits or spotting disease. Two things were in the way.

• The input. Most models average activity inside a few hundred regions (a parcellation). That cuts dimensionality by roughly $100\times$. Feeding the raw 4D volume keeps everything, but it’s expensive to train on.
• The yardstick. There wasn’t a shared, reproducible benchmark, so it was hard to tell whether a new model was actually better.

Our bet was a “goldilocks” representation in between, and an open benchmark to check it.

2. Flat maps

The cortex is a crumpled sheet only a few millimetres thick. If you could peel it off and lay it flat, like pressing an orange peel onto a table, you’d have an image. That’s more or less what we do. After standard surface-based preprocessing, each scan is mapped onto a template cortical surface, copied onto a flattened mesh from pycortex, and resampled to a regular grid.

2.1 From scan to video

A clip becomes a tensor

at one frame per second, so each clip covers 16 seconds. We cut it into spacetime patches of $4\times 16\times 16$ (frames × pixels × pixels). Anything outside the cortex is background, so we drop those patches and compute the loss only on real cortex. That leaves 364 spatial patches and about 77,000 cortical pixels per frame.

2.2 Normalisation

Two steps, and they matter more than I expected. Each location $v$ is z-scored over time, then each frame is normalised across space:

The first puts every location on the same scale. The second removes global swings that hit the whole brain at once. Without the first step, decoding brain states falls apart.

2.3 The other two contestants

To test the bet fairly, we trained the same model on two other inputs:

• Parcel: the Schaefer-400 atlas, one time series per region, so 400 patch streams.
• Volume: cortical voxels only, in $8\times 8\times 8$ cubes, giving 465 patches. That’s about a quarter of the sequence length of the full volume.

3. Masked autoencoding

CortexMAE is a plain spacetime masked autoencoder (He et al., 2022) with a ViT-B encoder. The idea fits in one sentence: hide most of the clip and make the model draw it back.

Write the patches as $x_1,\dots ,x_N$ and pick a mask $M\subset \{1,\dots ,N\}$ with $|M|=\rho N$. The encoder $f_{\theta }$ only ever sees the visible patches. The decoder $g_{\varphi }$ gets those encodings plus a learned mask token in every hidden slot and predicts pixels. The loss only counts the hidden patches:

where $\bar{M}$ is the visible set. Because the encoder skips 90% of the tokens, training is cheap for its size. We use $\rho =0.9$ with tube masking, where a hidden spatial patch stays hidden in every frame:

Otherwise the model could cheat by copying the same spot from the next second. Training runs for 625K steps at a batch size of 32 clips.

4. How much to hide, and where

What you hide decides what the model has to learn, so we spent a lot of time on masking. The paper compares three strategies at four ratios, scored by a probe that decodes which of 24 object categories someone was looking at in the Natural Scenes Dataset:

Two things jump out. Brains are redundant enough that you can hide 90% and still learn well, much like video. And where you hide matters as much as how much: at a moderate ratio ($\rho =0.75$) contiguous blocks beat both other strategies. That pointed me towards masks with more structure, shaped by the brain itself or by contiguous regions of context.

5. Masking by brain network

In 2011, Yeo and colleagues clustered resting-state fMRI from 1,000 people and found the cortex splits into seven large networks: visual, somatomotor, dorsal attention, ventral attention, limbic, frontoparietal control and default mode. Regions in the same network rise and fall together, even when you’re lying in the scanner doing nothing.

So instead of hiding random patches, what if you hide whole networks? Call the networks ${\mathcal{N}}_1,\dots ,{\mathcal{N}}_7$. Network masking samples a small set $K$ of them to keep and hides the rest:

Now the model has to predict, say, the default mode network from what the visual and attention networks are doing. That’s exactly the between-network structure we want it to pick up. It also makes reconstructions much easier to read, because you can check how well the model recovers each network separately.

6. Inverse block masking

Random masking has a quiet weakness. Holes are scattered evenly, so almost every hidden patch has a visible neighbour, and the model can get far by interpolating. Inverse block masking, from data2vec 2.0 (Baevski et al., 2022), flips the choice: pick a contiguous block $B$ to keep and hide everything else, so the model gets a solid region of context and has to reason outwards from it.

CAPI (Darcet et al., 2025) found a catch. Kept blocks tend to land near the centre, so the model learns to look at the middle and predict the edges. Their fix is a random circular shift of the whole mask:

where $H^{\prime }\times W^{\prime }$ is the patch grid. On images, inverse block masking with the roll reached 29.1 on ADE20K segmentation and 81.4% on ImageNet-1k, against 23.6 and 76.4% for random masking.

I adapted CAPI’s version to flat maps, sampling block aspect ratios and adding the roll. Brains add a twist that photos don’t have: a flat map has background around the cortex, so a block that lands mostly on background gives the model nothing to work with. The block has to cover enough real cortex to be useful context.

7. Scaling, with limits

We pretrained on the Human Connectome Project’s Young Adult release: 980 people, 2,058 hours, about 7.4 million frames. To see how much data helps, we trained on subsets from 400K frames (110 hours, 50 people) up to 6.6M frames (1,800 hours, 880 people) and fit a power law to the held-out loss:

That exponent is tiny. Doubling the data multiplies the loss by $2^{-0.01}\approx 0.993$, a 0.7% improvement. For language models, where $\alpha \approx -0.1$, the same doubling buys about 7%. On data from a different study (NSD) the loss improves even more slowly than the fit predicts, and downstream accuracy climbs before it flattens out.

Model size saturates early too. Going from 3 to 15 encoder layers at ViT-B proportions, both reconstruction and probe accuracy stop improving around layer 9, about 37M encoder parameters. A fairly small model can already capture HCP. My takeaway: the next jump probably comes from more diverse data, not just more of the same scanner and protocol.

8. Brainmarks

A benchmark is only useful if everyone can run it, so we released Brainmarks alongside the paper. It freezes each model’s encoder and trains only a small probe on top. That way it measures the representation, not fine-tuning tricks.

9.1 Traits

One scan per person: diagnosis on ABIDE (autism), ADHD-200, ADNI (Alzheimer’s) and PPMI (Parkinson’s), plus age (in quartiles) and sex on HCP-Aging. The probe is logistic regression on the average patch embedding,

scored by balanced accuracy, $\frac{1}{2}(\mathrm{TPR}+\mathrm{TNR})$ for two classes, averaged over 100 random splits so a lucky split can’t carry the result.

9.2 States

What is the brain doing right now? HCP-YA Task21 asks which of 21 tasks someone is doing from a 16-second clip, and NSD COCO24 asks which of 24 object categories they’re looking at. Here the probe attends over all patch embeddings instead of averaging them:

9. What we found

With the same model and training recipe, flat maps came out ahead on most tasks, and all three representations crush a functional-connectivity baseline (Schaefer-400 connectomes) at decoding states:

Against earlier fMRI foundation models (SwiFT, BrainLM, Brain-JEPA, BrainHarmonix-F, NeuroSTORM and Brain-Semantoks), CortexMAE leads on state decoding by a large margin, and the flat-map model is best overall. Several of those models were pretrained without per-location normalisation, which likely explains a lot of the gap.

Traits are the humbling part. On the four clinical datasets every model, ours included, lands within a few points of the connectome baseline, and nobody clearly beats it. Telling diagnosis from a single scan is still an open problem, and I think it’s good that a benchmark says so plainly.

10. Limits and what’s next

• Flat maps only cover the cortex. Subcortical structures like the thalamus aren’t in the picture.
• Flat maps don’t win everywhere: the volume model predicts age better, for example.
• Scaling is held back by data from a single study and weak transfer to new ones.
• Trait prediction is unreliable for every model we tested.

So the next steps more or less write themselves: better in-between representations, pretraining on many datasets instead of one, keeping benchmarks open and standard, and putting the state representations to work in the clinic.