Chest X-ray Triage Before the Report Exists: Distilling a Multimodal Teacher into an Edge Model

TL;DR. In an emergency department, chest X-rays are triaged before anyone writes the report, so a model that needs image and text cannot run at the moment it is needed. In our Smart Health paper (oral at IEEE/ACM CHASE 2026) we train an image-only student from a multimodal teacher, and we make the teacher's target the average of its predictions over five reports that MedGemma-27B writes for the same image. We call this Variational Risk Minimization (VRM). The student is 75 MB, runs in 14 ms on an RTX 2060, and reaches AUC 0.842 on MIMIC-CXR. Sending the 20.3% of cases it is unsure about to the cloud teacher gives AUC 0.941 at 103 ms average latency. Code: vrm-edge-triage.

The problem: the report comes after the decision

Multimodal models such as BiomedCLIP and Med-SigLIP are strongest when they see the image and the radiology report together. Triage is the step that decides which study a radiologist reads first, so at triage time the report does not exist yet. The text is not missing at random; the workflow removes it every time. We call this structural modality censorship: a train-test mismatch built into the clinical process.

Deployment adds a second constraint. A large model in the cloud brings latency, a network dependency and data-governance work. A model on the reading-room workstation is fast and keeps data local, but it has to be small, and it still has no report.

Two obvious fixes, and why they fall short

• Fine-tune an image encoder directly. This works and is the baseline to beat: EVA-X, the best of three medical foundation encoders we fine-tuned, reached AUC 0.828. It cannot use anything the report would have added.
• Distill from a teacher that reads one generated report. Ask a vision-language model to write a report, give it to the teacher with the image, and train the student on the teacher's output. The weakness is that the target now depends on which report the generator happened to write. When we picked one of five generated reports at random per case, 100 times over, test AUC ranged from 0.755 to 0.852. We call this the report lottery.

The idea: average over the reports you could have written

Treat the missing report as a latent variable. For each training image, sample K reports from a vision-language model, run the multimodal teacher on the image with each one, and average:

target(x) = (1/K) · Σ_k teacher(x, report_k),   report_k ~ LVLM(x)

This is a Monte Carlo estimate of the teacher's prediction marginalized over plausible readings of the image. Its variance shrinks roughly as 1/K, and one bad report moves it by a fifth instead of all the way.

The concrete setup in the paper:

• Sampler. MedGemma-27B prompted as an emergency radiologist to write findings and an impression, with temperature 0.9, top-p 0.95, top-k 50, repetition penalty 1.1 and at most 200 tokens. K = 5 reports per image, generated offline.
• Teacher. BiomedCLIP image and text encoders joined by eight learnable queries that attend to the text and then to the image (we call this Latent Query Alignment). It is frozen during distillation.
• Student. A compact image encoder plus a small projection head. It is trained with three losses: KL divergence to the averaged teacher distribution (temperature 2), cosine alignment to the teacher's averaged embedding, and weighted cross-entropy on the ground-truth urgency label.
• Task. Binary urgent versus non-urgent, defined with two chest radiologists. Pneumothorax, large effusion, acute pneumonia, pulmonary edema and severe cardiomegaly are urgent.

What averaging buys

The clearest evidence is in how the target behaves as K grows:

Three more results point the same way:

• Better than the best single report. Conditioned on each of the five reports alone, the teacher scored between 0.782 and 0.832. The average scored 0.842, within 1.4 points of an oracle that picks the best report for every case in hindsight (0.856).
• Less sensitive to hallucinated reports. Two radiologists rated 1,000 generated reports; 8% contained hallucinated findings. On test cases where at least one of the five reports was flagged, the teacher's error rate was 42.5% with one report, 14.6% with five and 8.5% with ten.
• More stable per case. The standard deviation of the per-case prediction fell from 0.095 with one report to 0.045 with five.

Returns flatten after five: the paper reports a 6.2% AUC gain from K = 1 to K = 5 and 0.3% more at K = 10, while sampling and training cost grow linearly.

Results against the baselines

The external average covers CheXpert, IU-CXR, PadChest and a private hospital cohort. The comparison follows an encoder-transfer protocol: the VRM student starts from the same encoder family as the fine-tuned baseline it is compared with, so the difference reflects how it was supervised, not which backbone it started from.

The gain over the best fine-tuned encoder is small in AUC: 1.4 points, significant under a paired DeLong test (p = 0.008). The larger differences show up where triage needs them:

• At a fixed 90% sensitivity, specificity is 65.2% for VRM, 60.3% for fine-tuned EVA-X and 45.5% for standard KD. That is fewer non-urgent studies pushed to the front of the queue.
• Calibration error (ECE) is 0.092 against 0.119 for standard KD, which matters when a threshold decides who waits.
• The negative predictive value at the balanced operating point is 0.924 on MIMIC-CXR.

Asking a large vision-language model directly is not a substitute: MedGemma-27B reached 0.762 with four examples in the prompt, at more than a hundred times the student's latency, and general-purpose models scored below 0.44.

A cascade: edge first, cloud when unsure

The student runs on every study. If its confidence is high and its predictive entropy is low, its answer stands; otherwise the study goes to the cloud teacher. The two thresholds were set on the validation split to escalate about one case in five and then frozen.

Average latency follows from the escalation rate: 0.797 × 14 ms + 0.203 × 450 ms ≈ 103 ms. The threshold is a dial: escalate more to approach the teacher's accuracy, less to stay local. In the paper's setup, escalated cases are de-identified at the edge or sent to the hospital's private cloud.

What it does not fix

• The sampler has to know radiology. With two general-purpose samplers (InternVL2-26B, Qwen2.5-VL-32B) the student reached 0.779 and 0.786: above standard KD, below every fine-tuned baseline. A medical sampler of similar size (HuatuoGPT-Vision-34B) reached 0.839. Averaging amplifies good reports; it cannot create them.
• Consistent biases survive averaging. Of 200 misclassified cases, 36% confused chronic and acute changes, 31% over-read lines, rotation or low exposure as opacity, and 21% over-triaged small effusions. Averaging removes random errors, not errors every sample shares. That is one reason uncertain cases should go to review.
• Some findings need context. Acute pneumonia had the lowest F1 among urgent classes (0.724), mostly from confusion with chronic atelectasis, a distinction that often depends on history only the report carries.
• The teacher sets the ceiling. The student cannot learn what the teacher does not know.

When this pattern applies

The setting is not specific to chest X-rays. Any time a modality is available for training but structurally absent when the decision is made, you can sample the missing modality from a generative model and distill from the averaged teacher: lab results that return hours later, notes written at discharge, reads that arrive after a scan is scheduled. A short checklist:

• Confirm the missing input is missing by workflow, not at random; that is what makes a training-time-only modality worth distilling.
• Use a domain sampler and have clinicians rate a sample of its outputs before trusting it as supervision.
• Start with K = 5 and check whether AUC still moves at 10.
• Measure stability, not only the mean: rerun with different single samples and report the spread.
• Report specificity at the sensitivity you would deploy at, and calibration, next to AUC.
• Put a confidence gate in front of the small model and decide the escalation budget explicitly.
• Look for errors that repeat across samples; averaging will not remove them.

Try it

The repository vrm-edge-triage has the teacher (BiomedCLIP with Latent Query Alignment), the student with EVA-X encoder transfer, the VRM losses and the scripts for each step:

python scripts/generate_variational_samples.py --data_root data/mimic_cxr --split train \
  --k_samples 5 --output_file data/mimic_cxr/train_variational_samples.json
python scripts/train_teacher.py --data_root data/mimic_cxr --output_dir outputs/teacher
python scripts/train_student_vrm.py --data_root data/mimic_cxr \
  --teacher_checkpoint outputs/teacher/best_model.pt \
  --samples_file data/mimic_cxr/train_variational_samples.json \
  --output_dir outputs/student_vrm
python scripts/evaluate.py --model student --checkpoint outputs/student_vrm/best_model.pt \
  --data_root data/mimic_cxr

Training needs your own credentialed copy of MIMIC-CXR. The repository README lists the data layout and the EVA-X checkpoint the student starts from. Q-DISTILL, from the same line of work, is a self-supervised Q-Former distillation for image-only chest X-ray triage using MedGemma pseudo-reports.

FAQ

How do you train an image-only model when reports are only available at training time? Train a multimodal teacher on image and report, then distill it into an image-only student. VRM makes the teacher target an average over several reports sampled from a vision-language model for the same image, instead of one report, so the student learns from a stable, uncertainty-aware target.

How is VRM different from standard knowledge distillation? Standard distillation matches one teacher output per image. VRM matches the teacher's prediction averaged over K sampled reports. On MIMIC-CXR this gave AUC 0.842, against 0.748 for standard KD and a mean of 0.808 when one randomly chosen report is used instead of five.

How many sampled reports are needed? Five was the practical choice in our experiments: the paper reports a 6.2% AUC gain from K = 1 to K = 5 and only 0.3% more at K = 10, while training time grows linearly with K.

Does the deployed model need a report or a language model at inference? No. The student reads the image only: 75 MB and 14 ms per image on an RTX 2060. In the cascade, only cases the student is unsure about (20.3%) go to the cloud teacher.

Is the code available? Yes, at github.com/Yangxinyee/vrm-edge-triage: the teacher, the student, the VRM losses, offline report sampling, and training and evaluation scripts. Training needs your own credentialed copy of MIMIC-CXR.