Learn AI Create the Future — with Pranay
Main site

Medical Imaging AI

Generating radiology reports

Writing a scan's findings in plain English is image captioning with unusually high stakes, where a fluent but wrong sentence can be mistaken for a real diagnosis.

Read these first

On this page 6
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. An honest warning
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Radiology report generation means a model writes the findings section of a scan report, in plain sentences.

Think about describing a photo to a friend who cannot see it. You would try to be accurate, and specific, about what is actually there.

Describing a scan is that same task, at much higher stakes. Getting a detail wrong in a photo caption is embarrassing. Getting a detail wrong in a radiology report can affect real patient care.

Why it exists

Radiologists write a findings report for every scan, in careful, specific medical language. This takes real time, and reporting backlogs are common in busy hospitals.

Report generation systems try to draft a first version of this text automatically. A radiologist then reviews, corrects, and approves it. It applies the same technique behind photo captioning — teaching a model to turn an image into a sentence. Here, though, a wrong sentence has real consequences.

How it works

Scan image
    |
    v
[ vision encoder ]  ->  learns what is in the image
    |
    v
[ language decoder ]  ->  writes a findings sentence
    |
    v
"The cardiac silhouette is enlarged, consistent with cardiomegaly."

The model has to get two things right at once. It must correctly notice what is in the image, and describe it using accurate medical language.

Where you have already seen it

  • Auto-generated photo captions on your phone or social media use the same basic image-to-text idea, at much lower stakes.
  • A subtitle-generation tool turning spoken audio into text has the same "produce fluent, plausible-sounding output" risk, in a different medium.

An honest warning

A generated report can sound completely confident and medically fluent, while describing a finding that is not actually present. This is called hallucination — the model produces plausible-sounding text instead of an honest "not sure." A fluent wrong sentence is more dangerous than one that plainly looks broken — it is harder to catch. No report-generation system belongs in real clinical use without a radiologist reviewing every output first.

Remember this

  • Report generation is image captioning, applied to scans, with far higher real-world stakes.
  • A generated report can sound confident and be factually wrong — hallucination is a real, documented risk.
  • A qualified radiologist reviewing every generated report is a necessary safeguard, not an optional extra.

What to learn next

  • Image captioning — the general technique this lesson specialises into a high-stakes domain.
  • Hallucination — the general risk this lesson's honest warning is built around.
  • Vision-language models — the modern architecture family behind real report-generation systems.

Developer — Code and libraries.

Setup

bash
python3 --version

This example uses the standard library only, and is deliberately not a trained model — it is a template-based generator, built to show the mechanics without needing a large pretrained network.

Minimal runnable code

template_report.py
FINDINGS = {
    "cardiomegaly": "The cardiac silhouette is enlarged, consistent with cardiomegaly.",
    "effusion": "There is blunting of the costophrenic angle suggesting pleural effusion.",
    "pneumonia": "A focal opacity is present, which may represent pneumonia.",
}

def build_report(finding_probs, threshold=0.5):
    positive = [name for name, prob in finding_probs.items() if prob >= threshold]
    if not positive:
        return "No acute cardiopulmonary abnormality is identified."
    sentences = [FINDINGS[name] for name in positive]
    return " ".join(sentences)

# Predicted probabilities from a (hypothetical, not trained here) classifier
# for one chest X-ray.
scan_1 = {"cardiomegaly": 0.82, "effusion": 0.10, "pneumonia": 0.05}
scan_2 = {"cardiomegaly": 0.48, "effusion": 0.51, "pneumonia": 0.30}

print("scan 1:", build_report(scan_1))
print("scan 2:", build_report(scan_2))
print()

# The fragile part: nudge one probability by 0.02 and the report flips.
scan_2_nudged = {"cardiomegaly": 0.48, "effusion": 0.49, "pneumonia": 0.30}
print("scan 2, effusion probability 0.02 lower:", build_report(scan_2_nudged))
Output
scan 1: The cardiac silhouette is enlarged, consistent with cardiomegaly.
scan 2: There is blunting of the costophrenic angle suggesting pleural effusion.

scan 2, effusion probability 0.02 lower: No acute cardiopulmonary abnormality is identified.

What actually happened

Nudging one probability by 0.02 — from 0.51 to 0.49 — completely flips the generated report, from mentioning a specific finding to declaring the scan entirely normal. Nothing about the underlying image changed. Only which side of an arbitrary threshold one number landed on changed.

This template system is fragile in an easy-to-see way: you can inspect the exact numbers, and see exactly why the report changed. A real, free-text-generating model has this same underlying instability, but hidden — a small shift inside the model can flip fluent, confident-sounding language, with no probability number visible to check.

Line by line, the parts that are not obvious:

  • build_report treats every finding independently, at a single shared threshold — real systems calibrate thresholds per finding, since different findings need different sensitivity/specificity trade-offs.
  • The "No acute cardiopulmonary abnormality" fallback matters. Systems without a clear negative-finding template can produce oddly empty or nonsensical text when nothing crosses the threshold.
  • This system never touches free text generation. Real report generators use a trained decoder that can write any sentence, which is far more flexible, and far harder to constrain to only true statements.

Common mistakes

Treating a fluent-sounding report as automatically accurate. Language fluency and factual correctness are produced by different parts of a generation system, and fluency gives no guarantee about correctness.

Evaluating a generated report with text-similarity metrics alone. A report can score well on BLEU or ROUGE against a reference report, while still stating a clinically wrong finding — text-overlap metrics do not capture factual correctness, a gap covered in more depth in the researcher block below.

Deploying a report generator without a defined human review step. Every real clinical use of this technology keeps a qualified radiologist in the loop, reviewing and signing off on generated text before it reaches a patient record.

Try it yourself

Add a "pulmonary_edema" finding, with its own template sentence. Run both scans through the updated FINDINGS dictionary, and set its probability slightly above and slightly below 0.5, to see the same threshold-flip behaviour on a new finding.

What to learn next

  • Image captioning — the general technique this lesson's template stands in for.
  • Hallucination — the deeper risk a real, free-text-generating version of this system faces.
  • Vision-language models — the actual architecture family used in production report generators.

Researcher — Mathematics and papers.

Architectures

Modern radiology report generation typically follows an encoder-decoder pattern: a vision encoder (a CNN or vision transformer, often pretrained on chest X-ray classification) produces image features, and a language decoder (an RNN, or increasingly a transformer) generates the report text conditioned on those features, optionally with cross-attention over spatial image regions per generated word.

R2Gen (Chen et al., 2020) introduced a memory-driven transformer decoder specifically for this task, using a relational memory module to encode generation patterns common across radiology reports — addressing the observation that radiology reports have far more repetitive structure than general image captions, which a generic captioning architecture does not exploit.

Evaluation beyond text overlap

Standard captioning metrics (BLEU, ROUGE, CIDEr) measure n-gram overlap with a reference report, and correlate poorly with clinical correctness — a report can achieve a high BLEU score while omitting or inventing a clinically significant finding.

CheXbert (Smit et al., 2020) is a labeller trained to extract structured clinical findings (14 conditions, each positive/negative/uncertain) from free-text reports, enabling a clinical efficacy (CE) metric: extract structured labels from both the generated and reference report, then compute precision/recall/F1 over those labels rather than over raw text:

CE-F1 = F1( labels_extracted(generated_report), labels_extracted(reference_report) )

RadGraph (Jain et al., 2021) goes further, extracting a full graph of clinical entities and relations from report text, enabling RadGraph F1, which compares generated and reference reports at the level of individual clinical facts and their relationships, not only presence or absence of a fixed set of conditions.

Documented factual inconsistency

Miura et al. (2021) documented that models optimised purely for standard text-overlap metrics (like BLEU or ROUGE) frequently produce reports with a substantial rate of factual inconsistency against the source image — fluent, well-structured sentences that do not match what is actually shown. This finding directly motivated the shift toward clinical-efficacy-based evaluation described above, and remains a central open concern in the field, not one considered fully resolved by any current architecture.

Cost

Training a competitive report-generation model typically requires a large paired image-report corpus — MIMIC-CXR (Johnson et al., 2019) with roughly 227,000 studies is the standard public benchmark — plus substantial compute for the vision-language joint training. Inference cost is dominated by the decoder's autoregressive generation, following the same token-by-token cost structure covered generally in how text is generated.

Key references

  • Chen, Z. et al. (2020). Generating Radiology Reports via Memory-driven Transformer. EMNLP. Introduces R2Gen.
  • Smit, A. et al. (2020). CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. EMNLP.
  • Jain, S. et al. (2021). RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. NeurIPS Datasets and Benchmarks.
  • Miura, Y. et al. (2021). Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation. NAACL.
  • Johnson, A. et al. (2019). MIMIC-CXR: A large publicly available database of labeled chest radiographs. Scientific Data 6.

Current state

Clinical-efficacy-based evaluation (CheXbert, RadGraph F1) is now standard practice in published report-generation research, largely displacing pure text-overlap metrics as the primary reported result. No current system is validated for autonomous, unsupervised clinical use; every serious deployment path in the published literature keeps a radiologist reviewing and editing generated drafts, consistent with the human-in-the-loop principle covered in validating a model before it touches patients.

What to learn next