Learn AI Create the Future — with Pranay
Main site

Summarisation

When a summary says something the source never said

A faithful summary only states what the source document actually said. Checking this automatically means comparing summary claims against the source with a separate model.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A faithful summary only states things the source document actually said, nothing invented and nothing changed.

Picture a friend retelling a story they heard secondhand. They get the gist right. But one number comes out wrong: "she waited two hours" becomes "she waited all day" in the retelling.

The story still sounds coherent. It is still wrong. A summary can fail the exact same way: fluent, confident, and quietly untrue to its source.

Why it exists

Abstractive summarisation writes new sentences instead of copying old ones. That freedom is the whole point, and it is also the risk.

Nothing forces a generative model to stick to only what the source said. It was trained to produce fluent, plausible-sounding text, not to pass a fact-check. Fluent and true are two different goals, and a model can hit one without the other.

This matters most in exactly the situations where summaries get used: medical notes, legal documents, financial reports. A single invented number can cause real harm, even inside an otherwise excellent summary.

How it works

Source: "The startup raised 4 million dollars, led by two angel investors."
                            |
                            v
Summary A: "The startup raised 4 million dollars to hire engineers."
   -> Faithful. Every claim traces back to the source.

Summary B: "The startup raised 40 million dollars in venture funding."
   -> Unfaithful. The number and the funding type are both wrong.

A faithfulness checker compares a summary sentence against the source. It asks one question: does the source support this claim, contradict it, or say nothing about it at all?

Where you have already seen it

  • "This may be inaccurate" warnings under AI summaries. A direct, honest acknowledgement of exactly this risk.
  • News organisations that fact-check AI-drafted summaries before publishing. A human faithfulness check, done manually.
  • Legal AI tools that highlight which sentence in the source backs each summary claim. Faithfulness made visible, not only assumed.
  • Health apps that show the exact line from your report a summary is based on. Letting you verify faithfulness yourself.

Remember this

  • A faithful summary states only what the source says. It never adds or changes facts.
  • Fluent, confident-sounding text can still be unfaithful. The two are unrelated qualities.
  • Faithfulness gets checked by comparing each summary claim back against the source, automatically or by a human.

What to learn next

Developer — Code and libraries.

The trick: reuse a natural language inference (NLI) model. NLI models are trained to answer one question. Given a premise, does a hypothesis follow from it, contradict it, or say something unrelated?

Treat the source as the premise, and each summary sentence as the hypothesis.

Setup

bash
pip install transformers torch

Checking two summaries against their source

faithfulness.py
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "typeform/distilbert-base-uncased-mnli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

source = (
    "The startup raised 4 million dollars in its seed round, led by two angel "
    "investors from Bengaluru. The founders plan to use the money to hire engineers."
)

# One faithful summary, and one that invents a number the source never gave.
summaries = [
    "The startup raised 4 million dollars to hire more engineers.",
    "The startup raised 40 million dollars in venture funding.",
]

for summary in summaries:
    inputs = tokenizer(source, summary, return_tensors="pt", truncation=True)
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]
    best = probs.argmax().item()
    label = model.config.id2label[best]
    print(f"{label:13s} ({probs[best]:.3f})  {summary}")
Output
ENTAILMENT    (0.992)  The startup raised 4 million dollars to hire more engineers.
CONTRADICTION (0.999)  The startup raised 40 million dollars in venture funding.

Line by line

tokenizer(source, summary, ...) feeds both texts to the model together, as a premise-hypothesis pair. This is the standard NLI input format. The model reads both texts, not only one.

model.config.id2label looks up what each output number actually means. Different NLI models order their labels differently. Always read this mapping from the model itself. Never hardcode a guessed order.

The faithful summary scores ENTAILMENT at 0.992. The source supports every claim it makes. Note it changed the wording, "to hire more engineers" instead of the source's exact phrasing. Faithfulness is about facts, not exact words.

The unfaithful summary scores CONTRADICTION at 0.999. The model was never told the real number was wrong. It compared two texts and found a direct numerical mismatch, 4 million against 40 million, on its own.

Common mistakes

Checking the whole summary against the whole source in one go. Long summaries make multiple separate claims. Checking sentence by sentence catches a single false claim. Checking the whole block at once can let one bad sentence hide among several true ones.

Treating NEUTRAL the same as CONTRADICTION. NEUTRAL means the source neither confirms nor denies the claim, often because the summary added outside knowledge. That is a real faithfulness problem too, only a different one from an outright contradiction.

Assuming a high entailment score means the summary is good. Faithfulness is necessary, not sufficient. A summary can be perfectly faithful and still miss the most important point in the source.

Using an NLI model trained on short, simple sentences to judge long, technical summaries. Model performance drops on inputs unlike its training data. Always spot-check faithfulness scores by hand before trusting them at scale.

Try it yourself

Write a summary that is faithful but incomplete, mentioning only the funding amount and leaving out the hiring plan entirely. Check whether it still scores ENTAILMENT. It should. Faithfulness never penalises leaving something out, only adding something false.

What to learn next

Researcher — Mathematics and papers.

Faithfulness as a formal property

A summary y is faithful to source x if every atomic claim c extractable from y is entailed by x. Formally: x |= c for all c in claims(y). This is distinct from relevance (does y cover x's important content) and from fluency (is y well-formed English). A summary can score highly on any one axis while failing badly on another.

Kryscinski et al. (2020) report that even strong pre-2020 abstractive systems produced unfaithful summaries on 25-30% of CNN/DailyMail outputs. That establishes faithfulness as a first-order evaluation axis, not a rare edge case.

Taxonomy of unfaithfulness

Maynez et al. (2020) distinguish two failure types, both worth naming precisely:

Intrinsic hallucination. The summary misuses information already present in the source. It might swap which entity performed an action, or alter a stated number. The Summary B example in the developer block is intrinsic: 4 million became 40 million, a real source fact, misstated.

Extrinsic hallucination. The summary introduces information absent from the source entirely, whether or not it happens to be true elsewhere. This is harder to catch automatically. Nothing in x contradicts it — the claim is only unsupported. An NLI model correctly reports that as NEUTRAL, not CONTRADICTION.

Automatic faithfulness metrics

NLI-based scoring, as used in the developer block, treats each summary sentence as a hypothesis against the source as premise. FactCC (Kryscinski et al., 2020) trains a purpose-built classifier on synthetically corrupted summaries, instead of an off-the-shelf NLI model. This improves precision on the task.

QA-based scoring takes a different approach. Generate questions from the summary. Answer them using the summary and the source independently, then compare the two answer sets. QAGS (Wang, Cho & Lewis, 2020) formalises this approach. A large disagreement between summary-grounded and source-grounded answers signals unfaithfulness, with no trained entailment model needed at all.

LLM-as-a-judge, covered in using an LLM to grade text, increasingly replaces both. A strong LLM can be prompted to check each claim against the source and explain its reasoning. This correlates well with human faithfulness judgements. The cost: slower and more expensive than a small NLI classifier.

Why fluency and faithfulness diverge during training

Standard training, covered in abstractive summarisation, optimises token-level likelihood against reference summaries. No term in that loss directly penalises an unsupported claim. A fluent hallucination and a fluent faithful sentence can carry near-identical loss, if both are locally plausible continuations. This is the core reason faithfulness is checked after generation, rather than trained away by better data alone.

Reference summaries themselves are not always perfectly faithful. Reference summaries in widely used datasets, including CNN/DailyMail, sometimes add background knowledge the source article never stated. This teaches models a subtle, correlated bad habit.

Key references

  • Maynez, J. et al. (2020). On Faithfulness and Factuality in Abstractive Summarization. arXiv:2005.00661
  • Kryscinski, W. et al. (2020). Evaluating the Factual Consistency of Abstractive Text Summarization. arXiv:1910.12840 — FactCC.
  • Wang, A., Cho, K. & Lewis, M. (2020). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. arXiv:2004.04228 — QAGS.

Current state and open problems

Faithfulness checking has moved from an academic evaluation concern to a production requirement. In medical, legal and financial applications, an unsupported claim is a liability, not only a quality issue.

No current automatic metric reliably catches every failure type at low cost. NLI-based methods handle intrinsic contradiction well but struggle with extrinsic additions. QA-based methods are more thorough but expensive to run at scale. LLM-as-a-judge is the most flexible. But it inherits its own faithfulness problem one level up. Nothing guarantees the judge model reasons faithfully, rather than pattern-matching plausibility.

What to learn next