Learn AI Create the Future — with Pranay
Main site

Evaluating Text Systems

Using an LLM to grade text

LLM-as-a-judge asks a language model to grade another model's output, which handles open-ended answers but inherits the judge's own blind spots.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

LLM-as-a-judge means asking one language model to grade the answers written by another.

Picture a school essay question with no single correct answer. A teacher cannot use an answer key. She reads the essay, uses her judgement, and gives it a mark.

Chatbot answers are like that essay. There is rarely one exact correct wording. LLM-as-a-judge hires an AI model to play the role of that teacher.

Why it exists

Every metric before this one needed something to compare against: a reference translation, a reference summary, an exact answer. Modern chatbots answer open-ended questions that have no single reference at all.

Human graders can judge open-ended answers well. They are slow and costly at scale. Grading ten thousand chatbot replies by hand is not realistic for a weekly product update.

An LLM judge is the middle ground. It is faster and cheaper than a human panel, and more flexible than a fixed answer key.

How it works

  Question:  "What is the capital of France?"
  Answer:    "The capital of France is Paris."

  Judge prompt:
    "Here is a question and an answer.
     Is the answer correct? Reply Yes or No."

  Judge model reads both, then replies:  "Yes"

The judge model is asked to compare the question, the answer, and sometimes a reference answer. It then produces a score or a verdict. The same trick that makes chatbots useful for writing gets reused for grading.

Where you have already seen it

  • Chatbot leaderboards. Many public chatbot rankings today use a strong model to judge which of two answers is better, instead of a fixed test.
  • AI product testing. Companies shipping a chatbot feature often use an LLM judge to catch quality drops before real users see them.
  • Content moderation. Some moderation pipelines ask a model to judge whether a piece of text violates a stated policy.

Remember this

  • LLM-as-a-judge asks one model to grade another's open-ended answers.
  • It works where there is no single correct answer to compare against.
  • A weak or small judge model can be an unreliable grader, exactly like a careless human grader.

What to learn next

Developer — Code and libraries.

This example is intentionally honest about a common failure: a small model makes an unreliable judge. Watch it fail, so you know what to check for in your own pipeline.

Setup

bash
pip install transformers torch

Asking a tiny model to judge two answers

judge_demo.py
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

def ask_judge(question, answer):
    prompt = [{"role": "user", "content": (
        f"Question: {question}\nProposed answer: {answer}\n"
        "Is the proposed answer factually correct? Reply with only Yes or No."
    )}]
    inputs = tok.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
    out = model.generate(**inputs, max_new_tokens=5, do_sample=False)
    return tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

question = "What is the capital of France?"
good_answer = "The capital of France is Paris."
bad_answer = "The capital of France is Berlin."

print("good answer, judge says:", repr(ask_judge(question, good_answer)))
print("bad answer,  judge says:", repr(ask_judge(question, bad_answer)))

This model is small (135 million parameters), around 270MB, and runs on CPU in a few seconds.

Output
good answer, judge says: 'Yes. The proposed answer'
bad answer,  judge says: 'Yes.'

Read that output carefully. The judge said "Yes" to both the correct answer and the wrong one. A 135-million-parameter model does not know enough about the world to reliably check a fact. This is a real result, not a contrived one, and it is the single most important thing to understand about this technique.

Line by line

apply_chat_template formats the prompt the way this specific model was trained to expect chat messages, with role markers the model recognises. Skipping this step, and feeding raw text instead, noticeably degrades instruction-following in chat-tuned models.

max_new_tokens=5 keeps the judge's reply short, since we only need a one-word verdict. A judge asked to also explain its reasoning would need a longer limit.

Common mistakes

Using a small or weak model as the judge. As shown above, judge quality is bounded by the judge model's own knowledge and reasoning. Production LLM-judge setups use frontier-scale models, precisely because a weak judge is worse than no automated check at all.

Trusting a judge's score without spot-checking it against humans. An LLM judge can carry its own consistent biases, such as preferring longer answers regardless of correctness. Always validate a new judge setup against a small batch of human-graded examples first.

Letting the same model judge its own output. A model tends to rate its own writing style favourably. Comparisons should generally use a different, ideally stronger, model as the judge.

Try it yourself

Rewrite the prompt to ask for a reason before the verdict: "First explain your reasoning in one sentence, then reply Yes or No." Rerun the script with a higher max_new_tokens.

Forcing an explanation first often improves smaller models' judgements, since it gives the model a chance to work through the fact rather than pattern-matching straight to an answer. Whether it fixes this particular model's mistake is worth checking for yourself.

What to learn next

Researcher — Mathematics and papers.

The evaluation setup

Two common protocols dominate practice:

Pointwise scoring: the judge assigns a single answer a score, typically on a fixed scale.

text
score = J(question, answer, [reference])

Pairwise comparison: the judge sees two answers and picks the better one, or declares a tie.

text
verdict = J(question, answer_A, answer_B) in {A, B, tie}

Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, found pairwise comparison correlates with human preference more reliably than pointwise scoring. A relative judgement is generally an easier task for a model than producing a well-calibrated absolute number.

Documented biases

The same paper, along with follow-up work, documented specific, reproducible judge biases:

  • Position bias: judges favour whichever answer appears first in the prompt, in a meaningful fraction of cases. The standard mitigation is running the comparison twice, with the order swapped, and only counting agreement.
  • Verbosity bias: judges tend to prefer longer answers, independent of actual quality.
  • Self-enhancement bias: a model judging its own family's outputs rates them more favourably than an independent judge would.

Wang et al. (2023), Large Language Models are not Fair Evaluators, quantified position bias directly, showing consistency rates between forward and reversed orderings well below what a fair judge should produce.

Calibrating against humans

The standard validation step computes agreement between the LLM judge and a human-labelled reference set. This uses the same inter-annotator agreement statistics covered in the next lesson, most often Cohen's kappa or a correlation coefficient against average human scores.

Zheng et al. (2023) reported GPT-4 as a judge reaching agreement with human majority vote in the same range as agreement between two individual human annotators, on their benchmark. This is the headline result that popularised the technique, and the comparison that should be reproduced on any new judging task before trusting it.

Complexity and cost

Unlike every metric earlier in this section, an LLM judge's cost scales with the judge model's own size. Pairwise comparison roughly doubles the forward passes of pointwise scoring, since each ordering is usually evaluated separately to control for position bias.

Key references

  • Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685
  • Wang, P. et al. (2023). Large Language Models are not Fair Evaluators. arXiv:2305.17926
  • Liu, Y. et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv:2303.16634

Current state and open problems

LLM-as-a-judge is now standard practice for evaluating open-ended generation, chatbot comparisons and agent trajectories, largely displacing older overlap-based metrics for these use cases specifically.

The open problem is a version of the one every metric in this section eventually hits. The judge's own reliability needs its own evaluation, which still traces back to human judgement somewhere in the loop. Fully automating evaluation, with no human baseline ever again, is not yet solved. Most serious practitioners still re-check their judge against fresh human ratings periodically, as models and use cases drift.

What to learn next