Learn AI Create the Future — with Pranay
Main site

Messy Real-World Text

Labelling data with an LLM and a human reviewer

An LLM can label most of a dataset automatically, as long as low-confidence cases get routed to a human instead of guessed at silently.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

LLM-assisted annotation uses a language model to label most of a dataset automatically. Cases it is unsure about go to a human for a final decision.

Think about a senior doctor and a junior resident reviewing X-rays together. The resident looks at every scan first. Clear, obvious cases get signed off immediately. Anything uncertain gets set aside for the senior doctor to look at directly.

That division of labour — confident cases handled automatically, uncertain ones escalated — is exactly how LLM-assisted annotation works. The model plays the resident's role.

Why it exists

Labelling a large dataset by hand takes enormous human time. Someone has to read every ticket, every review, every comment, and assign it a category. A model can label most examples in a fraction of that time.

A model left unsupervised on its own labelling is risky, though. It can be confidently wrong, and confident mistakes silently poison a training set. The fix is not choosing between "all human" and "all model." It is combining them. The model handles clear cases. A human reviews the cases the model itself is unsure about.

How it works

  unlabelled example
          |
          v
     model scores it against each possible label
          |
          v
   confident?  ---- yes ---->  auto-label, move on
          |
          no
          |
          v
   send to a human reviewer

Where you have already seen it

  • Content moderation queues, where a model auto-clears unambiguously safe posts and routes uncertain ones to a human moderator.
  • Customer support ticket routing, auto-tagging clear-cut categories and escalating ambiguous tickets.
  • Building training data for a new classifier, where a model labels first and a human spot-checks or corrects it.

Remember this

  • LLM-assisted annotation splits labelling work between a model and a human, by confidence. Neither one handles everything alone.
  • A confidence threshold decides which cases the model handles, and which get escalated. That threshold is a real, tunable choice.
  • The model's confidence is not the same as correctness. A badly-calibrated model can be confidently wrong as easily as it can be cautious.

What to learn next

  • Augmenting text data — another way to grow a labelled dataset, by reworking existing examples instead of labelling new ones.
  • Prompt engineering — writing the instructions that shape how well a model labels data in the first place.
  • Text classification — the general task this labelling workflow is usually feeding data into.

Developer — Code and libraries.

Calling a hosted LLM API needs a paid key, which this lesson cannot assume you have. This demo uses a small, free, local embedding model to stand in for the "model labels it" step — the confidence-threshold pattern shown is identical to how a real LLM-based pipeline routes work.

Setup

bash
pip install sentence-transformers numpy

The first run downloads sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, about 460 MB, cached after that.

Auto-labelling by similarity, routing low-confidence cases to a human

assisted_annotation_demo.py
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

# One prototype sentence per label. A real pipeline would prompt an LLM with
# the label definitions instead; this stand-in needs no API key, so it is
# free to run here.
labels = {
    "complaint": "This product is terrible and broke after one day.",
    "praise": "I love this product, it works perfectly.",
    "question": "Does this product come with a warranty?",
}
label_names = list(labels.keys())
label_emb = model.encode(list(labels.values()), normalize_embeddings=True)

unlabelled = [
    "This stopped working within a week, total waste of money.",
    "Absolutely fantastic, exceeded my expectations.",
    "Can I return this if it does not fit?",
    "It is okay I guess, does the job.",
]

CONFIDENCE_THRESHOLD = 0.35

item_emb = model.encode(unlabelled, normalize_embeddings=True)
scores = item_emb @ label_emb.T

for text, row in zip(unlabelled, scores):
    best_i = int(np.argmax(row))
    best_score = float(row[best_i])
    route = "auto-label" if best_score >= CONFIDENCE_THRESHOLD else "send to human"
    print(f"{best_score:.3f}  {label_names[best_i]:10} [{route}]  {text}")
Output
0.507  complaint  [auto-label]  This stopped working within a week, total waste of money.
0.359  praise     [auto-label]  Absolutely fantastic, exceeded my expectations.
0.254  question   [send to human]  Can I return this if it does not fit?
0.304  praise     [send to human]  It is okay I guess, does the job.

Line by line

Each label is represented by one prototype sentence, not a word. "This product is terrible and broke after one day" carries far more usable signal about what "complaint" means than the bare word "complaint" would, since the model is comparing meaning, not matching a keyword.

best_score is the model's confidence — how close the item sits to its nearest label prototype. Two of the four items clear the 0.35 threshold and get auto-labelled; two do not, and get routed to a human instead.

Look closely at the two routed-to-human cases. "Can I return this if it does not fit?" is genuinely a question, correctly identified as the closest label by rank, but at a low enough score that the system does not trust that guess unsupervised. "It is okay I guess, does the job" is a real edge case — flat, ambiguous sentiment that even a human reviewer might find hard to place confidently. The threshold is doing its job in both cases: catching a low-confidence-but-correct guess and a genuinely ambiguous item alike.

Common mistakes

Setting the confidence threshold too low. Every example clears a low enough bar, meaning the model auto-labels everything — including cases it should not be trusted on, silently defeating the entire point of having a threshold at all.

Setting the threshold too high. Nothing clears the bar, every example goes to a human, and the model provides no time savings at all — the opposite failure, equally real in practice.

Never re-checking the auto-labelled set. Even confident labels are not guaranteed correct. Production pipelines routinely sample a percentage of auto-labelled examples for human spot-checking, not only the ones that failed the threshold.

Try it yourself

Add a fifth, genuinely ambiguous example — a sentence that could reasonably belong to two different labels — and watch where it lands relative to the threshold. Then try lowering CONFIDENCE_THRESHOLD to 0.25 and rerun: check which additional item gets auto-labelled, and decide for yourself whether you would trust that specific label without a human checking it.

What to learn next

Researcher — Mathematics and papers.

Human-in-the-loop annotation as a cost-quality optimisation

Given an unlabelled pool U, a labelling function M (an LLM or other model) with a confidence score c(x) for each item x, and a human labelling cost k per item, the routing decision partitions U into an auto-labelled set and a human-reviewed set:

text
auto_label(x)  if  c(x) >= tau
human_review(x)  otherwise
  • tau is the confidence threshold, the single free parameter controlling the cost-quality trade-off directly: raising tau increases human labelling cost and typically increases overall label accuracy; lowering it decreases both.
  • Total expected cost is |{x : c(x) < tau}| * k, and total expected error rate is dominated by the auto-labelled set's error rate, which is itself a function of how well-calibrated c(x) actually is — see below.

Choosing tau in practice requires a small held-out set with ground-truth labels, used to measure the auto-labelled subset's real accuracy at each candidate threshold and pick the point meeting a required quality bar, rather than choosing tau by intuition alone as the developer demo does for illustration.

Calibration: why confidence must be validated, not trusted

A model's confidence score is only useful for routing if it is calibrated — meaning that among all items the model labels with confidence around 0.7, roughly 70% actually are correct. Modern large language models are frequently overconfident, especially when prompted directly for a numeric confidence score in natural language rather than using an internal probability (Xiong et al., 2023, benchmark LLM confidence calibration directly and find systematic overconfidence across several elicitation methods).

This matters directly for the auto-label/human-review split: an uncalibrated confidence score can pass low-quality labels through the "confident" path as easily as a well-calibrated one, defeating the entire purpose of thresholding. Production systems validate calibration empirically against held-out labelled data before trusting a threshold in the way the developer demo trusts 0.35 for illustration.

LLM-as-annotator versus LLM-as-judge

Using an LLM to produce the initial label (annotation) and using a separate LLM call to evaluate a label's quality (the "LLM-as-judge" pattern) are related but distinct uses. Annotation quality is commonly improved by combining both: one pass labels, a second independent pass (or the same model with a different, evaluation-framed prompt) scores confidence or flags disagreement, since a model critiquing an answer sometimes catches errors the same model missed while producing that answer directly.

Key references

  • Zhang, R., Li, Y., Ma, Y., Zhou, M. & Zou, J. (2023). LLMaAA: Making Large Language Models as Active Annotators. EMNLP Findings. arXiv:2310.19596 — LLM-driven annotation combined with active learning to prioritise which examples most need human review.
  • Xiong, M. et al. (2023). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv:2306.13063
  • Wang, S. et al. (2021). Want To Reduce Labeling Cost? GPT-3 Can Help. EMNLP Findings. — an early direct study of LLM-generated labels substituting for human annotation, including where it fails.
  • Gilardi, F., Alizadeh, M. & Kubli, M. (2023). ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. PNAS. arXiv:2303.15056 — a widely-cited empirical comparison, with important caveats about task type and domain that later work has qualified.

Current state and open problems

LLM-assisted annotation with human review of low-confidence cases is now standard practice for building new labelled datasets at reasonable cost, and has meaningfully lowered the barrier to starting a labelling project that previously needed a large crowdworker budget.

The unresolved risk is systematic bias in what the model gets confidently wrong, rather than randomly wrong. If a model is confidently mistaken in a consistent direction — under-flagging a particular dialect's complaints as neutral, for instance — a naive confidence threshold will not catch that error, since the model expresses high confidence exactly where it is systematically biased, not where it is randomly uncertain. Auditing auto-labelled data for this kind of systematic, confidently-wrong pattern, rather than only spot-checking overall accuracy, remains a manual, unautomated safeguard in most current pipelines.

What to learn next