Learn AI Create the Future — with Pranay
Main site

Generative AI

What is a large language model?

A large language model is a program that guesses the next piece of text, over and over, until an answer appears.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works, in one picture
  5. What the word "large" is doing there
  6. Where you have already met one
  7. Honest limits, so you are not caught out
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A large language model is a computer program that guesses the next piece of text.

That is the whole trick. It guesses one piece, sticks it on the end, and guesses again.

The analogy you have already lived

Open WhatsApp and start typing "I am on my". Your keyboard offers "way" before you finish. Nobody sat down and wrote a rule for that exact sentence. Your keyboard watched millions of messages and learned which word tends to follow.

A large language model is that keyboard, grown enormous. The short name is LLM. It means a program trained on a giant pile of text. Your phone looks back at two or three words. An LLM looks back at the whole conversation, sometimes a book's worth.

That is the only real difference. Same idea, much bigger memory, much better guessing.

Why it exists

Before LLMs, talking to a computer meant writing rules by hand. Somebody had to write "if the message contains the word balance, reply with the account balance."

Language destroys that approach in about five minutes. People misspell things. They mix Hindi and English in one sentence. They are sarcastic. They ask questions nobody planned for.

So researchers changed the question they were asking. Instead of "what are the rules of English?", they asked "given the text so far, what usually comes next?"

That second question can be answered from examples alone. And examples of written text are the one thing the internet has in unlimited supply.

How it works, in one picture

You type:   "The capital of France is"
                 |
                 v
         [ language model ]
                 |
                 v
   a scored list of every word it knows:
         " Paris"    very high
         " a"        low
         " home"     low
         " banana"   almost nothing
                 |
                 v
   it picks " Paris" and sticks it on the end
                 |
                 v
   "The capital of France is Paris"  ->  and now it guesses again

Read that loop twice. The model does not plan the whole answer in advance. It produces one piece, then treats its own output as fresh input, then produces the next piece.

Long, sensible paragraphs come out of that tiny step repeated hundreds of times. This surprises almost everyone the first time they hear it. If it feels strange, that is normal — it is genuinely strange.

What the word "large" is doing there

Three things are large, and all three matter.

  • The training text. Trillions of words, taken from books, websites, code and conversations.
  • The parameters. These are the internal numbers the model tunes while learning. A modern model has billions of them.
  • The bill. Training one costs millions of dollars in electricity and hardware. That is why very few organisations train them from scratch.

Where you have already met one

  • ChatGPT, Claude, Gemini — an LLM with a chat window wrapped around it.
  • Gmail's Smart Compose — the grey text finishing your sentence.
  • GitHub Copilot — the same idea, trained heavily on code.
  • Google Translate — modern versions are built on the same family of models.
  • The support chat on your bank or food delivery app.

Honest limits, so you are not caught out

  • It has no memory of you between chats, unless the app deliberately stores one.
  • It can state a wrong fact with total confidence. This is called hallucination — inventing something that sounds correct but is not.
  • It does not know today's news unless the news is handed to it in the message.
  • It cannot do anything outside the chat window unless tools are connected to it.

None of these are bugs that get patched next week. They come from what the model actually is: a very good guesser of text.

Remember this

  • An LLM predicts the next piece of text, then repeats that step.
  • It learned from examples, not from rules a person wrote down.
  • A confident tone is not evidence that the answer is correct.

What to learn next

Developer — Code and libraries.

Setup

The first example needs nothing but Python 3. The second one needs two libraries:

bash
pip install transformers torch

A language model you can read in one screen

Before touching a real model, build the smallest possible one. It counts which word follows which, then predicts the most common follower.

tiny_lm.py
# A tiny language model. It learns which word tends to follow which word.
from collections import defaultdict, Counter

corpus = """the cat sat on the mat
the cat sat on the rug
the cat drank the milk
the dog sat on the mat
the dog barked at the cat"""

following = defaultdict(Counter)
for line in corpus.strip().split("\n"):
    words = line.split()
    for first, second in zip(words, words[1:]):   # every neighbouring pair
        following[first][second] += 1

def predict(word):
    return following[word].most_common(1)[0][0]

print("after 'the' ->", predict("the"))
print("after 'cat' ->", predict("cat"))
print("after 'sat' ->", predict("sat"))

# Feed each guess back in as input. This loop is what a real LLM does.
word, out = "the", ["the"]
for _ in range(5):
    word = predict(word)
    out.append(word)
print("generated   ->", " ".join(out))
Output
after 'the' -> cat
after 'cat' -> sat
after 'sat' -> on
generated   -> the cat sat on the cat

Line-by-line, for the parts that trip people up

zip(words, words[1:]) walks the sentence in overlapping pairs. For ["the", "cat", "sat"] it yields ("the", "cat") then ("cat", "sat"). That is the training data: a word, and the word that followed it.

following is a dictionary of counters. following["the"] ends up as Counter({'cat': 4, 'mat': 2, 'dog': 2, 'rug': 1, 'milk': 1}). Prediction is then a lookup of the largest count.

The generation loop is the important part. Notice what came out: the cat sat on the cat. It starts sensible and then eats its own tail. That failure is not a bug in the code. It is the honest consequence of looking back at exactly one word — the model has no idea that "cat" already appeared.

Real LLMs fix this by looking at the whole context, meaning every token you have written so far, instead of only the previous one. That single change is most of the story, and the next lesson covers it.

Now the same idea, with a real model

distilgpt2 is a small public model, roughly 350 MB. It runs on a plain laptop CPU in a few seconds. The first run downloads the weights and prints progress bars, and you may see a harmless warning about pad tokens.

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

name = "distilgpt2"                      # small enough for any laptop, no GPU needed
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

batch = tok("The cat sat on the", return_tensors="pt")
with torch.no_grad():                    # we are not training, so skip gradient bookkeeping
    logits = model(**batch).logits

print("pieces :", tok.convert_ids_to_tokens(batch["input_ids"][0]))
print("shape  :", tuple(logits.shape))

probs = torch.softmax(logits[0, -1], dim=-1)   # scores for the word after "the"
print("choices:", probs.shape[0])
print("adds to:", round(float(probs.sum()), 3))
Output
pieces : ['The', 'Ġcat', 'Ġsat', 'Ġon', 'Ġthe']
shape  : (1, 5, 50257)
choices: 50257
adds to: 1.0

Three things to take from that output.

The Ġ character is not corruption. It is how this tokenizer marks "there was a space before this piece". So Ġcat means " cat" with its leading space attached. Pieces carry their own spacing.

The shape (1, 5, 50257) reads as: one input, five pieces of text, and a score for each of 50257 possible next pieces. The model scores its entire vocabulary — the full list of pieces it knows — at every single step.

Those 50257 scores become probabilities that add up to one. Choosing from that list is a separate decision, covered in temperature and sampling.

Common mistakes

Calling predict() on a word the model never saw. following["zebra"] creates an empty counter, and .most_common(1)[0] raises IndexError: list index out of range. Add a fallback:

python
def predict(word):
    guesses = following[word].most_common(1)
    return guesses[0][0] if guesses else "the"

Assuming one token equals one word. It does not. Rare words, names and numbers get split into several pieces, and API pricing is per token. Measure with the real tokenizer instead of counting words.

Forgetting torch.no_grad(). Without it, PyTorch stores everything needed for a backward pass. On a small model you waste memory. On a large one you run out of it.

Reading model confidence as truth. A high probability means "this text pattern is common", not "this statement is factually correct". Those are different things, and the model cannot tell them apart.

Try it yourself

Change predict to return the top three followers using most_common(3), then pick randomly among them. Watch the generated sentence stop looping and start varying. You have now built sampling, which is exactly what the temperature setting controls in a real model.

What to learn next

Researcher — Mathematics and papers.

The formal object

An autoregressive language model defines a distribution over token sequences using the chain rule of probability:

p_θ(x_1 … x_T)  =  ∏  p_θ( x_t | x_<t )
                  t=1..T

x_t is the token at position t. T is the sequence length. x_<t denotes every token before position t. θ is the full parameter vector. Nothing further is assumed.

That is worth sitting with. The model is not a knowledge base and not a reasoner by construction. It is a conditional distribution over the next symbol. Every capability people observe is an emergent property of fitting that distribution well on a very large corpus.

Training objective

Training minimises the mean negative log-likelihood, which is cross-entropy against the one-hot next token:

L(θ)  =  −(1/T) · ∑  log p_θ( x_t | x_<t )
                 t=1..T

L is measured in nats per token when the logarithm is natural. Perplexity is its exponential:

PPL  =  exp(L)

Perplexity has a clean reading: the effective number of equally likely options the model is hesitating between at each step. A perplexity of 10 behaves like rolling a fair 10-sided die per token. It is comparable across models only when the tokenizer is identical, which is why cross-family perplexity comparisons are usually meaningless.

Compute accounting

For a dense transformer with N non-embedding parameters, the standard estimates are:

forward pass   ≈  2N  FLOPs per token
backward pass  ≈  4N  FLOPs per token
full training  ≈  6ND FLOPs        (D = number of training tokens)

Worked example with N = 7e9 and D = 1.4e12:

C  ≈  6 × 7e9 × 1.4e12  =  5.88e22 FLOPs

At an effective sustained throughput of 1.6e14 FLOP/s per accelerator — around 40 percent utilisation of a device rated at 4e14 — that is roughly 3.7e8 device-seconds, near 1e5 device-hours. For calibration, the LLaMA 1 paper reports approximately 82,000 A100-hours for its 7B model.

Scaling laws

Kaplan et al. (2020) showed loss falls as a power law in parameters, data and compute, with smooth and predictable exponents. Hoffmann et al. (2022) corrected the compute-optimal allocation: for a fixed budget C, parameters and tokens should scale in roughly equal proportion, giving D/N ≈ 20. Under the earlier prescription, most large models of 2020 to 2022 were badly undertrained relative to their size.

One objective mismatch gets forgotten in practice. Chinchilla optimality minimises training loss for a training budget. It says nothing about inference cost, which is where nearly all lifetime compute goes for a deployed model. Training a smaller model on more tokens than compute-optimal is often the correct engineering decision.

From next-token predictor to assistant

A base model trained on the objective above is not an assistant. The usual pipeline adds two stages:

  1. Supervised fine-tuning. Curated instruction and response pairs, trained with the same cross-entropy loss, masked so only response tokens contribute.
  2. Preference optimisation. Either RLHF with a learned reward model optimised by PPO (Ouyang et al., 2022), or a direct method such as DPO (Rafailov et al., 2023), which reparameterises the RLHF objective so that no separate reward model and no rollout loop are required.

Preference optimisation changes the distribution over responses, not the underlying factual content. It reliably improves helpfulness, formatting and refusal behaviour. It does not install knowledge the base model lacks, and it tends to worsen calibration — a well-documented cost of the alignment stage.

Papers, in a sensible reading order

Where the field is genuinely unsettled

Sparse mixture-of-experts architectures decouple total parameter count from per-token FLOPs, so a model can hold far more parameters than it activates. Long-context work keeps pushing sequence length upward, but attention cost and the measured degradation of retrieval accuracy inside very long contexts both remain real constraints. And the mapping from benchmark score to deployed usefulness is weak enough that careful task-specific evaluation still beats leaderboard reading.

What to learn next

Related terms

What to learn next

These follow on from what you just read.

  • Generative AI

    How LLMs actually work

    Inside an LLM the text is chopped into pieces, turned into numbers, and mixed together many times so every piece can look at the others.

  • Generative AI

    Prompt engineering

    Prompt engineering is writing your request so clearly that the model can give you something useful on the first try.

  • Generative AI

    Context windows

    The context window is the total text a model can hold at once, and exceeding it does not warn you — it either errors, silently drops your oldest messages, or cuts the reply off mid-sentence.