Learn AI Create the Future — with Pranay
Main site

Multimodal AI

Text to image

Text-to-image models start from pure noise and remove it step by step, steered at every step by what your prompt means.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. The knob you will meet first
  6. Where you have already seen it
  7. What is honestly hard here
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A text-to-image model starts with a screen of random static and cleans it up, step by step.

Your words decide what it turns into.

The analogy you have already lived

Think about an old television tuned to a dead channel. Grey static, no picture. Now imagine slowly turning a dial, and with each small turn the static settles a little more into a shape.

Nobody drew that shape. It was pulled out of the noise, one small correction at a time.

Here is the part that makes it work. Say "more like a temple" before each turn of the dial. Two hundred tiny nudges then add up to a temple.

Your prompt is that whisper, repeated at every step.

Why it exists

Earlier image generators tried to produce a whole picture in one shot. GANs did this, and they were often beautiful and often unstable. Training could collapse, and the model would produce the same handful of images forever.

Making one leap from nothing to a finished picture is a very hard problem. Splitting it into two hundred small, similar steps is a much easier one.

That is the trade the whole field made. Slower to generate, far easier to train, and far more controllable.

How it works

Training runs the process backwards, which is the piece most people find confusing.

You take a real photo and add a little noise. Then more. Then more, until nothing is left but static. At each stage, the model is asked one question: what noise was added here?

That is a simple question with a known answer, so training is straightforward.

   training:   photo  ->  add noise  ->  model guesses the noise added

   generating: static ->  model guesses noise  ->  subtract a bit  ->  repeat
                    |                                                 |
                    +------------- about 20 to 50 times --------------+

Your prompt enters through the guessing step. The model does not guess "the noise". It guesses "the noise, given that this is meant to become a photo of a temple at sunrise".

The knob you will meet first

Every tool exposes a setting called guidance, sometimes CFG scale. It controls how hard the prompt pushes.

Turn it down and the model mostly ignores you, producing something generic. Turn it up and the model follows the prompt hard. The picture then starts to look overcooked: burnt colours, harsh edges, repeated details.

There is a sweet spot in the middle, and it moves depending on the model and the prompt. That is why every guide gives a different recommended number.

Where you have already seen it

  • Image generators in chat apps and design tools.
  • Phone photo editing that removes an object and fills the gap.
  • Stock illustration made to order rather than searched for.
  • Game and film concept art, where the first rough versions are now often generated.

What is honestly hard here

Text inside pictures. Signboards and labels come out as convincing gibberish in many models. Letters are shapes to the model. Shapes that must be exactly right are a different problem from shapes that must look right.

Hands and counting. "Three cats" often gives four. The model has no counter.

Consequences. These models were trained on images made by people who were not asked. They can imitate a living artist's style, and they can make convincing pictures of events that never happened. Those are unresolved problems, legal and ethical, not technical footnotes.

Remember this

  • Generation is noise removed step by step, not a picture drawn in one go.
  • Your prompt steers every step, not the first one.
  • Guidance controls how hard the prompt pushes, and too much of it ruins the picture.

What to learn next

  • Text to video — the same machinery with time added, and why that costs so much.
  • Diffusion models — the noise-removal process on its own terms.
  • CLIP — where the text encoder inside these models comes from.

Developer — Code and libraries.

You cannot train Stable Diffusion on a laptop. You can implement the exact sampling loop, with the exact classifier-free guidance formula. Run it on a toy distribution whose correct answer is known exactly. That is what this example does, and it demonstrates the guidance trade-off with real numbers.

Setup

bash
pip install numpy

No downloads. The whole run takes a few seconds.

Real diffusion sampling, toy data

The data here is two blobs of 2-D points. Each blob is a "prompt". The data is Gaussian, so the correct noise prediction is known exactly. That lets us skip training a network and study the sampler alone.

toy_diffusion.py
import numpy as np

# The "images" live in 2-D so you can see them. Two prompts, two places.
MEANS = {"left blob": np.array([-1.0, 0.0]), "right blob": np.array([1.0, 0.0])}
SD = 0.7                                    # how spread out each blob is

T = 200
betas = np.linspace(1e-4, 0.02, T)          # the standard DDPM noise schedule
alphas = 1.0 - betas
abar = np.cumprod(alphas)

def eps_for_mean(x, t, mu):
    """Exact noise prediction for a Gaussian blob. A real model learns this."""
    v = abar[t] * SD**2 + (1 - abar[t])     # variance of the blob after t steps of noising
    score = -(x - np.sqrt(abar[t]) * mu) / v
    return -np.sqrt(1 - abar[t]) * score

def eps_uncond(x, t):
    """No prompt: the model must cover both blobs, weighted by which is likelier."""
    v = abar[t] * SD**2 + (1 - abar[t])
    d = np.stack([((x - np.sqrt(abar[t]) * m) ** 2).sum(-1) for m in MEANS.values()])
    w = np.exp(-(d - d.min(0)) / (2 * v))
    w = w / w.sum(0)
    parts = np.stack([eps_for_mean(x, t, m) for m in MEANS.values()])
    return (w[..., None] * parts).sum(0)

def sample(prompt, guidance, n=2000, seed=1):
    r = np.random.default_rng(seed)
    x = r.normal(size=(n, 2))               # start from pure noise
    mu = MEANS[prompt]
    for t in range(T - 1, -1, -1):
        e_u = eps_uncond(x, t)
        e_c = eps_for_mean(x, t, mu)
        eps = e_u + guidance * (e_c - e_u)  # classifier-free guidance
        mean = (x - betas[t] / np.sqrt(1 - abar[t]) * eps) / np.sqrt(alphas[t])
        x = mean + (np.sqrt(betas[t]) * r.normal(size=x.shape) if t > 0 else 0.0)
    return x

for guidance in (0.0, 1.0, 3.0, 8.0):
    pts = sample("right blob", guidance)
    on_target = (pts[:, 0] > 0).mean()
    print(f"guidance {guidance:>4}  ->  {on_target*100:5.1f}% landed in the right blob, "
          f"mean x {pts[:,0].mean():+.2f}, spread {pts[:,0].std():.2f}")
Output
guidance  0.0  ->   49.0% landed in the right blob, mean x -0.01, spread 1.22
guidance  1.0  ->   90.5% landed in the right blob, mean x +0.93, spread 0.71
guidance  3.0  ->  100.0% landed in the right blob, mean x +1.38, spread 0.55
guidance  8.0  ->  100.0% landed in the right blob, mean x +1.68, spread 0.53

This output is the guidance trade-off, measured

Guidance 0 ignores the prompt entirely. Half the samples land in the wrong blob, and the spread of 1.22 covers both. This is what an unconditional model does.

Guidance 1 is plain conditional generation with no extra push. Mean x is +0.93 against a true blob centre of +1.0, and spread is 0.71 against a true spread of 0.7. The sampler reproduced the target distribution correctly. That is the number to notice: at guidance 1 the samples are right, statistically.

Guidance 3 and 8 send every sample to the correct side, and drag the mean past the true centre to +1.38 and then +1.68. The samples are no longer drawn from the data distribution. They are drawn from an over-sharpened caricature of it.

That is exactly what over-cooked, over-saturated high-CFG images are. The distribution has been distorted on purpose, and prompt adherence was bought with fidelity. There is no setting that gives both.

Line by line, the parts that are not obvious

abar is the cumulative product of 1 - beta. It says how much of the original signal survives after t noising steps. At t = 0 it is near 1; at t = 199 it is near 0.

eps_for_mean uses the fact that a Gaussian blob stays Gaussian when you add Gaussian noise. Its noised distribution has mean sqrt(abar) * mu and variance abar * SD^2 + (1 - abar). So the score — the direction of steepest increase in likelihood — is available in closed form. A trained network is estimating this exact quantity, and nothing else.

eps_uncond mixes the two blobs' predictions using their posterior weights. Subtracting d.min(0) before the exponential is the standard numerical-stability trick. Without it, np.exp underflows to zero early in sampling, giving a divide-by-zero.

eps = e_u + guidance * (e_c - e_u) is classifier-free guidance, written exactly as in the paper. At guidance 1 the unconditional term cancels completely and you are left with e_c.

The update line is the DDPM reverse step. Noise is added at every step except the last, which is why the final samples are not all collapsed onto one point.

Running a real text-to-image model

Be clear-eyed about the cost. A Stable Diffusion 1.5-class checkpoint is roughly 4 GB to download in full precision, about 2 GB in half precision. On a CPU it takes minutes per image, not seconds. Distilled few-step models cut the step count but not the download.

bash
pip install diffusers transformers torch accelerate
real_t2i.py
import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float32,             # float16 needs a GPU
)
pipe = pipe.to("cpu")

image = pipe(
    "a photo of a brass water pot on a wooden table, soft morning light",
    num_inference_steps=25,
    guidance_scale=7.5,                    # the same knob as the toy example
).images[0]
image.save("out.png")

No output block: the result is an image, and it differs on every run unless you pin a generator seed. Check the model card for the current download size and licence before you pull it.

If your machine cannot run this, the toy example above still teaches the mechanism honestly. The sampler is identical; only the noise predictor changed.

Common mistakes

Confusing steps with guidance. Steps control how finely the noise is removed. Guidance controls how hard the prompt pushes. Raising steps will not fix a prompt that is being ignored.

Guidance above about 12 on most models. Colours blow out and detail turns crunchy, for the reason the toy output shows. If you need stronger adherence, rewrite the prompt before raising the scale.

Expecting reproducibility without a seed. Pass an explicit torch.Generator seed, or every run differs.

Forgetting that guidance doubles the cost. Each step runs the model twice, once conditional and once unconditional. A guidance scale of exactly 1.0 lets most libraries skip the second pass.

Try it yourself

Move the blobs closer together — change the means to -0.4 and +0.4 — and rerun. Guidance 1 will no longer separate them cleanly, and you will need a higher scale to pick a side. That is the same reason near-synonym prompts need stronger guidance than distinct ones.

What to learn next

  • Text to video — the same machinery with time added, and why that costs so much.
  • Diffusion models — the noise-removal process on its own terms.
  • CLIP — where the text encoder inside these models comes from.

Researcher — Mathematics and papers.

The forward and reverse processes

DDPM (Ho, Jain and Abbeel, 2020) defines a fixed forward process that adds Gaussian noise on a schedule:

q(x_t | x_{t-1}) = N( x_t ; sqrt(1 - beta_t) * x_{t-1} , beta_t * I )

with  alpha_t = 1 - beta_t   and   abar_t = product of alpha_s for s <= t

closed form:  q(x_t | x_0) = N( x_t ; sqrt(abar_t) * x_0 , (1 - abar_t) * I )

x_0 is the clean data point, x_t the noised version at step t, beta_t the variance added at step t, and I the identity covariance. The closed form is what makes training cheap: any timestep can be sampled directly without simulating the chain.

The model eps_theta(x_t, t, c) predicts the noise, and the training objective reduces to a plain regression:

L = E over t, x_0, eps  of  || eps - eps_theta( sqrt(abar_t) x_0 + sqrt(1 - abar_t) eps , t, c ) ||^2

c is the conditioning — for text-to-image, the encoded prompt. eps is the standard normal noise actually drawn. This simplified objective drops the variational weighting terms and empirically trains better than the exact ELBO.

Score matching, and why the toy example is exact

The noise prediction and the score of the noised distribution are related by:

score(x_t) = grad_x log q(x_t)  =  - eps_theta(x_t, t) / sqrt(1 - abar_t)

Song and Ermon (2019) and Song et al. (2020) established this equivalence and the continuous SDE view. For a Gaussian data distribution, q(x_t) is Gaussian with known parameters, so the score is available analytically — which is why the developer example needs no training and is nonetheless the true optimum a network would converge to.

Classifier-free guidance

Dhariwal and Nichol (2021) steered sampling with the gradient of a separately trained noisy-image classifier. Ho and Salimans (2022) removed the classifier by training one model on both conditional and unconditional objectives, dropping the condition with probability around 10 percent, then extrapolating at sampling time:

eps_guided = eps_theta(x_t, t, empty) + w * ( eps_theta(x_t, t, c) - eps_theta(x_t, t, empty) )

w = 1 recovers standard conditional sampling. w > 1 samples from a distribution proportional to p(x | c) * p(c | x)^(w - 1), which sharpens the implicit classifier. The cost is two forward passes per step.

The distortion is not a bug to be tuned away. Guidance deliberately samples from a different distribution, trading distribution coverage for conditional fidelity. The FID-versus-CLIP-score curve traces this out: FID is best near w = 2 to 3, prompt alignment keeps improving well past that, and typical product defaults sit at w = 7 because human preference favours adherence over fidelity.

Dynamic thresholding (Saharia et al., 2022) rescues high-guidance samples by clipping predicted x_0 values to a per-sample percentile, which is why Imagen tolerates much higher scales than pixel-space predecessors.

Latent diffusion

Rombach et al. (2022) moved diffusion from pixels into the latent space of a pretrained VAE, at a downsampling factor of 8. A 512x512 image becomes a 64x64x4 latent — a 48-fold reduction in elements, and roughly that factor in compute per step.

Conditioning enters through cross-attention layers in the UNet, where the keys and values come from the encoded prompt. This is the architectural point of contact with CLIP: Stable Diffusion 1.x uses a frozen CLIP ViT-L/14 text encoder, and later versions add or swap in larger encoders including T5-XXL, which measurably improves the model's handling of long and compositional prompts.

Samplers

The DDPM ancestral sampler needs hundreds of steps. Faster alternatives reinterpret the reverse process:

  • DDIM (Song, Meng and Ermon, 2020): a non-Markovian deterministic sampler, usable at 20 to 50 steps, and invertible — which is what makes latent image editing possible.
  • DPM-Solver (Lu et al., 2022): treats the reverse process as an ODE and applies a high-order exponential integrator. Good samples at 10 to 20 steps.
  • Consistency models (Song et al., 2023) and adversarial distillation (SDXL-Turbo, Sauer et al., 2023): distil a teacher into a one to four step student, with a measurable loss in diversity.

Failure modes with known causes

Text rendering. Character-level detail is high-frequency and spatially precise, and both CLIP text encoders and VAE latents discard exactly that. Models using T5 text encoders and higher latent resolution do better, which points at the cause.

Counting and spatial relations. Cross-attention gives no mechanism for cardinality or for enforcing "left of". Attend-and-Excite (Chefer et al., 2023) improves subject presence by optimising attention maps at inference time, and does not solve counting.

Attribute binding. "A red cube and a blue sphere" frequently swaps colours. Cross-attention maps for adjectives leak across nouns.

Memorisation. Carlini et al. (2023) extracted near-exact training images from Stable Diffusion, concentrated on duplicated training examples. This is a genuine privacy and copyright issue, not a hypothetical one.

Evaluation

FID (Heusel et al., 2017) measures distribution distance in Inception feature space and is sensitive to sample count, resolution and the resizing implementation — cross-paper FID comparisons are frequently invalid. CLIPScore measures prompt alignment and inherits CLIP's compositional blind spots. Neither captures the failures above, so benchmarks such as T2I-CompBench and DrawBench exist to probe binding, counting and relations directly. Human preference remains the deciding evaluation, and it is expensive.

Papers

What to learn next

  • Text to video — the same machinery with time added, and why that costs so much.
  • Diffusion models — the noise-removal process on its own terms.
  • CLIP — where the text encoder inside these models comes from.