Learn AI Create the Future — with Pranay
Main site

How Models Know Word Order

Rotary position embeddings (RoPE)

RoPE rotates each query and key by an angle set by its position, so the dot product between any two tokens depends only on the gap between them.

Read these first

On this page 7
  1. Why this is a better idea than adding
  2. How it works
  3. Where the rotation is applied
  4. Where you have already seen this
  5. What is honestly hard here
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

RoPE does not add anything to a word. It turns the word, like a dial, by an amount set by its position.

Think of two people standing on a large clock face painted on the ground. Each one faces whichever hour their position points to. Person at slot 3 faces the 3, person at slot 8 faces the 8.

Now ask: what is the angle between the two people? It is five hours. Move both of them forward by two slots and the angle is still five hours.

That is RoPE. Absolute positions turn each token. What the model reads out is the angle between them, which depends only on the gap.

Why this is a better idea than adding

Earlier methods add a position vector to the word vector. That mixes two different things into one set of numbers, and the model has to untangle them later.

Rotating changes no lengths. A word vector of length 3 stays length 3 after rotating. Only the direction moves.

So the word's meaning, carried by its magnitude and its shape, survives intact. Position lives purely in the direction.

How it works

Split the word vector into pairs of numbers. Treat each pair as a point on a small circle. Rotate that point by an angle equal to the position, times a speed chosen for that pair.

   pair 0 (fast dial):    turns a lot per token
   pair 1:                turns less
   pair 2:                turns much less
   pair 3 (slow dial):    barely turns across a whole document

   token at slot 5  ->  every pair rotated by 5 x its own speed
   token at slot 9  ->  every pair rotated by 9 x its own speed

   attention compares the two  ->  sees the 4-step difference

Attention compares a query with a key by measuring how aligned they are. Rotate both and the alignment shifts by exactly the gap between their positions.

Where the rotation is applied

This detail matters. RoPE is not applied to the word at the start of the model. It is applied inside every attention head, to the query and the key, right before they are compared.

The value — the content that actually gets carried forward — is never rotated. Position influences who talks to whom, and never contaminates what is said.

Where you have already seen this

  • Llama, Mistral, Qwen, Gemma and DeepSeek all use RoPE.
  • Almost every open-weight chat model you can download today.
  • Long-context models advertising 128,000 tokens, which stretch RoPE to get there.
  • Vision-language models, which extend RoPE to rows and columns of an image.

What is honestly hard here

Two questions trip up nearly everyone.

Why does rotating help at all? Because the comparison step measures alignment. Alignment between two rotated arrows depends on the difference of the two angles, not on either angle alone.

Why pairs? Because a rotation needs two coordinates. One number has no direction to turn. This is the same reason a clock face is flat rather than a line.

If both answers feel slippery, that is normal. The code below shows the effect as plain numbers, which helps more than any amount of re-reading.

Remember this

  • RoPE rotates queries and keys by an angle set by their position.
  • The comparison between two tokens then depends only on the gap between them.
  • Vector lengths never change, so word meaning is not diluted by position.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy

RoPE from scratch, and the property that justifies it

rope.py
import numpy as np
np.set_printoptions(precision=4, suppress=True)

HEAD_DIM, BASE = 8, 10000.0

# One turning speed per pair of dimensions. Pair 0 turns fastest.
inv_freq = 1.0 / (BASE ** (np.arange(0, HEAD_DIM, 2) / HEAD_DIM))
print("turning speed of each pair (radians per token):", inv_freq)

def rope(x, pos):
    """Rotate each (even, odd) pair of x by an angle proportional to pos."""
    ang = pos * inv_freq                     # one angle per pair
    c, s = np.cos(ang), np.sin(ang)
    even, odd = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = even * c - odd * s           # a plain 2-D rotation
    out[1::2] = even * s + odd * c
    return out

rng = np.random.default_rng(7)
q = rng.normal(size=HEAD_DIM)
k = rng.normal(size=HEAD_DIM)

print("\nq.k with no positions at all:", f"{q @ k:.4f}")

print("\nsame gap, different absolute positions:")
for m, n in [(0, 3), (5, 8), (20, 23), (600, 603)]:
    print(f"  q at {m:>3}, k at {n:>3}  (gap {n-m:+d}):  {rope(q, m) @ rope(k, n):.6f}")

print("\ndifferent gaps, all starting from position 10:")
for gap in [0, 1, 2, 4, 8, 16]:
    print(f"  gap {gap:>2}: {rope(q, 10) @ rope(k, 10 + gap):9.6f}")

# Nothing is added to the vector, so its length never changes.
print("\nlength of q before rotation:", f"{np.linalg.norm(q):.6f}")
for p in [0, 1, 137, 100000]:
    print(f"  after rotating to position {p:>6}:", f"{np.linalg.norm(rope(q, p)):.6f}")
Output
turning speed of each pair (radians per token): [1.    0.1   0.01  0.001]

q.k with no positions at all: 1.1668

same gap, different absolute positions:
  q at   0, k at   3  (gap +3):  1.419923
  q at   5, k at   8  (gap +3):  1.419923
  q at  20, k at  23  (gap +3):  1.419923
  q at 600, k at 603  (gap +3):  1.419923

different gaps, all starting from position 10:
  gap  0:  1.166769
  gap  1:  1.092240
  gap  2:  1.228006
  gap  4:  1.466779
  gap  8:  1.083798
  gap 16:  1.603363

length of q before rotation: 1.986816
  after rotating to position      0: 1.986816
  after rotating to position      1: 1.986816
  after rotating to position    137: 1.986816
  after rotating to position 100000: 1.986816

Reading that output

Four identical numbers, to six decimal places. Positions 0 and 3, 5 and 8, 20 and 23, 600 and 603. Same gap, same score. This is the property RoPE exists to provide, and it holds exactly, not approximately.

Gap 0 reproduces the plain dot product. 1.166769 matches q @ k before any rotation. Two tokens at the same position are compared with no positional effect at all, which is the correct behaviour.

Different gaps give different scores, but not monotonically. Gap 4 scores higher than gap 2. RoPE does not build in a preference for nearby tokens. It gives the model a readable signal about the gap, and lets attention decide what to do with it. That distinction matters when you compare RoPE with ALiBi, which does the opposite.

Lengths are identical at position 100000. A rotation is orthogonal, so it preserves norms exactly. Compare that with an additive encoding, where the position vector at slot 100000 would need to coexist with a word vector of a similar scale.

One honest limit of this demo. With HEAD_DIM = 8 there are only four pairs, so gap-to-score is coarse and wiggly. Real heads use 64 or 128 dimensions, giving 32 or 64 pairs and a much smoother signal.

The layout difference you will hit in real code

The RoPE paper pairs dimension 2i with 2i+1. Hugging Face and most GPU kernels instead pair i with i + head_dim/2, using a helper called rotate_half. Both are the same rotation under a permutation of channels:

rope_layouts.py
import numpy as np

D, BASE = 8, 10000.0
inv = 1.0 / (BASE ** (np.arange(0, D, 2) / D))

def rope_interleaved(x, p):                       # the paper's layout
    a = p * inv
    c, s = np.cos(a), np.sin(a)
    e, o = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = e * c - o * s
    out[1::2] = e * s + o * c
    return out

def rope_half_split(x, p):                        # the layout in transformers
    a = np.concatenate([p * inv, p * inv])
    c, s = np.cos(a), np.sin(a)
    rot = np.concatenate([-x[D // 2:], x[:D // 2]])   # this is rotate_half
    return x * c + rot * s

rng = np.random.default_rng(7)
q, k = rng.normal(size=D), rng.normal(size=D)

print("interleaved, gap 3 at two places:",
      round(rope_interleaved(q, 5) @ rope_interleaved(k, 8), 6),
      round(rope_interleaved(q, 20) @ rope_interleaved(k, 23), 6))
print("half-split,  gap 3 at two places:",
      round(rope_half_split(q, 5) @ rope_half_split(k, 8), 6),
      round(rope_half_split(q, 20) @ rope_half_split(k, 23), 6))

perm = list(range(0, D, 2)) + list(range(1, D, 2))   # reorder into half-split layout
print("half-split on permuted inputs   :",
      round(rope_half_split(q[perm], 5) @ rope_half_split(k[perm], 8), 6))
Output
interleaved, gap 3 at two places: 1.419923 1.419923
half-split,  gap 3 at two places: 1.529955 1.529955
half-split on permuted inputs   : 1.419923

Both layouts give the relative property. They give different numbers on the same raw vector, and match once the channels are permuted. That is why loading weights trained with one convention into code using the other produces a model that runs, produces fluent text, and is quietly wrong.

Common mistakes

Rotating the values. RoPE applies to q and k only. Rotating v corrupts the content path and the model trains badly with no obvious error.

Rotating after the head split, or before it, inconsistently. The rotation acts on head_dim, not on hidden_size. Applying it to the full width before splitting into heads pairs channels that belong to different heads.

Ignoring position offsets during generation. With a KV cache, the new token is at past_len, not at 0. Cached keys were rotated at their original positions and must not be re-rotated. Getting this wrong yields output that starts well and drifts.

Using float16 for the cos and sin tables. Angles grow with position, and half precision loses resolution as the magnitude grows. Build inv_freq and the position products in float32, then cast the result. Most reference implementations do this deliberately.

Try it yourself

Raise BASE from 10000 to 500000, which is what Llama 3 uses, and reprint inv_freq. Every dial slows down. Then check gaps of 0 to 16 again: the scores change far less between neighbouring gaps. You have found the tradeoff that the next lesson is entirely about.

What to learn next

Researcher — Mathematics and papers.

Formulation

RoPE, introduced in Su et al. (2021), RoFormer, seeks a function $f$ such that the attention logit depends only on the relative offset:

$$ \langle f(q, m),\, f(k, n) \rangle = g(q, k, m - n) $$

The solution takes $f$ to be a rotation. Group the $d$ head dimensions into $d/2$ pairs. For pair $i$ define $\theta_i = b^{-2i/d}$ with base $b = 10000$. The rotation matrix is block diagonal:

$$ R^d_{\Theta, m} = \mathrm{diag}\big(R(m\theta_0),\, R(m\theta_1),\, \dots,\, R(m\theta_{d/2-1})\big), \qquad R(\alpha) = \begin{pmatrix} \cos\alpha & -\sin\alpha \ \sin\alpha & \cos\alpha \end{pmatrix} $$

Apply it to the projected query and key:

$$ \tilde{q}m = R^d{\Theta, m}\, W_Q x_m, \qquad \tilde{k}n = R^d{\Theta, n}\, W_K x_n $$

Why the relative property holds

$R^d_{\Theta, m}$ is orthogonal, so $(R_m)^\top = (R_m)^{-1} = R_{-m}$. Rotations about a common axis compose additively, giving $R_{-m} R_n = R_{n-m}$. Therefore

$$ \tilde{q}_m^\top \tilde{k}_n = (W_Q x_m)^\top R_m^\top R_n (W_K x_n) = (W_Q x_m)^\top R^d_{\Theta,\, n-m} (W_K x_n) $$

The absolute indices cancel exactly. Note this is stronger than the sinusoidal case: there the relative structure exists at the input and is destroyed by $W_Q W_K^\top$, whereas here the rotation is applied after the projections and survives into the logit.

Complex form and the efficient kernel

Identifying each pair with a complex number $z_i = x_{2i} + \mathrm{i}\,x_{2i+1}$, the rotation is a multiplication by a unit modulus scalar:

$$ f(x, m)_i = z_i \, e^{\mathrm{i} m \theta_i} $$

and

$$ \langle f(q,m), f(k,n) \rangle = \mathrm{Re} \sum_{i=0}^{d/2-1} q_i \bar{k}_i\, e^{\mathrm{i}(m-n)\theta_i} $$

In practice no matrix is materialised. Precompute $\cos(m\theta_i)$ and $\sin(m\theta_i)$ once and apply elementwise:

$$ \tilde{x} = x \odot \cos + \mathrm{rot}(x) \odot \sin $$

Cost is $O(nd)$ elementwise operations per head, negligible against the $O(n^2 d)$ of attention. Zero learned parameters.

Long-term decay

The paper shows the relative logit exhibits decay with $|m-n|$, via Abel summation on $\sum_i q_i \bar{k}_i e^{\mathrm{i}(m-n)\theta_i}$: an upper bound on the sum of partial sums of $e^{\mathrm{i}(m-n)\theta_i}$ shrinks as the offset grows.

The bound is loose and the decay is not monotone, as the developer output shows. This is a real point of confusion: RoPE provides a readable relative signal, not a hard recency prior. ALiBi provides the opposite. That difference is why RoPE needs interpolation to extend context while ALiBi does not.

Wavelengths and the extrapolation boundary

Pair $i$ has wavelength $\lambda_i = 2\pi b^{2i/d}$. For $d = 128$ and $b = 10000$ these span $2\pi \approx 6.3$ to $2\pi \cdot 10^4 \approx 62832$ tokens.

The critical quantity is how many full rotations each pair completes inside the training window $L$. Pairs with $\lambda_i \ll L$ see every phase during training. Pairs with $\lambda_i > L$ never complete a turn, so at inference beyond $L$ they present angles the model has never encountered. This is the precise mechanism behind RoPE's extrapolation failure, and the target of every scaling method in the next lesson.

Partial rotary

Several models rotate only a fraction of each head. GPT-NeoX applied RoPE to 25 percent of the head dimensions; Phi-family models use partial_rotary_factor values below 1. The unrotated channels behave like a NoPE subspace, carrying content without positional modulation. Transformers exposes this as partial_rotary_factor in rope_parameters.

Adoption

RoPE is the default in essentially every open-weight decoder released since 2023: Llama 2 and 3, Mistral, Qwen, Gemma, DeepSeek, Phi. It extends naturally to more than one axis, which is why vision-language models adopted multi-axis variants — covered in the images and video lesson.

Papers

What to learn next