Transformers · Position Encoding

Understanding RoPE (Rotary Position Embeddings)

A visual tutorial on Rotary Position Embeddings: position chooses how far to turn, head-dim pairs choose which clock speed to use, and the attention score reads the relative angle.

Simon Dong • June 2026 • ~16 min read

RoPE is one of the quiet pieces of infrastructure behind modern language models. It does not add a learned position vector or a bias table. It rotates queries and keys so that the attention score naturally depends on relative distance. The key mental model: RoPE does one rotation per channel-pair; the token position says how far to turn, and the channel-pair index says which clock speed to use.

What you'll need

You should be comfortable with vectors and dot products, basic trig (sin/cos), matrix multiplication, and the rough idea of self-attention (Q, K, V). Everything else is built up from scratch.

Notation: d = head dimension, p/m/n = token positions, i = channel-pair index, theta_i = the speed of pair i, and angle[p,i] = p * theta_i.

1.Background — where RoPE came from

Why we need position encodings at all. Self-attention is permutation-invariant: the raw Q·K dot product compares token content with no notion of order. Without an additional signal, "the cat sat" and "sat the cat" present the same set of token vectors. Something has to tell the model where each token sits — otherwise a Transformer is an orderless bag of words.

What came before RoPE. The original Transformer (2017) added a fixed sinusoidal vector to each token embedding. Later models learned an absolute position embedding per slot, or added a learned relative-position bias directly to the attention scores (e.g. T5). These approaches work, but they attach position to the inputs or logits rather than to the geometry of Q·K itself — and absolute schemes tend to extrapolate poorly beyond the sequence lengths seen in training.

RoPE (2021). RoFormer (Su et al.) took a different route: rotate Q and K by an angle proportional to position, so relative distance appears inside the dot product itself. GPT-NeoX popularized the half-split / rotate_half implementation, and LLaMA (2023) made RoPE the default pattern for open decoder LLMs.

Why it won — and still wins in 2026

How it's used in 2026. RoPE is the de-facto positional encoding for decoder LLMs, almost always with a long-context extension layered on (the rope_scaling config): NTK-aware scaling, Position Interpolation, YaRN, and LongRoPE rescale the frequency spectrum so a model trained at 8K runs at 128K–1M+ tokens. Multimodal RoPE (M-RoPE) extends the rotation to image/video position grids in vision-language models, and it now appears in some encoders (ModernBERT) and audio/time-series stacks too.

Top 2026 models using it

Qwen3.5 DeepSeek-V4 GLM-5.2 Kimi K2.7 MiniMax M2 Gemma 4 Mistral 4 GPT-OSS

Two RoPE "dialects" show up across these:

A packaging note: GLM-5.2 runs on the native glm_moe_dsa class in transformers 5.13, while Kimi K2.7 loads via trust_remote_code=True with its own modeling file (text tower = DeepSeek-V3). In both cases, the positional signal underneath is still RoPE.

2.Where RoPE plugs in: the attention score

Scaled dot-product attention is:

Attention(Q, K, V) = softmax( (Q @ K^T) / sqrt(head_dim) ) @ V

The piece we care about is Q @ K^T. For one head it produces a (seq, seq) matrix of scores:

(Q @ K^T)[i, j] = q_i . k_j     # similarity between query token i and key token j

Every entry is a dot product between one query vector and one key vector; after softmax these scores decide who attends to whom. The problem: q_i . k_j is position-blind — by itself, it cannot tell whether token j is adjacent to i or 4000 tokens away. RoPE rotates q and k before the product so the score depends on the gap i - j.

3.The two axes (don't merge them)

Inside a head, Q and K have shape:

query / key:  (batch, n_heads, seq, head_dim)

Two axes drive RoPE, and they play different roles:

seq axis   → chooses position p       → how FAR to rotate
head_dim   → split into pairs i      → which clock speed theta_i to use

Position is just an integer per token: position_ids = [0, 1, 2, 3, ...]. Head dimension is where the clocks live: a head of size d gives d/2 two-dimensional pairs, and each pair gets its own speed theta_i. The actual rotation angle is the product:

angle[p, i] = position p * speed theta_i

4.One clock hand: rotating a single 2-D pair

Smallest case: head_dim = 2, so there is only one pair and therefore only one clock. In this toy case, freeze the speed at theta = 1 rad per position. Now the only thing changing is the token position p: bigger p means turn the same hand farther. A hand starting at (1, 0):

position p=0:  turned 0 rad → ( 1.000,  0.000)
position p=1:  turned 1 rad → ( 0.540,  0.841)
position p=2:  turned 2 rad → (-0.416,  0.909)
position p=3:  turned 3 rad → (-0.990,  0.141)
p=0
angle 0
p=1
angle 1
p=2
angle 2
p=3
angle 3

Rotation of a point (x, y) by angle a is the standard 2-D rotation matrix:

\[ \begin{pmatrix} x' \\ y' \end{pmatrix}= \begin{pmatrix} \cos a & -\sin a \\ \sin a & \cos a \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} \]

A clock hand has fixed length — only its direction changes. In this one-pair example, position has become an angle. Later, with more head dimensions, we will have many clocks with different speeds; position still tells each clock how far to turn.

5.Why turning hands? The relative-position property

Query at position m, key at position n, both hands starting at (1, 0):

q_rot = (cos m, sin m)
k_rot = (cos n, sin n)

q_rot . k_rot = cos m cos n + sin m sin n = cos(m - n)

The score depends only on m - n — the angle between the hands. Check:

m=2,  n=5:   cos(2 - 5)   = cos(-3) = -0.990
m=10, n=13:  cos(10 - 13) = cos(-3) = -0.990   # same gap → same score

Turning each hand by its position makes the dot product read the relative angle. Formally, rotations compose so that

\[ (R_m\,q)^{\top}(R_n\,k) = q^{\top} R_m^{\top} R_n\, k = q^{\top} R_{\,n-m}\, k . \]

6.One speed isn't enough: the hand wraps onto itself

A single hand comes back to the same spot once per full turn, so different gaps land on the same angle and get the same score. A readout based on that one hand cannot tell them apart.

To see exact collisions, take a hand whose period is 6 positions (one full turn every 6 tokens; theta = 2*pi/6 ≈ 1.047). Score for a gap d is cos(theta * d):

gap d:          0      1      2      3      4      5      6      7
cos(theta*d): 1.00   0.50  -0.50  -1.00  -0.50   0.50   1.00   0.50

Read off the actual collisions — this hand cannot distinguish:

gap 0  vs  gap 6              → both score  1.00
gap 1  vs  gap 5  vs  gap 7   → all score  0.50
gap 2  vs  gap 4              → both score -0.50

So with this one hand, a token 1 position away is indistinguishable from one 5 or 7 positions away — identical score, 0.50. That's the wrap problem in concrete form: one speed gives one ambiguous dial. We need more hands.

7.Many hands at different speeds (a clock)

Now add a second pair in the head dimension: a slow hand with period 60 positions (theta = 2*pi/60 ≈ 0.105) alongside the fast period-6 hand. This is not a second rotation applied to the same pair; it is a different pair in head_dim, with a different speed. Compare the two gaps that collided above — gap 1 vs gap 7:

gap d:            0       1       6       7
fast hand (T=6):  1.00    0.50    1.00    0.50      ← 0&6 collide, 1&7 collide
slow hand (T=60): 1.00    0.995   0.809   0.743     ← all four distinct

The fast hand confuses gap 0 with gap 6, and gap 1 with gap 7. The slow hand breaks both ties. Combine them into a fingerprint and every gap is unique:

gap 0 → (1.00, 1.00)
gap 1 → (0.50, 0.995)
gap 6 → (1.00, 0.809)
gap 7 → (0.50, 0.743)     # no two pairs match

This is exactly a clock: the second hand and hour hand are different hands, not two turns of the same hand. Each hand has its own speed; the same position gap turns each hand by a different amount. The fast hand pins down fine local differences, while the slow hand keeps long-range gaps from wrapping into the same reading.

fast hand (low i):  big move per step  → separates NEARBY gaps, wraps quickly
slow hand (high i): tiny move per step → coarse nearby, stable over long range

Real models don't pick round periods — they use theta_i = base^(-2i/d), a whole geometric spread of speeds. Periods 6 and 60 were chosen only so the collisions land on clean integers.

8.Which two channels share a hand?

With many pairs, we choose which two channels form each clock hand. Each pair is one two-dimensional plane that gets one speed theta_i. The convention in essentially all current models is half-split: pair channel i with channel i + d/2.

d = 4:   pair 0 = (channel 0, channel 2)    # share the fast hand
         pair 1 = (channel 1, channel 3)    # share the slow hand

The original paper paired adjacent channels (0,1),(2,3); half-split is equivalent up to a fixed permutation but slices contiguously — faster on GPUs.

9.The full formula, broken down

For a token at position p, RoPE performs one rotation per pair. The angle of that rotation combines the two ideas we've been separating: position chooses how far, and pair index chooses which speed.

# 1. Speeds, one per hand/pair  (i = 0 .. d/2 - 1)
theta_i = base ** (-2*i / d)

# 2. Angle = position * speed
alpha[p, i] = p * theta_i

# 3. Turn each pair (x_i, x_{i+d/2}) by alpha[p, i]
x'_i       = x_i       * cos(alpha[p,i]) - x_{i+d/2} * sin(alpha[p,i])
x'_{i+d/2} = x_{i+d/2} * cos(alpha[p,i]) + x_i       * sin(alpha[p,i])

p and i are different axes: p comes from the sequence axis; i comes from the head-dim pair. theta_i is the fixed speed of pair i; the angle is p * theta_i. In the original RoPE formulation, and in many standard implementations, the default base is 10000. Long-context models often change this value (for example through rope_theta or rope_scaling), but 10000 is the useful default to learn first.

Table for d=4, base=10000 (theta_0=1.0, theta_1=0.01):

                hand i=0       hand i=1
              (theta=1.0)    (theta=0.01)
position p=0:   0 * 1 = 0.0    0 * 0.01 = 0.00
position p=1:   1 * 1 = 1.0    1 * 0.01 = 0.01
position p=2:   2 * 1 = 2.0    2 * 0.01 = 0.02
position p=3:   3 * 1 = 3.0    3 * 0.01 = 0.03

Down a column: speed fixed, position grows. Across a row: same position, different speeds.

10.The vectorized form (what real code runs)

rotate_half(x) = concat( -x[d/2:],  x[:d/2] )

Example:

x = [1, 2, 3, 4]
   first half  = [1, 2]
   second half = [3, 4]
rotate_half(x) = [-3, -4, 1, 2]

Then:

angles = [alpha_0, alpha_1]              # one per hand, length d/2
emb    = concat(angles, angles)          # length d
x_rope = x * cos(emb) + rotate_half(x) * sin(emb)

Why it equals §9: emb stores the angle p * theta_i for every pair, duplicated so both channels in the pair see the same angle. The rotation matrix has a cos part (keeps each channel aligned with itself → x * cos) and a sin part (pulls in the paired channel, swapped and sign-flipped → exactly rotate_half).

11.A full worked example (two hands active)

d = 4, theta_0 = 1, theta_1 = 0.01.

q = [1, 2, 3, 4]   at position m = 1
k = [1, 1, 1, 1]   at position n = 2

Turn q at m=1 (cos1=0.5403, sin1=0.8415; cos0.01≈1, sin0.01=0.01):

pair0 (q0,q2)=(1,3):  q'_0 = 1*0.5403 - 3*0.8415 = -1.9842
                      q'_2 = 3*0.5403 + 1*0.8415 =  2.4624
pair1 (q1,q3)=(2,4):  q'_1 = 2*1 - 4*0.01        =  1.9599
                      q'_3 = 4*1 + 2*0.01        =  4.0198
q_rot = [-1.9842, 1.9599, 2.4624, 4.0198]

Turn k at n=2:

k_rot = [-1.3254, 0.9798, 0.4932, 1.0198]

Score:

score = (-1.9842)(-1.3254) + (1.9599)(0.9798)
      + ( 2.4624)( 0.4932) + (4.0198)(1.0198)
      = 9.864

The dot product is index-aligned: channel 0 of Q meets channel 0 of K (both the fast hand), channels 1,3 meet their twins (slow hand). Each hand only ever compares against the same hand, so the score splits into per-hand terms, each depending on (m - n) * theta_i.

12.When frequencies interfere

Yes — they can. The rotations are geometrically independent (each hand lives on its own channel-pair). But the final attention score sums all hands into one scalar, and summed sinusoids can interfere — so two different gaps can produce the same total score.

Concrete collision. Say a head's contributions reduce to score(d) = cos(d) + cos(2d). Using cos(2d) = 2cos²d − 1, this is 2c² + c − 1 with c = cos d. Solve score = 0:

2c^2 + c - 1 = 0  →  c = 0.5  or  c = -1

c = 0.5  → d = pi/3 ≈ 1.047
c = -1   → d = pi   ≈ 3.142

check d=1.047:  cos(1.047) + cos(2.094) =  0.5 + (-0.5) = 0
check d=3.142:  cos(3.142) + cos(6.283) = -1.0 + ( 1.0) = 0

A gap of ≈1.047 and a gap of ≈3.142 give the identical score 0. So the rotations are independent, but the scalar score is a sum of all hands → they do interfere, and one score can alias. What rescues uniqueness is dimensionality: many hands per head (d/2) and many heads. The full vector of components is a richer fingerprint than any single scalar — collisions in one number are resolved by the others.

13.Why meaning is preserved

RoPE mixes two numbers inside each pair, but it does not compress them into one number. It applies a rotation: two inputs go in, two outputs come out, and the transformation is exactly reversible if you rotate back by the negative angle.

That distinction matters. An average like (x + y) / 2 loses information because many pairs can produce the same average. A rotation keeps the pair as a 2-D point and only changes its coordinate system.

pair (3,4), length 5, turned by 37 deg (cos=0.8, sin=0.6):
  out_i       = 3*0.8 - 4*0.6 = 0.0
  out_{i+d/2} = 4*0.8 + 3*0.6 = 5.0
  new length = sqrt(0^2 + 5^2) = 5   # unchanged

recover the original by rotating back:
  x = out_i*0.8 + out_{i+d/2}*0.6 = 0*0.8 + 5*0.6 = 3
  y = out_{i+d/2}*0.8 - out_i*0.6 = 5*0.8 - 0*0.6 = 4

The two values are mixed, but not destroyed. RoPE changes the direction of the 2-D pair so position can influence attention; it does not average, pool, or discard the original information.

14.Limitations: what could be improved

Vanilla RoPE is elegant but leaves real things on the table. Each item below is a concrete opening for a better position encoding.

  1. No built-in distance emphasis. RoPE applies every hand at every gap, with no mechanism to say "this is a far pair, lean on the slow hands." An improved scheme could learn how much each frequency band should count as a function of gap or content.
  2. Fixed, hand-set frequencies. The spectrum theta_i = base^(-2i/d) is set by one constant (base), not learned, not per-head, not data-dependent. A model that learns its frequencies could allocate resolution where the task needs it.
  3. Poor native length extrapolation. Far beyond training length, the fast hands hit angle regimes never seen in training and quality collapses — hence the bolt-on patches (NTK/YaRN/LongRoPE). A scheme that extrapolates by design wouldn't need them.
  4. Scalar aliasing. As §12 showed, summing hands into one score lets distinct gaps collide. The frequency set isn't chosen to minimize collisions over a target window — one could design the spectrum for maximal separability up to, say, 1M tokens.
  5. Wasted capacity at long range. Many high-frequency channels contribute near-noise once tokens are far apart, so part of head_dim does little useful work in long-context regimes.
  6. Token-distance only, not semantic distance. RoPE measures raw index gaps; it has no notion that some tokens "shouldn't count." Content-adaptive positioning is an open direction it doesn't touch.
  7. One spectrum for all heads and layers. Letting heads specialize (some short-range, some long-range) explicitly, rather than faking it through learned W_q, W_k magnitudes, could use the budget more deliberately.

The throughline: RoPE gives the model a fixed, uniform, content-blind bank of clock hands. Most improvement ideas amount to making those hands learned, adaptive, or better-conditioned for long context — without losing RoPE's two non-negotiables: the relative-position property and KV-cache compatibility.

15.Beyond vanilla RoPE

Open challenge: learned, content-adaptive emphasis that stays KV-cache-friendly and keeps long-range retrieval intact.

16.The reference implementation

The models in §1 share this exact code (the half-split dialect) via transformers' # Copied from / modular mechanism — learn it once, read almost every modern open LLM's attention:

# transformers/models/qwen3/modeling_qwen3.py  (lines 151-180)
def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)


@use_kernel_func_from_hub("rotary_pos_emb")
def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
    ...
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)

17.Summary — the one-paragraph mental model

seq axis        → gives position p
head_dim        → split into d/2 channel-pairs
pair i          → chooses speed theta_i = base^(-2i/d)
angle[p, i]     → p * theta_i
RoPE            → one rotation of pair i by angle[p, i]

The important relationship is: there are not two rotations; there is one rotation whose angle has two inputs. The sequence axis gives p, the head-dim pair gives theta_i, and their product gives the angle. If a head has dimension d, RoPE forms d/2 two-dimensional pairs, so there are d/2 speeds. Low-index pairs turn quickly and distinguish nearby positions; high-index pairs turn slowly and stay useful over longer ranges. When query position m meets key position n in Q·K, matching pairs compare against matching pairs, so each speed sees the relative gap m - n. One speed is ambiguous, but the whole bank of speeds forms a multi-scale clock for relative position.

RoPE is position turned into clock-hand angles, read back as relative distance.

How to cite this post

If this helped you and you want to reference it in a write-up, use the canonical URL below.

Dong, S. (2026). Understanding RoPE (Rotary Position Embeddings).
https://simondong1.github.io/rope.html
@misc{dong2026understandingrope,
  author = {Dong, Simon},
  title = {Understanding RoPE (Rotary Position Embeddings)},
  year = {2026},
  month = {June},
  url = {https://simondong1.github.io/rope.html},
  note = {Technical blog post}
}

Further reading

Note

The period-6/60 hands in §6–7 are illustrative — chosen for clean arithmetic, not a specific model's exact spectrum. Standard RoPE commonly uses base=10000; long-context models often change that through rope_theta or rope_scaling.