Rotary position embedding and why rotation beats addition
RoPE rotates query and key vectors instead of adding an index, so the distance between two tokens stays the same wherever the sequence sits. Covers the rotation intuition, why dimension pairs rotate at different speeds, and a PyTorch implementation for a custom transformer.