hn.today

Understanding Positional Embeddings in Modern LLMs

magazine.sebastianraschka.com239 points27 comments
Screenshot of Understanding Positional Embeddings in Modern LLMs

This technical walkthrough explains why transformer language models need information about token order and how that information enters their computations. Token embeddings represent token identity, so repeated words begin with the same vector; without positional signals, attention may not reliably distinguish different arrangements. A causal attention mask provides some implicit ordering because each token can see only itself and earlier tokens, but it does not directly encode how far apart tokens are. Explicit positional methods therefore give the model a clearer signal about sequence structure.

The tutorial first develops learned absolute position embeddings using GPT-2: each sequence position has a trainable vector, added element by element to the corresponding token embedding before the first transformer block. This makes repeated tokens at different positions distinct from the start, but GPT-2’s table of 1,024 position vectors limits its input length. The broader tutorial promises to compare this approach with contemporary techniques, explain relative positional embeddings - especially Rotary Position Embeddings (RoPE) - with visual examples, mathematics, and a small PyTorch implementation, and survey positional choices in recent open-weight models. It also flags architectures that adapt or omit explicit encodings, including NoPE, which relies on order information implicit in causal attention. The material assumes familiarity with transformer attention and is intended as a detailed technical foundation for understanding modern LLM architectures.

Read on magazine.sebastianraschka.com27 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

ChatGPT Wallet

ChatGPT Wallet

ChatGPT Wallet lets ChatGPT agents make purchases using tokenized card details, with users reviewing each purchase and approving it through a security check. Launching first for US ChatGPT Pro subscribers, it is built with major payment networks and preserves eligible card rewards and protections. (openai.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.