Description

Level 2 — Complete Anatomy of the 2017 Paper (, Layers, Heads)

1. The Encoder Stack ( identical layers):
• Takes source sentence embeddings + sinusoidal positional encodings.
• Each layer has two sub-layers: (a) Bidirectional Multi-Head Self-Attention (every source word attends to every other source word), and (b) a Position-wise Feed-Forward Network (FFN):
• Around each sub-layer is a Residual Connection followed by Post-Layer Normalization: .

2. The Decoder Stack ( identical layers):
• Generates the translated target sentence autoregressively.
• Each layer has three sub-layers:
1. Masked Multi-Head Self-Attention: Causal mask ( on future tokens) prevents target position from peeking at .
2. Encoder-Decoder Cross-Attention: Queries () come from the Decoder layer below; Keys () and Values () come from the top of the Encoder stack!
3. Position-wise FFN + Residual + LayerNorm.

3. Why Self-Attention Beat RNNs & CNNs (Section 4 of the Paper):
• Maximum Path Length between any two tokens: in Self-Attention vs. in RNNs and in dilated CNNs!
• Sequential Operations Required: in Self-Attention (fully parallelizable across the sequence during training) vs. in RNNs!

Level 3 — Advanced Depth: What Changed Between 2017 and Modern Frontier LLMs (2024–2026)?

1. Encoder-Decoder Decoder-Only: Modern LLMs drop the Encoder and Cross-Attention entirely, unifying prompt processing (Prefill) and generation (Decode) in a single causal Decoder stack.

2. Post-LN Pre-LN with RMSNorm: The 2017 paper placed LayerNorm after the residual addition (), which caused unstable gradients near the output and required a strict learning-rate warmup. Modern LLMs use Pre-Norm (), leaving an unobstructed "gradient highway" from the loss straight back to the first layer, and replace LayerNorm with RMSNorm (dropping mean-centering and bias to save compute).

3. Sinusoidal Absolute PE RoPE: Replaced additive sine/cosine waves at the bottom layer with multiplicative Rotary Position Embeddings inside every attention layer.

4. ReLU FFN SwiGLU FFN: Replaced standard 2-matrix ReLU MLP with a 3-matrix gated linear unit ().

5. MHA GQA / MLA: Replaced 1:1 Multi-Head Attention with Grouped-Query Attention to slash KV-cache memory.

Learning notes

Original 2017 Encoder-Decoder vs. Modern Pre-Norm Decoder-Only Block

Section-by-Section Cheat Sheet of Attention Is All You Need

2017 Transformer vs. Modern 2026 Frontier LLM (Interview Gold Table!)

Example or implementation

PyTorch Implementation: Modern Pre-RMSNorm + SwiGLU Transformer Block vs. 2017 Post-LN

import torch
import torch.nn as nn
import torch.nn.functional as F

class RMSNorm(nn.Module):
    """Root Mean Square Layer Normalization (used in LLaMA, Mistral, Gemma)."""
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Compute RMS in FP32 for numerical stability: x / sqrt(mean(x^2) + eps)
        norm_x = x.float() * torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps)
        return (norm_x.type_as(x)) * self.weight

class SwiGLUFFN(nn.Module):
    """Gated Linear Unit with Swish (SiLU) activation: (SiLU(x W_gate) * (x W_up)) W_down."""
    def __init__(self, d_model: int, hidden_dim: int):
        super().__init__()
        self.w_gate = nn.Linear(d_model, hidden_dim, bias=False)
        self.w_up   = nn.Linear(d_model, hidden_dim, bias=False)
        self.w_down = nn.Linear(hidden_dim, d_model, bias=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

# Verify RMSNorm + SwiGLU shapes (using 8/3 * d_model hidden dimension so parameter count matches 4*d_model 2-matrix FFN!)
d_model = 512
hidden_dim = int(2 * (4 * d_model) / 3)  # 1365 -> 3 * 512 * 1365 ~= 2 * 512 * 2048 parameters!
norm = RMSNorm(d_model)
ffn = SwiGLUFFN(d_model, hidden_dim)
x = torch.randn(2, 10, d_model)
out = x + ffn(norm(x))  # Pre-Norm residual connection
print("Pre-RMSNorm + SwiGLU Output shape:", out.shape)

Interview-ready explanation

90-Second Spoken Pitch for Interviews

"Vaswani et al.'s 2017 paper Attention Is All You Need replaced recurrent and convolutional sequence models with a purely attention-based architecture. Its core insight was that RNNs force information through a sequential bottleneck—preventing GPU parallelization across time steps and losing long-range context—whereas Self-Attention connects any two tokens in path length and processes all training tokens simultaneously.

The original paper introduced an Encoder-Decoder architecture with layers, , and attention heads (), combining Scaled Dot-Product Attention, Multi-Head projections, position-wise ReLU Feed-Forward Networks, residual connections, Post-LayerNorm, and sinusoidal positional encodings.

If asked what has evolved from the 2017 paper to modern frontier LLMs like LLaMA-3, there are five critical upgrades:

1. We moved from Encoder-Decoder to Decoder-Only causal architectures.

2. We moved LayerNorm from Post-Norm to Pre-Norm so gradients flow unimpeded through the residual stream without exploding at initialization, and replaced LayerNorm with faster RMSNorm.

3. We replaced additive sinusoidal positional encodings with RoPE (Rotary Position Embeddings) at every attention layer.

4. We replaced the 2-matrix ReLU FFN with a 3-matrix SwiGLU gated FFN (scaling intermediate width to to keep parameter count equal).

5. We replaced standard Multi-Head Attention with Grouped-Query Attention (GQA) to cut KV-cache memory during inference."

Questions to revisit

Q1: Why does SwiGLU use a hidden dimension of instead of the 2017 paper's ?

Q2: Why is Pre-LayerNorm so much more stable to train at scale than the 2017 paper's Post-LayerNorm?

Q3: What was Label Smoothing () in Section 5.4 of the paper, and do modern LLMs still use it?