Description

Level 2 — How It Works (The 3-Stage Training Pipeline)

1. Stage 1 — Self-Supervised Pretraining (Base Model):
• Data: Trillions of tokens () of filtered web text, books, papers, and code.
• Objective: Autoregressive Next-Token Prediction (Cross-Entropy Loss over all tokens):
• Result: High world knowledge ("entropy compression"), zero instruction-following format.

2. Stage 2 — Supervised Fine-Tuning (SFT / Instruction Tuning):
• Data: high-quality (prompt, response) pairs (written by human experts or distilled from stronger frontier models).
• Objective: Same Cross-Entropy Loss, but masked to compute loss ONLY on the assistant response tokens (not the user prompt ):
• Result: Learns chat templates (<|user|>, <|assistant|>), tone, and format ("Behavioral Cloning").

3. Stage 3 — Preference Alignment (RLHF via PPO vs. DPO):
• Why SFT alone isn't enough: SFT forces the model to imitate every token of a reference answer, even when multiple valid phrasings exist, and cannot explicitly penalize subtle hallucinations, sycophancy, or unsafe completions that the base model assigns high probability to. It is much easier for humans to judge which of two answers is better () than to write the gold-standard answer from scratch.

Level 3 — Advanced Depth (RLHF/PPO vs. DPO vs. GRPO Math)

• RLHF with PPO (Proximal Policy Optimization):
1. Collect preference triples where is preferred ("winner") over ("loser").
2. Train a separate Reward Model using the Bradley-Terry pairwise loss:
3. Optimize the policy online via RL (PPO) to maximize reward while staying close to the frozen SFT reference model via a KL-divergence penalty (which prevents "Reward Hacking", where the model outputs gibberish that tricks the reward model):
• Drawback of PPO: Requires loading 4 models in GPU memory simultaneously (Policy Actor , Reference , Reward , Critic/Value network ) and unstable online sampling.

• Direct Preference Optimization (DPO — Rafailov et al., 2023):
• Mathematically solves for the closed-form optimal policy of the KL-constrained reward maximization problem, substituting directly into the Bradley-Terry loss!
• Eliminates the separate Reward Model and online RL loop entirely—optimizing preference directly with a simple binary cross-entropy loss using only 2 models ( and frozen ):

• GRPO (Group Relative Policy Optimization — DeepSeekMath / DeepSeek-R1): Removes the Value/Critic network from PPO by sampling a group of outputs per prompt and normalizing their rewards relative to the group mean and standard deviation ()—ideal for verifiable reasoning/coding rewards (RLVR).

Learning notes

End-to-End LLM Training Lifecycle

Head-to-Head Comparison Across Training Stages

Example or implementation

PyTorch Implementation of the DPO Loss Function from Scratch

import torch
import torch.nn.functional as F

def compute_dpo_loss(
    policy_chosen_logps: torch.Tensor,    # log pi_theta(y_w | x), shape: (B,)
    policy_rejected_logps: torch.Tensor,  # log pi_theta(y_l | x), shape: (B,)
    ref_chosen_logps: torch.Tensor,       # log pi_ref(y_w | x),   shape: (B,)
    ref_rejected_logps: torch.Tensor,     # log pi_ref(y_l | x),   shape: (B,)
    beta: float = 0.1,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """
    Computes Direct Preference Optimization (DPO) loss and implicit rewards.
    """
    # 1. Log-ratios between active policy and frozen SFT reference policy
    chosen_logratios = policy_chosen_logps - ref_chosen_logps
    rejected_logratios = policy_rejected_logps - ref_rejected_logps

    # 2. Implicit reward margin: beta * (log(pi/ref)_w - log(pi/ref)_l)
    logits = beta * (chosen_logratios - rejected_logratios)

    # 3. Binary cross-entropy loss: -log(sigmoid(logits))
    losses = -F.logsigmoid(logits)

    # 4. Track implicit rewards for monitoring reward margin during training
    chosen_rewards = beta * chosen_logratios.detach()
    rejected_rewards = beta * rejected_logratios.detach()
    return losses.mean(), chosen_rewards.mean(), rejected_rewards.mean()

# Quick verification
loss, r_win, r_lose = compute_dpo_loss(
    policy_chosen_logps=torch.tensor([-12.0, -8.5]),
    policy_rejected_logps=torch.tensor([-15.0, -11.0]),
    ref_chosen_logps=torch.tensor([-13.0, -9.0]),
    ref_rejected_logps=torch.tensor([-14.0, -10.5]),
    beta=0.1,
)
print(f"DPO Loss: {loss.item():.4f} | Implicit Reward (Win): {r_win.item():.3f} > (Lose): {r_lose.item():.3f}")

Interview-ready explanation

Questions to revisit

Q1: Why do we need the KL-divergence penalty in RLHF and DPO?

Q2: When does online RL (PPO / GRPO) outperform offline DPO?

Q3: Why do we mask out the prompt tokens when computing the SFT loss?