Description

Level 2 — How It Works (The Three Core Knobs)

1. Temperature ():
• Divides the logits by before applying the softmax function:
• (Greedy Limit): Amplifies differences between logits; the highest logit takes of the probability mass (deterministic).
• : Preserves the model's unaltered learned probability distribution.
• : Flattens the distribution toward uniform, increasing randomness and creativity (along with risk of incoherence).

2. Top- Sampling:
• Sorts tokens by probability, keeps only the most likely tokens (e.g., ), sets all other logits to , and re-normalizes the remaining probabilities to sum to .
• Limitation: is fixed! When the distribution is flat (many valid choices, e.g., "I traveled to ___"), cuts off good options; when the distribution is peaked (only 1–2 valid choices, e.g., "United States of ___"), admits bad tokens into the pool.

3. Top- (Nucleus) Sampling:
• Dynamically selects the smallest set of top tokens whose cumulative probability exceeds threshold (e.g., ):
• Why it shines: When the model is confident ("America": 95%), the nucleus contains just token! When the model is uncertain across many good synonyms, the nucleus automatically expands to include tokens.

Level 3 — Advanced Depth (Execution Order, Min-, Repetition Penalties & Speculative Decoding)

• Standard Sampling Pipeline Order: Raw Logits Repetition / Frequency Penalty Temperature Scaling (z / T) Top-k Filter Softmax Top-p (Nucleus) Filter Re-normalize & Categorical Sample.

• Why in APIs Can Still Vary Slightly: Even at temperature=0, floating-point addition in GPU kernels is non-associative across dynamic batch sizes ((a + b) + c != a + (b + c)), which can flip a close tie between two top logits unless deterministic kernels and a fixed seed are used.

• Min- Sampling (Modern Alternative to Top-): Keeps tokens whose probability is at least (e.g., ), scaling dynamically with the top token's confidence and avoiding Top-'s failure mode at high temperatures ().

Learning notes

Sampling Pipeline Flowchart

Numerical Walkthrough: How Temperature, Top-, and Top- Transform Logits

Token:                  "blue"     "clear"    "gray"     "falling"  "banana"
Raw Logits (z):          4.0        3.0        2.0        1.0        0.0
--------------------------------------------------------------------------------
1) Temperature = 1.0:    0.636      0.234      0.086      0.032      0.012
2) Temperature = 0.5:    0.844      0.114      0.015      0.002      0.0003  (Sharp / Confident!)
3) Temperature = 2.0:    0.405      0.246      0.149      0.091      0.055   (Flat / Creative!)
--------------------------------------------------------------------------------
Apply Top-k (k=3) on T=1.0:
   Keep top 3 ->        [0.636,     0.234,     0.086,     0.000,     0.000]
   Re-normalize ->      [0.665,     0.245,     0.090,     0.000,     0.000]
--------------------------------------------------------------------------------
Apply Top-p (p=0.85) on T=1.0:
   Cumulative sum ->    [0.636,     0.870,     0.956,     0.988,     1.000]
   Cutoff at >= 0.85 -> Keep "blue" (0.636) + "clear" (+0.234 = 0.870 >= 0.85)
   Re-normalize ->      [0.731,     0.269,     0.000,     0.000,     0.000]

Practical Production Settings by Task

Example or implementation

Complete PyTorch Implementation of Temperature + Top- + Top- Sampler

import torch
import torch.nn.functional as F

def sample_next_token(
    logits: torch.Tensor,
    temperature: float = 1.0,
    top_k: int | None = None,
    top_p: float | None = None,
) -> torch.Tensor:
    """
    Samples the next token ID from raw logits of shape (batch_size, vocab_size).
    """
    # 1. Greedy decoding shortcut when temperature == 0
    if temperature <= 0.0:
        return torch.argmax(logits, dim=-1, keepdim=True)

    # 2. Temperature scaling
    logits = logits / temperature

    # 3. Top-k filtering (in logit space before softmax)
    if top_k is not None and top_k > 0:
        top_k = min(top_k, logits.size(-1))
        kth_vals, _ = torch.topk(logits, top_k, dim=-1)
        min_top_k = kth_vals[:, -1].unsqueeze(-1)
        logits = torch.where(logits < min_top_k, torch.full_like(logits, float("-inf")), logits)

    # 4. Top-p (Nucleus) filtering
    if top_p is not None and 0.0 < top_p < 1.0:
        sorted_logits, sorted_indices = torch.sort(logits, descending=True, dim=-1)
        sorted_probs = F.softmax(sorted_logits, dim=-1)
        cumulative_probs = torch.cumsum(sorted_probs, dim=-1)

        # Shift cumulative probs right by 1 so we ALWAYS keep the first token that crosses top_p
        sorted_indices_to_remove = (cumulative_probs - sorted_probs) >= top_p
        sorted_logits[sorted_indices_to_remove] = float("-inf")

        # Scatter masked logits back to original vocabulary indices
        logits = torch.zeros_like(logits).scatter_(dim=-1, index=sorted_indices, src=sorted_logits)

    # 5. Convert filtered logits to valid probability distribution and sample
    probs = F.softmax(logits, dim=-1)
    return torch.multinomial(probs, num_samples=1)

# Demo
raw_logits = torch.tensor([[4.0, 3.0, 2.0, 1.0, 0.0]])
torch.manual_seed(42)
sampled_id = sample_next_token(raw_logits, temperature=0.8, top_k=4, top_p=0.85)
print("Sampled Token ID:", sampled_id.item())

Interview-ready explanation

90-Second Spoken Pitch for Interviews

"At each decoding step, an LLM outputs a vector of raw logits over its vocabulary. How we convert those logits into a chosen token governs the trade-off between determinism and diversity. Greedy decoding simply picks the argmax token, which is optimal for deterministic tasks like SQL generation, but in open-ended text it leads to repetitive loops and locally safe but globally dull phrasing. Pure categorical sampling over all tokens fixes repetition, but the cumulative probability mass of the long tail of thousands of tiny-probability tokens causes occasional incoherent derailments.

We control this with three hyperparameters:

1. Temperature (): Divides logits by before softmax. sharpens the distribution around high-probability tokens, while flattens it.

2. Top-: Keeps only the highest-probability tokens, which truncates the tail but uses a rigid cutoff size regardless of whether the distribution is flat or peaked.

3. Top- (Nucleus Sampling): Dynamically keeps the smallest subset of top tokens whose cumulative probability exceeds (like ). When the model is certain of the next word, the nucleus shrinks to tokens; when many continuations are valid, the nucleus expands automatically."

Questions to revisit

Q1: Why is there a cumulative_probs - sorted_probs >= top_p shift in the Top- implementation?

Q2: Should you tune both temperature and top_p aggressively at the same time?

Q3: How does Beam Search differ from Top-/Top- sampling, and why is it rarely used in modern chat LLMs?