ML basics: bias-variance, precision/recall/F1/AUC, cross-entropy
Section focus: Explain every item aloud without notes
Description
Level 1 — The Simple Intuition (ELI5)
Imagine practicing archery and grading a medical test:
- Bias vs. Variance (Archery): High Bias (Underfitting) means your bow sights are crooked—all your arrows land tightly grouped, but feet to the left of the bullseye (your model is too simple). High Variance (Overfitting) means your hands shake—your arrows scatter wildly all around the target depending on the wind (your model memorized training noise instead of the true rule).
- Precision vs. Recall (Fishing Net): Precision asks: "Of everything inside my net, what percentage are actual tuna instead of old boots?" Recall asks: "Of all the tuna swimming in the bay, what percentage did my net manage to catch?"
- Cross-Entropy (Surprise Meter): Measures how "surprised" your model is by the true answer. If the model says
"99% chance the next word is Paris"and the answer is"Paris", cross-entropy loss is near . If it says"0.01% chance", loss shoots toward !
Level 2 — Core Mathematical Formulations
1. The Bias-Variance Decomposition
For a true target (where noise has variance ) and a model trained on random dataset draws:
2. Classification Metrics: Precision, Recall, , , ROC-AUC & PR-AUC
From the Confusion Matrix (True Positives , False Positives , False Negatives , True Negatives ):
- Precision (Positive Predictive Value): — Use when False Positives are costly (e.g., spam filter deleting a job offer, or flagging a legitimate credit card user).
- Recall (Sensitivity / True Positive Rate): — Use when False Negatives are fatal (e.g., cancer screening, prompt-injection security guardrail).
- Score (Harmonic Mean):
Why Harmonic Mean instead of Arithmetic Mean? If a lazy classifier predicts Positive on a rare disease, and . The arithmetic mean is (misleadingly high!), whereas the harmonic mean punishes extreme imbalance.
- ROC-AUC vs. PR-AUC:
- ROC Curve plots vs. across all thresholds . Probabilistic meaning of ROC-AUC: The probability that the model assigns a higher score to a randomly chosen positive instance than to a randomly chosen negative instance!
- Trap on Imbalanced Data: When negatives outnumber positives (e.g., fraud detection), is massive, keeping tiny even when is large! Always use PR-AUC (Precision-Recall AUC) for heavily imbalanced datasets.
3. Cross-Entropy Loss, KL Divergence & Perplexity (The LLM Connection)
Given true distribution and predicted distribution :
In LLM next-token prediction, the ground truth is a one-hot vector (, all others ), so Cross-Entropy simplifies to the Negative Log-Likelihood (NLL) of the true token :
And Perplexity () is simply the exponentiated average cross-entropy loss:
Intuitively, means the model is as uncertain at each step as if it were rolling a uniform -sided die among equally likely words.
Level 3 — Advanced Depth (Double Descent in Modern Deep Learning)
Classical ML says test error follows a U-curve: as parameters exceed dataset size, variance explodes. In modern overparameterized neural networks (like Transformers), once pushes past the interpolation threshold (where training loss hits ), test error drops a second time ("Double Descent") because stochastic gradient descent (SGD/AdamW) with weight decay implicitly selects the minimum-norm, smoothest interpolating solution!
Learning notes
Bias-Variance & Double Descent Regime Map
Loading diagram…
Confusion Matrix & Metric Selection Guide
ACTUAL GROUND TRUTH Positive (1) Negative (0) +-------------------+-------------------+ PREDICTED Pos | True Positive(TP) | False Positive(FP)| --> Precision = TP / (TP + FP) CLASS (1) | (Hit / Catch) | (False Alarm) | +-------------------+-------------------+ Neg | False Negative(FN)| True Negative(TN) | (0) | (Miss!) | (Correct Reject) | +-------------------+-------------------+ | | v v Recall (TPR) = FPR = FP / (FP + TN) TP / (TP + FN)
Metric | Formula | When to Optimize It | Real AI Engineering Example |
Precision | False Positives are expensive | Auto-sending customer refund emails without human review | |
Recall (TPR) | False Negatives are dangerous | Retrieval stage of RAG (if top- chunks miss the answer, the LLM cannot recover!) | |
Score | Balanced cost of and | Named Entity Recognition (NER) / Guardrail classification | |
ROC-AUC | Threshold-independent ranking on balanced classes | Comparing two binary classifiers when class ratio is roughly | |
PR-AUC | Rare positive class ( prevalence) | Detecting jailbreak attacks or toxic outputs in production logs | |
Cross-Entropy / PPL | Calibrated probability distribution training | Pretraining & SFT loss for LLMs |
Example or implementation
PyTorch & NumPy Implementation: Cross-Entropy, Perplexity, Precision/Recall/F1 & ROC-AUC
import numpy as np import torch import torch.nn.functional as F # 1. numerically stable Cross-Entropy & Perplexity from raw logits (LogSumExp trick) def manual_cross_entropy_and_ppl(logits: torch.Tensor, targets: torch.Tensor) -> tuple[float, float]: """ logits: (seq_len, vocab_size), targets: (seq_len,) Uses LogSumExp trick: log(sum(exp(z))) = m + log(sum(exp(z - m))) to prevent overflow! """ max_logits, _ = logits.max(dim=-1, keepdim=True) log_sum_exp = max_logits.squeeze(-1) + torch.log(torch.exp(logits - max_logits).sum(dim=-1)) target_logits = logits[torch.arange(logits.size(0)), targets] nll_per_token = log_sum_exp - target_logits ce_loss = nll_per_token.mean().item() perplexity = float(np.exp(ce_loss)) return ce_loss, perplexity # 2. Classification Metrics + Mann-Whitney U (Wilcoxon) Exact ROC-AUC def binary_metrics(y_true: np.ndarray, y_scores: np.ndarray, threshold: float = 0.5) -> dict[str, float]: y_pred = (y_scores >= threshold).astype(int) tp = int(np.sum((y_pred == 1) & (y_true == 1))) fp = int(np.sum((y_pred == 1) & (y_true == 0))) fn = int(np.sum((y_pred == 0) & (y_true == 1))) precision = tp / (tp + fp + 1e-12) recall = tp / (tp + fn + 1e-12) f1 = 2 * precision * recall / (precision + recall + 1e-12) # Exact ROC-AUC via probabilistic definition: P(score_pos > score_neg) pos_scores = y_scores[y_true == 1] neg_scores = y_scores[y_true == 0] pairwise_wins = sum((p > n) + 0.5 * (p == n) for p in pos_scores for n in neg_scores) roc_auc = pairwise_wins / (len(pos_scores) * len(neg_scores)) return {"Precision": round(precision, 4), "Recall": round(recall, 4), "F1": round(f1, 4), "ROC_AUC": round(roc_auc, 4)} # Verify both logits = torch.tensor([[2.5, 0.3, -1.2], [0.1, 3.2, 0.5]]) targets = torch.tensor([0, 1]) ce, ppl = manual_cross_entropy_and_ppl(logits, targets) print(f"CE Loss: {ce:.4f} (PyTorch: {F.cross_entropy(logits, targets).item():.4f}) | Perplexity: {ppl:.4f}") print("Binary Metrics:", binary_metrics(np.array([1, 0, 1, 0, 1]), np.array([0.92, 0.31, 0.81, 0.60, 0.75])))
Interview-ready explanation
90-Second Spoken Pitch for Interviews"These foundational ML metrics govern how we train, diagnose, and evaluate both classical classifiers and modern LLMs.First, the Bias-Variance Trade-off decomposes generalization error into Bias (systematic underfitting because the model lacks capacity), Variance (overfitting to training noise), and Irreducible Noise. In classical ML, increasing parameters past the dataset size causes variance to explode, whereas in modern LLMs we operate in the Double Descent regime where massive overparameterization paired with weight decay and vast data yields smooth generalization.Second, for classification and RAG evaluation, raw accuracy fails on imbalanced data. We track Precision (, minimizing false alarms) and Recall (, minimizing missed positives), combining them via the harmonic mean . In a RAG pipeline, we optimize Recall@k in the vector retriever so we don't miss the relevant document, and Precision in the cross-encoder reranker so we don't feed noisy chunks to the LLM. To evaluate classifiers across all thresholds, ROC-AUC measures the probability that a random positive ranks higher than a random negative, but on rare-event tasks like toxicity or fraud detection ( positives), we must use PR-AUC because massive True Negatives mask False Positives in ROC-AUC.Finally, Cross-Entropy Loss measures the KL divergence between the true distribution and the model's predicted probabilities. For one-hot token labels, it simplifies to , and its exponential is Perplexity—the effective branching factor of uncertainty per token."