Regularization & Generalization

AI/ML Engineering Track | Complexity: [COMPLEX] | Time: 4-6 hours

Prerequisites: Module 1.1.5 (Loss Functions — stable softmax cross-entropy), Module 1.1.7 (Tiny NumPy NN Lab — manual SGD loop), Module 1.3 (Training Neural Networks — PyTorch training loop, validation discipline), Module 1.3.1 (Initialization & Signal Propagation), Module 1.3.2 (Optimizers & Learning-Rate Dynamics — weight decay vs AdamW).

Primary tools: PyTorch 2.12, torch.nn, torch.optim, NumPy, Matplotlib (optional plotting).

---

Hypothetical scenario: a computer-vision team ships a ResNet that achieves 99.8% training accuracy on a curated internal dataset of 2,000 labeled images. Demo day goes well. Two weeks later, the model is deployed against real warehouse footage and misclassifies roughly one in four frames — worse than a linear baseline trained on the same labels. Post-mortem review reveals the training script never held out a validation set, the model had twenty times more parameters than training examples, and dropout was accidentally left disabled during the final “evaluation” run that produced the 99.8% headline number. The team did not fail because regularization is mysterious; they failed because they never measured the gap between memorization and generalization, and they treated training accuracy as a proxy for production quality.

That gap — the difference between performance on data the model has seen during optimization and performance on genuinely new data drawn from the same underlying distribution — is the central engineering problem of supervised deep learning. In Block A you derived cross-entropy by hand and watched a tiny NumPy network memorize XOR before generalizing to Fashion-MNIST. In Block B you replaced the manual update loop with PyTorch and learned how initialization and optimizers control whether training converges at all. This module addresses the next question: even when training loss goes to zero, why does test loss often stay high, and what engineering levers close that gap without throwing away representational power?

Every regularization technique in this module trades something on the training set — usually peak accuracy or loss — for better behavior on unseen data. None of them replaces the discipline of a held-out validation set, careful hyperparameter tuning, or honest reporting. They are, however, the standard toolkit that makes large models trainable on finite data, and each one maps cleanly onto intuition you already built in earlier modules. The Block B spine continues here: when you write nn.Dropout(0.5), you are implementing the same random masking idea you could write in NumPy in twenty lines; when you pass label_smoothing=0.1, you are changing the target vector in the cross-entropy you coded in Module 1.1.5. PyTorch did not invent regularization — it industrialized patterns you already understand.

---

Learning Outcomes

By the end of this module, you will be able to:

1. Define overfitting precisely using simultaneous training and validation curves, and diagnose capacity mismatch (model size vs dataset size) using a controlled PyTorch experiment.
2. Implement inverted dropout with nn.Dropout, explain why scaling by 1/(1−p) at train time makes inference deterministic, and configure train vs .eval() behavior correctly in a full loop.
3. Apply weight decay through the optimizer interface, reference the decoupled AdamW mechanics from Module 1.3.2 without conflating L2-in-gradient with true weight decay, and select typical penalty strengths for dense networks.
4. Build early stopping with validation monitoring, patience, and best-checkpoint restoration, and configure label smoothing via nn.CrossEntropyLoss(label_smoothing=ε) connected back to Module 1.1.5’s cross-entropy derivation.
5. Compare data augmentation, stochastic depth, DropPath, and weight averaging (EMA/SWA) at a practitioner level, and design a combined regularization stack tuned on validation metrics rather than test leakage.

---

Why This Module Matters

A model that minimizes training loss is solving the wrong problem if your deployment environment presents inputs the optimizer never saw. Production ML systems fail quietly on this point: dashboards show declining training loss while validation loss creeps upward, and nobody notices until a downstream service starts returning nonsense. The engineering skill is not memorizing a list of regularizers; it is reading the train–validation gap, choosing interventions that address the diagnosed failure mode, and verifying that each intervention improves generalization rather than merely slowing convergence. Incident post-mortems in mature ML organizations almost always include a loss-curve replay; teams that log only training metrics discover overfitting only after customer impact.

Regularization also connects the Block A from-scratch story to modern PyTorch idioms in a way that demystifies “magic” defaults. Dropout is not a mysterious regularizer bolted onto nn.Module; it is a train-time random mask with a deterministic inference path you can implement in ten lines. Weight decay is not separate from optimization; it is a penalty on weight magnitude that you already saw implicitly when Module 1.3.2 explained why AdamW decouples L2 from adaptive scaling. Label smoothing is not a hack for leaderboard points; it is a direct modification of the cross-entropy target distribution you derived in Module 1.1.5, discouraging logits from exploding toward ±∞ on easy examples. When you read a published recipe listing dropout 0.1, weight decay 0.05, and label smoothing 0.1 together, you should see three independent answers to the same question: how do we stop this model from memorizing?

Finally, regularization choices interact. Dropout changes effective learning rate; weight decay interacts with initialization from Module 1.3.1; batch normalization (covered next in Module 1.3.4) has its own regularizing side effects. Practitioners who treat each knob independently often over-regularize — training loss stalls while validation barely moves — or under-regularize — validation improves for three epochs then degrades while training loss still falls. This module teaches you to compose the toolkit deliberately and tune on validation data, reserving the test set for a single final report. The engineering mindset is iterative diagnosis: measure the gap, apply one intervention, re-measure, document what moved.

---

Part 1: The Generalization Problem

1.1 Overfitting in Precise Terms

Overfitting occurs when a model’s predictions fit the training sample including its noise, idiosyncrasies, and sampling artifacts, rather than the underlying pattern that would transfer to new draws from the same distribution. Operationally, you observe it as training error continuing to improve while validation error worsens or plateaus below training performance. It is not enough to say “training accuracy is high”; overfitting is a divergence between train and validation curves over time or capacity.

Consider a binary classification task with a small dataset. A linear model may underfit: both train and validation loss stay high because capacity is too low to capture the decision boundary. A moderately sized MLP may fit well: train and validation loss decrease together and stabilize at similar values. An enormous MLP with more parameters than examples can achieve near-zero training loss while validation loss rises — classic overfitting. The pattern is not about absolute accuracy numbers; it is about divergence. A model at 95% train and 94% validation is healthy; a model at 99.9% train and 82% validation is overfitting even if 82% beats your baseline.

The bias–variance tradeoff frames the same phenomenon statistically. Bias is error from overly rigid assumptions (underfitting). Variance is error from sensitivity to training-set fluctuations (overfitting). Total expected error decomposes (under idealized assumptions) into bias, variance, and irreducible noise. Increasing model capacity reduces bias but typically increases variance; regularization pushes variance down, sometimes at the cost of slightly higher bias, seeking a better total error on unseen data. In practice you rarely compute bias and variance explicitly for a ResNet, but the vocabulary explains why adding capacity without regularization eventually hurts validation metrics even as training metrics keep improving.

1.2 Capacity, Data, and the Train/Val/Test Split

Model capacity is the effective flexibility of the function class your architecture and training procedure can represent. Depth, width, parameter count, and training duration all increase capacity. Dataset size and label quality constrain how much capacity you can use productively. A rule of thumb practitioners internalize: if your model has orders of magnitude more parameters than independent training examples, you need strong regularization, more data, or both.

The train/validation/test split is non-negotiable engineering hygiene:

Split	Purpose	Used during training?
Train	Update weights via backprop	Yes — every step
Validation	Tune hyperparameters, pick checkpoints, compare architectures	Yes — for decisions, never for gradients
Test	Final unbiased estimate of generalization	No — touch once at the end

Leakage happens when validation or test examples influence weight updates (e.g., tuning on test loss, or augmenting validation into training without relabeling splits). Module 1.3 established the evaluation loop with model.eval() and torch.no_grad(); this module assumes that loop exists and adds what to monitor (validation loss, not training loss alone) and what to do when the gap widens. Stratified splits matter for imbalanced classes: a random 80/10/10 split may leave rare classes with zero validation examples, making validation accuracy a noisy proxy that hides overfitting on head classes while the tail memorizes.

1.3 Monitoring the Gap in Practice

Production training pipelines should log both train_loss and val_loss every epoch (or every N steps for step-based training) to the same dashboard. The actionable signal is the delta: if train loss falls 20% over five epochs while val loss rises 5%, you are in the overfitting region regardless of absolute val performance. Set alerts on val loss divergence, not only on training failure. Checkpoint on best val metric, not latest epoch — Module 1.3’s checkpointing section pairs directly with the early stopping logic in Part 4.

When datasets are tiny, k-fold cross-validation estimates generalization more reliably than a single split, at the cost of k training runs. For engineering workflows with one fixed split, report val metrics with confidence intervals from bootstrap resampling if the validation set has fewer than a few thousand examples. The goal is the same: never optimize hyperparameters against noise you cannot measure.

1.4 Underfitting, Overfitting, and the Goldilocks Region

Diagnosis precedes prescription. Use train and validation curves together to classify behavior before adding regularizers:

Symptom	Train loss	Val loss	Likely diagnosis	First lever (not always regularization)
Both high, similar	High	High	Underfitting	More capacity, train longer, better features, lower weight decay
Train low, val high	Low	High or rising	Overfitting	Dropout, weight decay, augmentation, early stopping
Both low, similar	Low	Low	Good fit	Monitor for drift; avoid unnecessary regularization
Train low, val noisy	Low	Oscillates	Small val set or high variance	More val data, k-fold, EMA weights

Underfitting with heavy regularization is a common mistake after reading this module. If you add dropout 0.5, weight decay 0.1, and label smoothing 0.2 simultaneously on a already-small model, train accuracy may stall near chance while validation looks equally poor — you regularized away the model’s ability to learn signal. Back off regularizers until train performance is acceptable, then tighten until validation stops improving.

1.5 A Minimal Overfitting Demo in PyTorch 2.12

The following script trains a deliberately oversized MLP on a tiny subset of Fashion-MNIST. Watch train accuracy approach 100% while validation accuracy stalls or drops:

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms

torch.manual_seed(0)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Tiny train set (500 images), full test set for "validation"
transform = transforms.ToTensor()
full_train = datasets.FashionMNIST(root="./data", train=True, download=True, transform=transform)
test_set = datasets.FashionMNIST(root="./data", train=False, download=True, transform=transform)

train_subset = Subset(full_train, range(500))  # 500 samples
train_loader = DataLoader(train_subset, batch_size=64, shuffle=True)
val_loader = DataLoader(test_set, batch_size=256, shuffle=False)

class HugeMLP(nn.Module):
    def __init__(self):
        super().__init__()
        # ~1.6M parameters — far more than 500 examples
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 2048),
            nn.ReLU(),
            nn.Linear(2048, 2048),
            nn.ReLU(),
            nn.Linear(2048, 10),
        )

    def forward(self, x):
        return self.net(x)

model = HugeMLP().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

def accuracy(loader):
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for x, y in loader:
            x, y = x.to(device), y.to(device)
            pred = model(x).argmax(dim=1)
            correct += (pred == y).sum().item()
            total += y.size(0)
    return correct / total

for epoch in range(30):
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()
    train_acc = accuracy(train_loader)
    val_acc = accuracy(val_loader)
    print(f"epoch {epoch:02d}  train_acc={train_acc:.3f}  val_acc={val_acc:.3f}")

Typical output pattern (exact numbers vary with seed): train accuracy climbs above 0.95 by epoch 10 and toward 1.0 by epoch 25, while validation accuracy peaks around 0.78–0.82 early and may drift downward as training continues. That divergence is the signature you will recognize in production logs if you plot both curves — which you should, every run. Logging only train_loss is how teams discover overfitting weeks after deployment; logging val_loss alongside it turns the problem into a same-day hyperparameter decision.

If you shrink the hidden width from 2048 to 128 while keeping 500 training samples, the train–validation gap usually narrows without any explicit regularizer — capacity alone moved you along the bias–variance curve. That ablation is worth running once: it separates “my model is too big for my data” from “my optimization is broken.” Module 1.3.1 taught you that initialization controls whether signal propagates; this module assumes signal propagates and asks whether the model uses that signal to learn generalizable structure or to memorize labels.

  loss / error
      |
      |   training loss  ----\
      |                         \___  keeps falling
      |   validation loss  ---\      \
      |                        \______/  rises or flatlines
      +---------------------------------> epochs
              ^ overfitting region

---

Part 2: Dropout — Random Subnetworks at Train Time

2.1 Mechanism and Inverted Dropout

Dropout (Srivastava et al., 2014) randomly zeroes activations with probability p during training. For an activation vector h, a mask m ~ Bernoulli(1−p) element-wise multiplies: h' = m ⊙ h. Intuitively, each forward pass trains a thinned subnetwork; across many passes, the full network behaves like an ensemble of exponentially many sparse networks whose predictions are averaged at test time. With n hidden units, there are 2^n possible subnetworks in principle; dropout samples one per step rather than training each explicitly.

Inverted dropout scales surviving activations by 1/(1−p) during training so that inference requires no scaling. If you did not scale at train time, expected activation magnitude at train would be (1−p) times inference magnitude, and you would need to multiply by (1−p) at test — easy to forget in a codebase with twelve model variants. PyTorch’s nn.Dropout(p) implements inverted dropout: train multiplies by 1/(1−p) on kept units; eval passes activations through unchanged. The scaling preserves the expected value of each activation conditional on the input to the dropout layer, which is why deeper stacks can stack many dropout layers without systematic shrinkage at inference.

import torch
import torch.nn as nn

x = torch.ones(4, 8)  # batch=4, features=8
drop = nn.Dropout(p=0.5)
drop.train()
y_train = drop(x)
print(y_train[0])  # roughly half zeros, half 2.0 (because 1/(1-0.5)=2)

drop.eval()
y_eval = drop(x)
print(torch.equal(y_eval, x))  # True — identity at eval

2.2 Where to Place Dropout

Place dropout after activation functions in fully connected stacks, not on the raw logits layer before softmax/cross-entropy. Dropping logits arbitrarily changes the implied class prior; dropping hidden activations encourages redundant distributed representations that survive random unit removal. In transformer blocks, dropout appears on attention weights, residual outputs, and embedding layers — each location regularizes a different pathway; the same “after nonlinearity” intuition applies to MLP sublayers inside a block.

Typical dropout rates vary by architecture generation. Classic fully connected nets on MNIST-era tasks often used p=0.5 on hidden layers. Modern ResNets and ViTs frequently use lighter dropout (p=0.1–0.2) because batch normalization (Module 1.3.4) and massive datasets already constrain overfitting. When you inherit a recipe from a paper, treat dropout rate as dataset-dependent: CIFAR-scale with 50k images tolerates more than a 500-image fine-tuning job.

Layer type	Typical p	Notes
Input / embedding	0.1–0.2	Light — preserve information
Hidden FC layers	0.3–0.5	Standard for MLPs
CNN conv layers	0.1–0.3	Often lower; spatial correlation matters
Output logits	0	Do not dropout the head

In Module 1.3 you learned that model.train() and model.eval() toggle not only BatchNorm statistics (Module 1.3.4 preview) but also dropout behavior. A validation pass with model.train() left on silently applies random masks — validation metrics become noisy and optimistically biased if you average many stochastic forward passes, or pessimistically biased if a single unlucky mask zeros critical units.

2.3 Ensemble Interpretation and Back-Reference to Your MLP

During training with dropout, each mini-batch optimizes a different subnetwork. At inference, the full network with all units active approximates the geometric mean of those subnetworks’ predictions (under certain assumptions). That is why dropout reduces co-adaptation: neuron A cannot rely exclusively on neuron B always being present, because B may be dropped. Co-adaptation is particularly harmful with small data — two units may jointly memorize a spurious pixel pattern that disappears when one unit is removed.

Compare to Module 1.1.7’s Layer class: there, every hidden unit participated in every forward pass. Dropout is the engineering admission that, with finite data, full co-adaptation memorizes. Random omission forces robust features — the same reason bagging random forests reduces variance. From a backprop perspective, gradients flow only through surviving units in a given step; dropped units receive zero gradient for that step, which slows their individual updates but encourages alternate pathways to carry information. You do not need to implement that masking manually — nn.Dropout registers as an nn.Module and respects model.train() / model.eval() the same way BatchNorm will in the next module.

class MLPWithDropout(nn.Module):
    def __init__(self, p=0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Dropout(p=p),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Dropout(p=p),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.net(x)

Re-run the overfitting demo from Part 1 with p=0.4 after each ReLU. Validation accuracy typically improves and the train–val gap narrows, though peak train accuracy may fall — the intended trade. At inference, standard practice uses the deterministic full network (model.eval()). Monte Carlo dropout — running multiple stochastic forward passes at test time with dropout enabled and averaging predictions — is an optional refinement that approximates the ensemble average more closely; it costs latency and is rarely used in production vision serving, but it clarifies the train-time ensemble story.

2.4 Dropout and Effective Learning Rate

Dropout implicitly reduces the magnitude of activations that reach downstream layers (despite inverted scaling on survivors, the mask zeros information). Practitioners often find they can use a slightly higher learning rate with dropout than without, because the noise acts as a form of gradient regularization. If you add dropout and training stalls, try increasing learning rate modestly before removing dropout — the interaction with AdamW from Module 1.3.2 is empirical, not purely predictable from theory.

---

Part 3: Weight Decay and the L2 View

3.1 Penalizing Large Weights

Weight decay adds a penalty proportional to the squared L2 norm of weights. In the regularization view, the objective becomes:

L_total = L_data + (λ/2) · ||W||²

Large weights allow sharp, wiggly decision boundaries that fit noise. Penalizing magnitude encourages smoother functions — similar spirit to early stopping, but applied every gradient step. The scalar λ (or its optimizer-specific analogue) controls strength. Geometrically, weight decay pulls parameters toward the origin, shrinking the volume of parameter space reachable under a fixed learning-rate budget. For linear models, L2 regularization corresponds to a Gaussian prior on weights; the MAP estimate trades off data fit against prior belief that weights are small.

In PyTorch you usually pass weight_decay to the optimizer rather than adding an explicit L2 term to the loss, because optimizers apply decay with the correct coupling to momentum buffers and (for AdamW) decoupled scaling:

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=3e-4,
    weight_decay=0.01,  # typical range 1e-4 to 1e-1 depending on scale
)

3.2 Reference Module 1.3.2 — Do Not Conflate Mechanisms

Module 1.3.2 already explained that L2 regularization baked into the gradient is not equivalent to decoupled weight decay under Adam. Classic Adam adds L2 penalty to the gradient before adaptive scaling, so parameters with large second-moment estimates get weakened regularization. AdamW applies weight decay directly on weights after the adaptive update — the configuration you want when “weight decay” means “keep weights small” uniformly.

This module owns the generalization motivation for weight decay. Module 1.3.2 owns the optimizer mechanics. When you tune weight_decay=0.01 on AdamW, you are regularizing; when you mistakenly use L2-in-loss with Adam on a transformer, you may get inconsistent effective penalties across layers. If training underfits (train and val both poor), reduce weight decay; if overfitting persists with dropout, increase it modestly. A useful diagnostic: plot L2 norm of all parameters epoch-by-epoch. Steady growth alongside rising validation loss suggests decay is too weak; flat norm with stalled validation may mean decay is too strong relative to learning rate.

Typical starting points for dense image classifiers: 1e-4 to 1e-2 on AdamW, always validated on a held-out set. Weight decay interacts with initialization (Module 1.3.1): poorly scaled init plus aggressive decay can stall learning entirely. Convolutional layers and bias terms are sometimes excluded from decay in legacy recipes; PyTorch applies weight_decay to all parameters in the optimizer group unless you construct separate param groups with weight_decay=0 for biases and normalization scale parameters — a pattern you will see in vision transformer training scripts.

Explicit L2 in the loss (loss = ce_loss + 0.5 * lambda * sum(p.pow(2).sum() for p in model.parameters())) is equivalent to weight decay under pure SGD, which is why older tutorials combine them. With AdamW, prefer the optimizer argument — duplicating both paths double-counts the penalty. Module 1.3.2’s Adam versus AdamW discussion is the reference when a codebase mixes styles across files.

---

Part 4: Early Stopping — Regularization by Truncation

4.1 Monitoring Validation Loss

Early stopping halts training when validation loss stops improving for patience epochs (or steps), then restores weights from the best validation checkpoint. It limits effective capacity by preventing the optimizer from continuing into the overfitting regime where training loss still decreases but validation loss increases. Unlike dropout or weight decay, early stopping adds no noise to forward passes and no penalty term to the objective — it only controls how long you optimize. That makes it the first regularizer to try when compute is expensive and the model architecture is already fixed.

Tie this directly to Module 1.3’s loop: you already compute validation loss inside torch.no_grad() with model.eval(). Early stopping adds bookkeeping — track the best epoch, deep-copy state_dict(), and break the loop when improvement stalls:

import copy

best_val_loss = float("inf")
best_state = None
patience = 5
epochs_without_improve = 0

for epoch in range(max_epochs):
    train_one_epoch(model, train_loader, optimizer, criterion)
    val_loss = evaluate_loss(model, val_loader, criterion)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_state = copy.deepcopy(model.state_dict())
        epochs_without_improve = 0
    else:
        epochs_without_improve += 1
        if epochs_without_improve >= patience:
            print(f"Early stop at epoch {epoch}")
            break

model.load_state_dict(best_state)

You can wrap the same logic in a small helper class or use community libraries, but the state machine is always: track best, count stalls, restore on stop. Module 1.3’s checkpointing recommendation — save state_dict() on improvement — is the persistence layer for this pattern. In distributed training, only rank zero should write the best checkpoint to avoid race conditions on shared filesystems.

4.2 Relationship to the Manual Training Loop (Module 1.1.7)

In Module 1.1.7 you stopped training after a fixed epoch count because the NumPy lab prioritized clarity over production concerns. Early stopping is the production replacement for that fixed horizon: instead of for epoch in range(50), you ask whether epoch 50 is justified by validation evidence. The manual loop’s W -= lr * dW still runs every step; early stopping only decides when to halt and which W to keep. Combining early stopping with the AdamW schedule from Module 1.3.2 is standard — cosine annealing may still run for the full planned budget while early stopping exits early if validation saturates, or you couple them by reducing LR when validation plateaus (ReduceLROnPlateau scheduler).

4.3 Patience, Checkpoints, and Implicit Regularization

Patience balances noise and responsiveness. Validation loss is stochastic — especially with small validation sets — so patience=1 stops on single-batch noise; patience=20 may waste compute deep into overfitting. Values of 3–10 epochs are common for medium datasets. Step-based patience (number of optimizer steps without improvement) is preferable when epochs are enormous — fine-tuning a LLM may never complete a full epoch over billions of tokens.

Save state_dict() (or full checkpoints with optimizer state if you will resume training). Early stopping is not merely “stop the loop”; it is “revert to the best generalizing parameters.” Production training jobs should persist the best checkpoint to object storage, not the final epoch weights. When you reload for deployment, document which validation metric selected that checkpoint so incident reviewers can trace model lineage.

Framed as regularization, early stopping restricts the path length through parameter space. The trajectory that minimizes training loss may pass through regions that generalize well early, then wander into memorization valleys. Stopping is cheaper than dropout when compute is the bottleneck — but it does not help if the model overfits within the first epoch (you need architectural or data-side fixes). Combining early stopping with dropout is standard: dropout slows memorization per epoch, early stopping cuts off epochs when memorization wins anyway.

---

Part 5: Label Smoothing — Softening the Cross-Entropy Target

5.1 From One-Hot to Smoothed Distributions

Module 1.1.5 derived cross-entropy against a one-hot target: true class probability 1, others 0. That objective pushes logits for the correct class toward +∞ and all others toward −∞ — optimal for hard classification on infinite data, but on finite samples it encourages over-confident predictions that do not calibrate and can hurt generalization. Recall the stable softmax-CE implementation you wrote: subtract row max before exp, compute log-probabilities, dot with one-hot. Label smoothing changes only the target vector in that dot product — the forward numerics stay identical.

Label smoothing replaces the one-hot vector y with:

y_smooth[k] = (1 - ε) + ε / K    if k == y_true
            = ε / K              otherwise

for K classes and smoothing factor ε (often 0.1). PyTorch’s nn.CrossEntropyLoss(label_smoothing=ε) distributes the smoothing mass uniformly over all K classes, so the true class keeps (1−ε) + ε/K rather than (1−ε) alone. Szegedy et al. (2016) originally used ε/(K−1) on the non-target classes only; the built-in PyTorch argument uses the uniform ε/K form instead. The model is trained to predict a slightly uncertain distribution, penalizing extreme logits even when the example is easy. Worked example for K=10, ε=0.1, true class index 3: target vector entries are 0.91 at index 3 and 0.01 at each of the other nine indices; cross-entropy against that target still penalizes a prediction of [10, -10, -10, ...] because the non-target logits are far from their smoothed targets, not from zero.

PyTorch 2.12 exposes label smoothing directly on the loss module so you do not hand-build the target tensor on every batch:

criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
logits = model(x)  # shape: (batch, 10)
loss = criterion(logits, y)  # y: integer class indices, shape (batch,)

Internally, CrossEntropyLoss combines log_softmax with the smoothed target distribution — the same stable softmax-CE pipeline you implemented in Module 1.1.5, with a different target vector. You can verify equivalence on a toy batch by constructing the smoothed target manually and using F.kl_div or - (target * log_softmax).sum(dim=1).mean(); the built-in argument applies the uniform ε/K smoothing for you, so you never hand-build the smoothed target.

5.2 Calibration and Generalization

Calibration means predicted probabilities match empirical frequencies: among examples assigned 80% confidence, roughly 80% should be correct. Overfit models often assign 99.9% confidence to training memorized points while being wrong on shifted data. Label smoothing reduces the incentive to produce arbitrarily sharp logits, which often improves expected calibration on validation and test sets (Szegedy et al., 2016). Temperature scaling at inference is a separate post-hoc calibration technique; label smoothing acts during training.

Label smoothing interacts with weight decay and dropout: all three discourage memorization of exact label boundaries. It does not replace handling class imbalance (that requires different techniques like weighted CE or focal loss). Typical ε values: 0.05–0.2 for ImageNet-scale classification; start with 0.1 when experimenting. Very large ε approaches uniform targets and can underfit; if train accuracy collapses, reduce smoothing before removing dropout. On imbalanced datasets, label smoothing applies uniformly across classes — minority classes receive the same ε/K mass on non-target slots, which can slightly help or hurt depending on base rates; monitor per-class validation recall when experimenting.

5.3 Connection to Softmax Temperature and Inference Calibration

At inference, some practitioners apply temperature scaling — dividing logits by T > 1 before softmax — to soften over-confident predictions without retraining. Label smoothing acts during training to prevent the model from learning infinitely sharp logits in the first place. The two techniques address calibration at different stages; smoothing is cheaper at deployment because it requires no held-out calibration set, though post-hoc temperature tuning on a validation slice can still help if production calibration drift appears.

---

Part 6: Data Augmentation — Expanding the Effective Dataset

Data augmentation applies label-preserving transformations to inputs at train time: random crops, flips, color jitter, cutouts, and similar perturbations. Each epoch presents slightly different views of the same underlying example, which enlarges the effective training set without collecting new labels. The regularization mechanism is not mystery noise — it encodes invariances the model should satisfy. If a horizontal flip does not change whether an image is a cat, training on flipped copies teaches the network that left-right position is irrelevant to the label, so it cannot memorize a spurious pixel pattern tied to one orientation. Augmentation is often the strongest regularizer in vision pipelines — stronger than moderate dropout on modern CNNs and ViTs — because it attacks overfitting at the input boundary where memorization of raw pixel patterns begins.

Concrete vision examples make the principle tangible. Random horizontal flip (RandomHorizontalFlip) is standard on natural images where mirror symmetry preserves class identity. Random crop with padding (RandomCrop with padding=4 on CIFAR-sized images) forces the model to recognize objects that are partially occluded or off-center rather than relying on a fixed background framing. Color jitter (small random shifts in brightness, contrast, saturation, and hue) prevents the network from overfitting to lighting conditions in a small studio dataset — a common failure mode when deployment cameras differ from training capture. Each transform must preserve semantic label: a horizontal flip preserves “cat vs dog”; a 90° rotation may not preserve “6 vs 9” in digit recognition unless your problem definition treats rotation as label-preserving.

Mixup (Zhang et al., 2018) forms convex combinations of two input images and their one-hot labels, training the model on interpolated examples with soft targets — effectively merging augmentation and label smoothing in one step. CutMix (Yun et al., 2019) replaces a rectangular patch of one image with a patch from another and blends labels proportionally to the patch area, forcing the model to classify from partial context rather than memorizing global texture shortcuts. Both appear frequently in competitive vision training recipes; Module 1.4 (CNNs and computer vision) will implement full image-augmentation pipelines with torchvision.transforms.v2. Here the takeaway is architectural: augmentation is regularization through input noise, dropout through activation noise, weight decay through parameter noise. Stacking all three without monitoring validation loss often yields underfitting — the model never sees a clean example during training.

A typical torchvision.transforms training pipeline composes deterministic preprocessing with stochastic augmentations; evaluation uses only the deterministic steps:

# Sketch — full pipelines in the CNN module
train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomCrop(28, padding=4),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
])
# Eval: no random transforms — only deterministic preprocessing
eval_transform = transforms.ToTensor()

Pass train_transform to datasets.*(..., transform=train_transform) for the training split and eval_transform for validation and test. The DataLoader then serves augmented batches only during training; validation metrics reflect performance on clean, consistently preprocessed inputs — the same discipline as model.eval() for dropout.

For tabular or text models where geometric transforms do not apply, analogous regularizers include feature noise injection, token dropout (randomly masking input tokens during training), and back-translation for text augmentation. The unifying idea remains label-preserving perturbation of inputs so the optimizer cannot rely on exact memorization of raw features. Vision dominates the augmentation literature because spatial invariances are well understood; the engineering transfer to other modalities is conceptual, not a copy-paste of RandomCrop.

---

Part 7: Modern Regularizers — Brief Practitioner Survey

7.1 Stochastic Depth and DropPath

Stochastic depth (Huang et al., 2016) randomly drops entire residual blocks during training rather than individual activations. For a network with L blocks, block l is skipped with probability that increases with depth — shallow blocks almost always run, deep blocks are dropped more often. When a block is dropped, its residual contribution is zeroed and the input flows through the skip connection unchanged; surviving blocks are scaled so inference uses the full depth deterministically. The regularization effect is an implicit ensemble over depths: each training step optimizes a shallower subnetwork, and the full model at inference approximates an average over those depth configurations. This matters for very deep ResNets where per-activation dropout under-utilizes depth — dropping whole blocks forces the network to produce useful representations even when several layers are absent, reducing co-adaptation across the entire stack.

DropPath applies the same idea to individual sub-branches within a residual path, which is the dominant pattern in vision transformers (ViTs) and hierarchical transformers like Swin. Instead of dropping an entire transformer block, DropPath may zero the output of the attention sublayer or the MLP sublayer before it is added back to the residual stream, with drop probability often scheduled to increase in deeper stages. Like stochastic depth, the surviving branch is scaled at train time so eval requires no adjustment. The intuition matches dropout’s ensemble story but at coarser granularity: the model cannot rely on any single deep pathway always being present, so it distributes information across skip connections and shallower routes. You will encounter DropPath in timm and Hugging Face vision model configs; the drop_path_rate hyperparameter controls maximum drop probability at the deepest layer.

7.2 Weight Averaging — EMA and SWA

Exponential moving average (EMA) of model weights maintains a shadow copy updated each step: θ_ema ← β·θ_ema + (1−β)·θ. EMA weights often generalize better than raw training weights because they average out noise from mini-batch gradients — high-frequency oscillations along steep loss-surface directions cancel while the consistent descent toward a broad minimum accumulates. Geometrically, EMA traces a smoothed path through weight space, landing closer to flat regions of the loss landscape where small perturbations do not spike validation error. Diffusion and generative model training frequently deploy EMA copies for sampling even while the online weights continue optimizing aggressively.

Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) collects weight snapshots from the latter portion of training — or from multiple points along a learning-rate schedule — and averages them element-wise. SWA targets wider optima: individual SGD trajectories may oscillate within a narrow valley, but their average often sits in a flatter basin with better test error. SWA differs from EMA in mechanism: EMA tracks every step with exponential decay, while SWA may average only a handful of checkpoints from an explicitly flat region after warmup. Both methods regularize by smoothing the parameter path rather than injecting noise into forward passes. PyTorch does not ship a single EMA module in core nn, but libraries and training loops implement it in a dozen lines around optimizer.step(). When validation loss is noisy but trending down, comparing EMA or SWA weights against raw weights on validation is a low-cost generalization check before deployment.

---

Part 8: Composing the Toolkit — Engineering Practice

No single regularizer fixes all overfitting. A practical stack for a medium-sized classifier might include:

Technique	What it regularizes	Typical cost
Train/val split + early stopping	training duration / path	compute saved
Weight decay (AdamW)	parameter magnitude	slight underfit risk
Dropout	co-adapted activations	lower train accuracy
Label smoothing	logit sharpness	softer train CE
Data augmentation	input memorization	longer epochs to converge
Init (Module 1.3.1)	explosive activations	indirect
Normalization (Module 1.3.4)	internal covariate shift	indirect + sometimes less dropout need

Tune on validation metrics. Adding dropout and weight decay simultaneously may over-regularize; ablate one knob at a time. Plot train and validation loss on the same axes every run. If validation improves when you remove a regularizer, you were over-penalizing. A practical ablation order for a new vision task: (1) establish baseline train/val curves with only early stopping, (2) add augmentation if inputs are images, (3) add weight decay on AdamW, (4) add dropout if gap persists, (5) add label smoothing if logits look over-confident on validation reliability diagrams.

Normalization layers (BatchNorm, LayerNorm — Module 1.3.4) reduce internal drift and carry their own mild regularization via batch noise at train time; they are not substitutes for held-out validation but often let you use slightly higher learning rates and less dropout. Good initialization (Module 1.3.1) prevents early divergence so regularizers can act during meaningful learning rather than during recovery from blow-up. When BatchNorm is present, dropout before the next linear layer is sometimes reduced because normalized activations already vary across batches.

The test set remains sacred: one final evaluation after all choices are frozen. Reporting test numbers while still tweaking augmentation strength is the same leakage as tuning on test loss directly. Document your validation protocol in experiment metadata — split ratios, seed, whether test was touched — so cross-family reviewers and future you can trust the generalization claims.

# Combined example — not a full training script, but idiomatic PyTorch 2.12
model = MLPWithDropout(p=0.3).to(device)
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

# Inside train loop (Module 1.3 pattern):
model.train()
optimizer.zero_grad(set_to_none=True)
loss = criterion(model(x), y)
loss.backward()
optimizer.step()

# Validation + early stopping bookkeeping from Part 4
# DataLoader uses augmentation in train_transform only (Part 6)

When reporting results to stakeholders, separate training metrics (useful for debugging convergence) from validation metrics (useful for model selection) and reserve test metrics for the final acceptance gate. A common anti-pattern in internal demos is showing monotonically improving training accuracy while validation flatlines — this module’s opening scenario is fiction, but the failure mode appears in real organizations that optimize for demo metrics. Honest reporting of the train–validation gap builds trust and saves rework when models reach production.

Choosing regularization strength is not a one-time architectural decision. Transfer learning often starts with a frozen backbone and small head: weight decay on the head only, light dropout, heavy augmentation on the new domain. Full fine-tuning increases capacity utilization and usually demands stronger regularization or early stopping because the backbone can adapt to spurious patterns in a small target dataset. The validation curve tells you which regime you are in within the first few epochs — if validation degrades while train improves on epoch 2, intervene immediately rather than waiting for a 100-epoch schedule to finish.

Hyperparameter search over regularizer strengths should use validation loss or validation accuracy as the objective, never test performance. Random search over (dropout_p, weight_decay, label_smoothing) with ten to twenty trials often finds a workable combination faster than grid search, because regularizer interactions are nonlinear — the best dropout at weight_decay=0 may differ from the best dropout at weight_decay=0.01. Log every trial with seed and split hash so results are reproducible when a reviewer asks why you chose p=0.3 instead of p=0.5.

---

Did You Know?

• Dropout was originally motivated by preventing co-adaptation of feature detectors in deep networks, not by ensemble theory — the ensemble interpretation came later and helped explain why inverted scaling at train time matches test-time averaging (Srivastava et al., 2014, JMLR).
• Label smoothing was popularized in the Inception-v2/v3 line of work as “label smoothing regularization” and became a default trick in large-scale image classification because it consistently improved top-1 accuracy and calibration on ImageNet (Szegedy et al., 2016).
• Early stopping is equivalent to L2 regularization in linear models under certain quadratic approximations of the loss — the number of training iterations and the weight decay coefficient are coupled, which is why you should not tune both blindly without validation curves.
• Mixup and CutMix treat augmentation and label smoothing as one operation by training on convex combinations of examples and their labels, which is why competitive vision recipes often enable them together rather than stacking many other regularizers on top.

---

Common Mistakes

Mistake	Problem	Solution
Validating with model.train()	Dropout and BatchNorm train behavior corrupt metrics	Call model.eval() before validation; use torch.no_grad()
No validation split	Cannot detect overfitting until production	Hold out validation data; never tune on test
Dropout on logits layer	Distorts class scores unpredictably	Dropout after hidden activations only; keep output head dense
Using Adam L2 instead of AdamW for weight decay	Uneven regularization across parameters	Use AdamW with weight_decay= as in Module 1.3.2
Training until train loss = 0	Often maximizes overfitting	Monitor val loss; apply early stopping
Tuning all regularizers on test set	Optimistic bias; false confidence	Tune on validation; test once at end
Forgetting 1/(1−p) scaling when implementing custom dropout	Train/inference mismatch	Use nn.Dropout or scale explicitly

---

Quiz

1. You observe training loss decreasing while validation loss increases after epoch 15. Name this phenomenon and list two interventions from this module that address it.

   Answer

   This is **overfitting** — the model improves on training data while generalization worsens. Interventions include **dropout** (reduce co-adaptation), **weight decay** (penalize large weights), **early stopping** (restore best validation checkpoint), **label smoothing** (reduce logit over-confidence), and **data augmentation** (expand effective training diversity). In practice you would apply one or two at a time, tuning on validation loss rather than training loss.

2. Why does PyTorch nn.Dropout(p=0.5) multiply surviving activations by 2 during training but apply no scaling during evaluation?

   Answer

   This is **inverted dropout**. During training, each unit is kept with probability `1−p`, so the expected activation magnitude is `(1−p)` times the input unless compensated. Multiplying survivors by `1/(1−p)` restores expected magnitude to match inference, when all units are active. At eval, dropout is disabled and activations pass through unchanged — no extra scaling needed. Without inverted dropout, you would multiply by `(1−p)` at test time, which is easy to forget and causes train/test skew.

3. How does label smoothing with ε=0.1 change the target distribution for a 10-class problem when the true label is class 3?

   Answer

   Class 3 receives probability `(1 − ε) + ε/K = 0.9 + 0.01 = 0.91`. Each of the other nine classes receives `ε / K = 0.1 / 10 = 0.01`. Cross-entropy then penalizes logits that are infinitely sharp toward class 3, encouraging the model to maintain modest logits on non-target classes — improving calibration and often generalization compared to hard one-hot targets from Module 1.1.5.

4. You use torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0.01) and read that this applies L2 regularization. Why might Module 1.3.2 recommend AdamW instead for weight decay?

   Answer

   In classic **Adam**, L2 penalty gradients are added to the adaptive gradient before division by root second moments, so parameters with larger historical gradient magnitudes experience **weaker effective regularization**. **AdamW** decouples weight decay: it shrinks weights directly after the Adam update step, applying uniform L2-style decay independent of adaptive scaling. For generalization, AdamW's decoupled decay matches the intended "keep all weights small" regularization view.

5. Early stopping with patience=3 restores the best validation checkpoint. Explain how this acts as implicit regularization even without dropout or weight decay.

   Answer

   Early stopping truncates optimization before the model fully minimizes training loss in regions where validation error rises. It limits **effective model complexity along the optimization trajectory** — similar in spirit to restricting the norm of the solution in linear models. The best checkpoint often lies at an intermediate epoch where bias and variance trade off favorably; continuing training increases variance (memorization) without validation benefit.

6. Why is data augmentation often stronger than dropout for convolutional image classifiers?

   Answer

   Augmentation increases **diversity of inputs** the network must handle — translations, flips, and photometric changes simulate new examples with the same label — directly addressing finite dataset size. Dropout only perturbs internal activations given whatever inputs appear; if inputs are limited, the network can still memorize specific pixels. Augmentation also avoids destroying spatial structure as aggressively as high dropout on conv features. Both can be combined, but augmentation is usually the first lever in vision.

---

Hands-On Exercise

Task: Starting from the Part 1 overfitting demo (500-sample Fashion-MNIST subset, oversized MLP), implement a regularization ablation: baseline, +dropout, +weight decay, +label smoothing, +early stopping. Compare validation accuracy curves across configurations and record which combination yields the best validation peak without touching the test set.

Follow the numbered steps below in order. Copy the Part 1 script into a single runnable file and record train and validation accuracy per epoch for 30 epochs as your baseline with no regularization beyond the Adam optimizer. Then add nn.Dropout(p=0.4) after each ReLU, re-run, and compare epoch-wise train versus validation accuracy — expect a narrower gap and lower peak train accuracy. Switch to AdamW(..., weight_decay=0.01) with dropout disabled for one run to isolate weight decay, noting the validation peak epoch. Combine your best findings: use CrossEntropyLoss(label_smoothing=0.1) with dropout and AdamW together. Finally, implement early stopping with patience=5 and restore best_state, comparing final validation accuracy of the restored checkpoint against the epoch-29 weights.

When you finish all runs, confirm the success criteria below. Each item is a verifiable outcome; if any fails, rerun the range test or verify you selected regularizer strengths on validation data only.

• Baseline shows train accuracy above 0.95 with validation plateau or decline below 0.85.
• At least one regularized configuration improves best validation accuracy by two or more absolute points over baseline.
• Early-stopped checkpoint beats final-epoch checkpoint on validation accuracy when overfitting is present.
• All validation passes call model.eval() before computing metrics.

Use the verification snippet below to print best validation accuracy after each configuration:

# After each configuration, assert you used eval mode for validation
assert not model.training
val_acc = accuracy(val_loader)
print(f"best val_acc={val_acc:.3f}")
# Expect combined reg stack: val_acc often 0.82-0.88 vs baseline 0.78-0.82

---

Key Takeaways

1. Generalization is measured by the train–validation gap, not training loss alone. Overfitting is train improving while validation worsens — diagnose it with curves before reaching for regularizers.
2. Dropout trains an ensemble of subnetworks via inverted dropout (nn.Dropout scales by 1/(1−p) at train time); always toggle model.eval() for validation and deployment.
3. Weight decay penalizes large weights for smoother functions; use AdamW’s weight_decay per Module 1.3.2 rather than assuming L2-in-loss works identically under adaptive optimizers.
4. Early stopping is cheap, implicit regularization — monitor validation loss, keep the best checkpoint, and stop when patience exhausts.
5. Label smoothing softens one-hot targets in CrossEntropyLoss, connecting back to Module 1.1.5’s CE while reducing over-confident logits and improving calibration.
6. Data augmentation is often the highest-leverage regularizer in vision; modern additions include Mixup/CutMix, stochastic depth, DropPath, and EMA/SWA for parameter-path smoothing.
7. Compose regularizers deliberately and tune on validation; normalization (Module 1.3.4) and initialization (Module 1.3.1) interact with the stack — ablate one knob at a time.

---

Sources

• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(1), 1929–1958. https://jmlr.org/papers/v15/srivastava14a.html
• Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. arXiv:1512.00567. https://arxiv.org/abs/1512.00567
• Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep Networks with Stochastic Depth. arXiv:1603.09382. https://arxiv.org/abs/1603.09382
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 7: Regularization. https://www.deeplearningbook.org/contents/regul.html
• Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2023). Dive into Deep Learning — Weight Decay. https://d2l.ai/chapter_linear-regression/weight-decay.html (see also Chapter 5 Dropout: https://d2l.ai/chapter_multilayer-perceptrons/dropout.html)
• PyTorch 2.12 nn.Dropout documentation. https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html
• PyTorch 2.12 CrossEntropyLoss documentation. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
• PyTorch 2.12 torch.optim.AdamW documentation. https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
• Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). Averaging Weights Leads to Wider Optima and Better Generalization. arXiv:1803.05407. https://arxiv.org/abs/1803.05407
• Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. arXiv:1710.09412. https://arxiv.org/abs/1710.09412
• Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. arXiv:1905.04899. https://arxiv.org/abs/1905.04899

---

Next Module

Continue to Normalization Layers — how BatchNorm, LayerNorm, and related techniques stabilize internal activations, interact with dropout and weight decay, and carry their own regularizing effects during training.

---

Learner check

| 1.3.3 | Regularization & Generalization |