Activation Functions in Depth

Цей контент ще не доступний вашою мовою.

Hypothetical scenario: Imagine you are a deep learning researcher in the early 2010s. You have just designed a massive, deeply stacked neural network with 50 layers, intent on solving complex image recognition tasks that have historically stymied shallower models. You painstakingly initialize the weights, load your carefully curated dataset, and begin the training process. Hours turn into days as the compute clusters churn through matrix multiplications. Finally, you evaluate the loss—only to discover that your 50-layer behemoth performs no better than a simplistic linear regression model. The network has failed to capture any of the intricate, nonlinear boundaries in the data. After weeks of debugging, pouring over gradient flows and parameter updates, the startling revelation hits you: you forgot to include activation functions between the layers. Without these crucial mathematical components, the entire stack of linear transformations has collapsed into a single, mathematically equivalent linear operation. Your architectural depth was entirely illusory, buying you no additional representational power while costing an immense amount of computational overhead. This foundational realization highlights one of the most fundamental principles in the architecture of neural networks: depth without nonlinearity is mathematically indistinguishable from a shallow linear model, fundamentally limiting the network’s capacity to learn complex, real-world patterns.

This scenario, while hypothetical, reflects the genuine architectural constraints that govern the design of deep neural networks. The inclusion of activation functions is not merely a stylistic choice or an optional enhancement; it is the absolute mathematical prerequisite for deep learning. Activation functions introduce the critical nonlinearities that allow neural networks to act as universal function approximators, capable of mapping highly complex, non-convex decision boundaries in high-dimensional spaces. Without them, the entire edifice of deep learning collapses into classical linear algebra, unable to solve tasks like computer vision, natural language processing, or complex pattern recognition. In this module, we will explore the profound mathematical implications of activation functions, dissecting their properties, their derivatives, and their impact on the propagation of gradients during the training process.

Learning Outcomes

After you finish this module, you will be able to connect activation choices to gradient flow, implement them in NumPy, and justify hidden-layer versus output-layer selections in a from-scratch network:

Mathematically prove that stacking multiple linear layers without intermediate non-linear activation functions collapses into a single linear transformation, demonstrating the absolute necessity of non-linearities for deep architectures.
Derive the analytical derivatives for classical saturating activation functions (sigmoid and tanh) and modern piecewise/smooth functions (ReLU, Leaky ReLU, GELU, SiLU), explicitly connecting their functional forms to the dynamics of backpropagation.
Implement forward and backward passes for a diverse suite of activation functions in pure NumPy, maintaining rigorous consistency with vector and matrix dimensionalities.
Empirically validate the correctness of analytical derivatives by comparing them against numerical gradients computed via finite differences, establishing a robust testing methodology for mathematical operations.
Strategically select appropriate activation functions for distinct components of a neural network architecture, balancing the need for sparsity, gradient stability, and representational capacity based on established heuristics and empirical evidence.

Why This Module Matters

Understanding activation functions at a fundamental, mathematical level is essential for any serious practitioner of artificial intelligence and machine learning. In the previous modules of this from-scratch arc, we meticulously constructed the forward pass of a neural network, focusing on the mechanics of linear transformations, matrix multiplications, and the propagation of inputs through sequential layers. However, linear transformations alone are fundamentally insufficient for capturing the complexity of real-world data, which is almost entirely non-linear. The role of the activation function is to introduce a non-linear distortion to the feature space, allowing the network to fold, twist, and warp the decision boundary until it can successfully separate complex classes or approximate intricate functions.

Furthermore, the choice of activation function directly dictates the dynamics of gradient flow during backpropagation. The derivative of the activation function serves as a mathematical gateway through which error signals must pass as they travel backward from the output layer to the input layer. If an activation function saturates, its derivative approaches zero, causing the error signal to diminish exponentially—a phenomenon known as the vanishing gradient problem. Conversely, certain activation functions can cause gradients to explode, leading to numerical instability and the complete failure of the training process. By studying the mathematical properties of various activation functions—such as the sigmoid, the hyperbolic tangent (tanh), the Rectified Linear Unit (ReLU), and modern variants like GELU and SiLU—you will gain the ability to purposefully select the appropriate non-linearity for specific architectural components, ensuring robust gradient flow and optimal learning dynamics.

This module will transition you from a passive consumer of deep learning frameworks to an active architect who understands the precise mathematical mechanics governing network behavior. We will rigorously derive the gradients for each activation function, implement them in pure NumPy, and numerically verify their correctness using the finite differences technique established in the first module. This rigorous, empirical validation is critical for ensuring the structural integrity of the micro-framework we are building, as any error in the derivative calculation will inevitably derail the backpropagation algorithm we will construct in subsequent modules.

Saturating activations deserve an extra intuition before we derive them: when a neuron’s pre-activation sits in the flat tail of sigmoid or tanh, its local derivative is tiny, so backpropagation multiplies many small factors and early layers stop learning even though the loss at the output is still large. That saturation picture is what motivates the ReLU family and the smooth modern alternatives in the sections that follow.

Part 1: Why nonlinearity at all

The foundational premise of deep learning is that hierarchical representations—features constructed from simpler features, which in turn are constructed from raw inputs—can capture the underlying structure of complex data. This hierarchical composition is achieved by stacking multiple layers of neurons, with each layer applying a transformation to the output of the preceding layer. However, if the transformation applied by each layer is strictly linear, the entire architectural stack collapses mathematically into a single, functionally equivalent linear layer. This collapse completely neutralizes the benefits of depth, rendering the network incapable of learning complex, non-linear relationships, regardless of how many layers are included or how many parameters are optimized.

To understand this phenomenon rigorously, let us examine the mathematical formulation of a multi-layer linear network. Consider a network with an input vector $x$ of dimensionality $D$ . The first hidden layer applies a linear transformation defined by a weight matrix $W_1$ and a bias vector $b_1$ . The output of this first layer, denoted as $h_1$ , is given by the equation: $h_1 = W_1 x + b_1$

Now, suppose we introduce a second hidden layer that also applies a strictly linear transformation, characterized by a weight matrix $W_2$ and a bias vector $b_2$ . This second layer takes the output of the first layer, $h_1$ , as its input. The output of the second layer, denoted as $h_2$ , is given by: $h_2 = W_2 h_1 + b_2$

If we substitute the expression for $h_1$ into the equation for $h_2$ , we obtain the following expanded form: $h_2 = W_2 (W_1 x + b_1) + b_2$

By distributing the multiplication of $W_2$ across the terms within the parentheses, we can rewrite this equation as: $h_2 = (W_2 W_1) x + (W_2 b_1 + b_2)$

This resulting equation reveals a profound architectural limitation. The term $(W_2 W_1)$ represents the product of two matrices, which is itself just another single matrix, let us denote it as $W_{eq}$ . Similarly, the term $(W_2 b_1 + b_2)$ represents the product of a matrix and a vector added to another vector, which evaluates to a single new vector, let us denote it as $b_{eq}$ . Therefore, the output of the two-layer network can be perfectly represented by the equivalent, single-layer equation: $h_2 = W_{eq} x + b_{eq}$

This mathematical proof demonstrates that any sequence of linear transformations, no matter how deep or how wide, can always be algebraically reduced to a single, equivalent linear transformation. In the context of neural networks, this means that adding more linear layers does not increase the representational capacity of the model; it simply introduces redundant parameters that waste computational resources and complicate the optimization landscape without providing any corresponding benefit in modeling complexity. The network remains fundamentally constrained to learning linear decision boundaries, such as straight lines in two dimensions, flat planes in three dimensions, or hyperplanes in higher-dimensional spaces.

To break this linear equivalence and unleash the true power of deep architectures, we must interleave non-linear activation functions between the linear transformations. Let us introduce a non-linear function, denoted as $\sigma$ , and apply it to the output of the first layer before passing it to the second layer. The equations now become: $z_1 = W_1 x + b_1$ $h_1 = \sigma(z_1)$ $z_2 = W_2 h_1 + b_2$ $h_2 = \sigma(z_2)$

By substituting $h_1$ into the equation for $z_2$ , we get: $z_2 = W_2 \sigma(W_1 x + b_1) + b_2$

Because the function $\sigma$ is non-linear, we cannot algebraically distribute $W_2$ through it. The non-linearity acts as an algebraic barrier, preventing the collapse of the multiple weight matrices into a single equivalent matrix. The inclusion of the non-linear activation function essentially “locks in” the transformation performed by the first layer, allowing the second layer to perform a subsequent, independent transformation on a newly warped and distorted feature space. As we stack more layers, each followed by a non-linear activation, the network progressively bends, folds, and sculpts the feature space, gaining the capacity to approximate highly complex, non-convex functions. This sequence of linear-affine transformations interleaved with point-wise non-linearities is the defining characteristic of deep neural networks, and it is the fundamental mechanism that enables them to solve complex tasks that are completely intractable for classical linear models.

Part 2: Sigmoid & tanh

Historically, the earliest neural networks relied heavily on activation functions that were inspired by biological analogies, specifically the firing rate of biological neurons. These functions were designed to take a continuous range of input values and squash them into a bounded output range, typically between 0 and 1, or between -1 and 1. This “squashing” behavior mimics the saturation of biological neurons, which cannot fire at infinite rates but instead reach a maximum frequency of action potentials. The two most prominent functions in this category are the logistic sigmoid and the hyperbolic tangent, commonly referred to as tanh. While these functions played a crucial role in the early development of deep learning, they suffer from significant mathematical drawbacks that have largely relegated them to specific architectural niches in modern models.

The Logistic Sigmoid

The logistic sigmoid function, often denoted simply as $\sigma(z)$ , maps any real-valued input $z$ to a value strictly in the range $(0, 1)$ . The mathematical definition of the sigmoid function is: $\sigma(z) = \frac{1}{1 + e^{-z}}$

The sigmoid function possesses several appealing properties. It is a smooth, continuous, and everywhere differentiable function, which facilitates the calculation of gradients during backpropagation. Its output can be interpreted as a probability, making it a natural choice for the output layer of binary classification models. For large positive inputs, $e^{-z}$ approaches 0, and the sigmoid output asymptotically approaches 1. For large negative inputs, $e^{-z}$ grows extremely large, and the sigmoid output asymptotically approaches 0. When the input is exactly 0, the output is $0.5$ .

However, the sigmoid function has two major flaws when used as an activation function in the hidden layers of a deep neural network. The first and most critical flaw is the vanishing gradient problem. To understand this, we must examine the derivative of the sigmoid function with respect to its input $z$ . The derivative can be elegantly expressed in terms of the sigmoid function itself: $\frac{d\sigma(z)}{dz} = \sigma(z) \cdot (1 - \sigma(z))$

Let us analyze the behavior of this derivative. The maximum value of the derivative occurs when $z = 0$ , where $\sigma(0) = 0.5$ . At this point, the derivative is $0.5 \cdot (1 - 0.5) = 0.25$ . As the magnitude of the input $z$ increases—either becoming very positive or very negative—the sigmoid output approaches 1 or 0, respectively. In either case, the term $\sigma(z) \cdot (1 - \sigma(z))$ rapidly approaches 0. This means that for input values that are significantly far from zero, the local gradient of the sigmoid function is vanishingly small. During backpropagation, the global gradient is computed by multiplying these local gradients together according to the chain rule. If a network has many layers, and the local gradients at each layer are consistently small numbers (less than 0.25), their product will decay exponentially toward zero as the error signal propagates backward to the earlier layers. Consequently, the weights in the earlier layers receive almost no gradient updates, stalling the learning process entirely.

The second flaw of the sigmoid function is that its outputs are not zero-centered; they are strictly positive. If the outputs of a hidden layer are always positive, then the gradients flowing backward into the weights of that layer will either be all positive or all negative, depending on the upstream gradient. This introduces a zigzagging, highly inefficient trajectory during gradient descent optimization, as the weight updates are forced to move in correlated directions rather than taking the optimal path toward the minimum of the loss function.

The Hyperbolic Tangent (tanh)

The hyperbolic tangent function, denoted as $\tanh(z)$ , is a scaled and shifted version of the sigmoid function. It maps real-valued inputs to the range $(-1, 1)$ , and its mathematical definition is: $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$

The primary advantage of the tanh function over the sigmoid is that it is strictly zero-centered. When the input is 0, the output is 0. This zero-centering property alleviates the zigzagging gradient optimization issues associated with the all-positive outputs of the sigmoid function, generally allowing networks utilizing tanh activations to converge much faster during training. For this reason, tanh is almost universally preferred over the sigmoid function for hidden layers in architectures where saturating non-linearities are desired, such as in Recurrent Neural Networks (RNNs).

The derivative of the tanh function also possesses an elegant form, expressed in terms of its output: $\frac{d\tanh(z)}{dz} = 1 - \tanh^2(z)$

The maximum value of the tanh derivative occurs at $z = 0$ , where $\tanh(0) = 0$ , yielding a local gradient of $1 - 0 = 1$ . This is a significant improvement over the sigmoid’s maximum gradient of 0.25, meaning that error signals can propagate more robustly through tanh layers. However, the tanh function is still fundamentally a saturating activation function. As the magnitude of the input $z$ increases, the tanh output asymptotically approaches 1 or -1, causing the term $1 - \tanh^2(z)$ to rapidly approach 0. Therefore, the tanh function still suffers heavily from the vanishing gradient problem, particularly in very deep networks, where the repeated multiplication of gradients that are less than 1 will inevitably lead to exponential decay. This fundamental limitation spurred the development of non-saturating activation functions that could maintain robust gradient flow regardless of the depth of the network.

Part 3: The ReLU family

The Rectified Linear Unit (ReLU) revolutionized the field of deep learning by providing a devastatingly simple yet incredibly effective solution to the vanishing gradient problem. The widespread adoption of ReLU was one of the primary catalysts that enabled the training of truly deep neural networks, facilitating breakthroughs in computer vision and natural language processing. The fundamental design philosophy behind ReLU is to embrace piecewise linearity, abandoning the complex exponential computations of saturating functions in favor of a simple thresholding operation that preserves gradient flow for positive activations.

The Rectified Linear Unit (ReLU)

The mathematical definition of the ReLU function is elegantly straightforward. It simply outputs the input directly if the input is positive, and it outputs zero if the input is negative. Formally, it is defined as: $\text{ReLU}(z) = \max(0, z)$

Despite its simplicity, ReLU is a non-linear function due to the hard threshold at zero, which is sufficient to break the linear equivalence of stacked layers and allow the network to act as a universal approximator. The true brilliance of ReLU lies in its derivative: $\frac{d\text{ReLU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{if } z \le 0 \end{cases}$

(Note: The derivative at exactly $z=0$ is technically undefined as the function is non-differentiable at this point, but in software implementations, it is conventionally assigned a value of 0.)

For all positive inputs, the local gradient of the ReLU function is exactly 1. This means that as long as a neuron is active (i.e., its pre-activation is positive), the error signal passes through it completely undiminished. There is no exponential decay caused by repeated multiplication of small fractional gradients. This property allows networks utilizing ReLU activations to be exceptionally deep without suffering from the vanishing gradient problem, enabling the optimization of complex, multi-layered architectures. Furthermore, the ReLU function is computationally trivial to evaluate, requiring only a simple comparison operation rather than expensive exponential calculations, which significantly accelerates both the forward and backward passes during training.

However, ReLU introduces a different structural issue known as the “dead ReLU” problem. For any negative input, the derivative of the ReLU function is exactly 0. If a large gradient update causes the weights of a neuron to shift such that its pre-activation is always negative for all inputs in the dataset, that neuron will forever output zero. Because its local gradient is zero, it will never receive any further weight updates during backpropagation; it becomes completely unresponsive and effectively “dies,” ceasing to contribute to the network’s representational capacity. In networks with high learning rates, it is not uncommon for a substantial fraction of the ReLU neurons to die during training, reducing the effective size and power of the model.

To diagnose dead ReLU neurons during training, track the fraction of units whose pre-activations are strictly negative across a mini-batch (or whose activations are exactly zero after ReLU). If that fraction climbs early in training and never recovers, try a lower learning rate, He initialization for the layer feeding the ReLU, or switch to Leaky ReLU so negative inputs still carry a small gradient. The symptom is representational collapse in a slice of the layer, not a mysterious optimizer bug.

Leaky ReLU and Parametric ReLU (PReLU)

To mitigate the dead ReLU problem, researchers developed modified versions of the ReLU function that allow a small, non-zero gradient to flow even when the input is negative. The Leaky ReLU function introduces a small, fixed slope for the negative domain, typically denoted by a hyperparameter $\alpha$ (often set to a small value like 0.01). Its formal definition is: $\text{Leaky\_ReLU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha z & \text{if } z \le 0 \end{cases}$

The derivative of Leaky ReLU is similarly piecewise: $\frac{d\text{Leaky\_ReLU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ \alpha & \text{if } z \le 0 \end{cases}$

By ensuring that the gradient is never exactly zero, Leaky ReLU allows “dead” neurons to potentially recover if the optimization landscape shifts, maintaining the overall capacity of the network. A more advanced variant is the Parametric ReLU (PReLU), which treats the slope $\alpha$ not as a fixed hyperparameter, but as a learnable parameter that is updated via gradient descent during the training process alongside the network’s weights. This allows the network to adaptively learn the optimal degree of “leakiness” for each layer or even for each individual neuron, providing greater flexibility at the cost of a slightly larger parameter count.

Exponential Linear Unit (ELU)

The Exponential Linear Unit (ELU) attempts to combine the advantages of ReLU with the zero-centering properties of the tanh function. ELU is linear for positive inputs, maintaining the non-saturating gradient flow of ReLU, but it smoothly curves down to a negative saturation value (determined by a parameter $\alpha$ ) for negative inputs. Its definition is: $\text{ELU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha (e^z - 1) & \text{if } z \le 0 \end{cases}$

The smooth curvature in the negative domain pushes the mean activation closer to zero, which can speed up convergence and improve generalization. The derivative is also continuous (assuming $\alpha=1$ ): $\frac{d\text{ELU}(z)}{dz} = \begin{cases} 1 & \text{if } z > 0 \\ \text{ELU}(z) + \alpha & \text{if } z \le 0 \end{cases}$

While ELU offers robust empirical performance, its reliance on the exponential function makes it computationally more expensive than the simpler piecewise variants of the ReLU family, presenting a trade-off between optimization efficiency and computational throughput.

When you implement these activations in NumPy or in your micro-framework, watch numerical hygiene: clip extreme inputs before exp in sigmoid and SiLU to avoid overflow, use the tanh-based GELU approximation in training loops unless you truly need the exact erf form, and remember that ReLU’s derivative at $z=0$ is a convention (0 or 1)—frameworks pick one, but finite-difference checks should avoid testing exactly at zero if you want bit-identical comparisons.

Part 4: Modern smooth activations

As the deep learning community moved toward increasingly massive and complex architectures, particularly the Transformer models that dominate modern natural language processing and computer vision, researchers discovered that the sharp, non-differentiable corner of the ReLU function at $z=0$ could hinder the optimization process. The hard piecewise linearity can introduce abrupt changes in the gradient landscape, making it difficult for advanced optimization algorithms to traverse the parameter space efficiently. To address this, modern architectures have increasingly adopted smooth, continuous non-linearities that approximate the overall shape of ReLU but eliminate the sharp discontinuity, facilitating smoother gradient descent dynamics.

Gaussian Error Linear Unit (GELU)

The Gaussian Error Linear Unit (GELU), introduced by Hendrycks and Gimpel (arXiv:1606.08415), has become the de facto standard activation function in state-of-the-art Transformer architectures, including BERT, GPT-3, and Vision Transformers. The intuition behind GELU is to merge the concepts of activation functions and regularization (specifically dropout) by multiplying the input by the cumulative distribution function (CDF) of the standard normal distribution. In dropout, inputs are randomly multiplied by 0 or 1 with a certain probability. In GELU, the input is multiplied by a value between 0 and 1, where the value is determined by the probability that a random variable drawn from a standard normal distribution is less than the input $z$ .

The exact mathematical definition of the GELU function is: $\text{GELU}(z) = z \cdot \Phi(z) = z \cdot \frac{1}{2} \left[ 1 + \text{erf}\left( \frac{z}{\sqrt{2}} \right) \right]$

where $\Phi(z)$ is the standard normal CDF and $\text{erf}$ is the error function.

Because computing the error function is relatively slow, a highly accurate and faster analytical approximation involving the hyperbolic tangent is widely used in practice, and this approximation is what is fundamentally implemented in most major deep learning frameworks when optimizing for speed: $\text{GELU}_{\text{approx}}(z) = 0.5 \cdot z \cdot \left( 1 + \tanh\left[ \sqrt{\frac{2}{\pi}} \left( z + 0.044715 \cdot z^3 \right) \right] \right)$

GELU exhibits a non-monotonic “bump” slightly below zero; it becomes slightly negative before asymptotically approaching zero as the input decreases. This smooth curvature allows for richer gradient information to flow when the input is near zero, avoiding the dead neuron problem while maintaining the non-saturating behavior of ReLU for large positive inputs. The smoothness of GELU provides a more continuous and predictable optimization landscape, which is critical for training massive models with billions of parameters.

SiLU (Swish)

The Sigmoid Linear Unit (SiLU) was introduced by Elfwing et al. (arXiv:1702.03118) as $x \cdot \sigma(x)$ for reinforcement-learning networks. The same functional form was later rediscovered and named Swish by Ramachandran et al. (arXiv:1710.05941) through architecture search; practitioners often use the names interchangeably. GELU (above) is a separate activation from Hendrycks & Gimpel—it is not the same as SiLU/Swish. SiLU is constructed by multiplying the input by the sigmoid of the input. Its mathematical definition is: $\text{SiLU}(z) = z \cdot \sigma(z) = \frac{z}{1 + e^{-z}}$

Like GELU, SiLU is a smooth, non-monotonic function that resembles ReLU but lacks the sharp discontinuity at zero. It also exhibits a small negative region before saturating at zero for large negative inputs. The derivative of SiLU is particularly interesting, as it can be expressed in terms of the function itself and the sigmoid function: $\frac{d\text{SiLU}(z)}{dz} = \sigma(z) + z \cdot \sigma(z) \cdot (1 - \sigma(z)) = \text{SiLU}(z) + \sigma(z) \cdot (1 - \text{SiLU}(z))$

SiLU has demonstrated empirical superiority over ReLU in numerous deep learning tasks, particularly in deep convolutional neural networks (such as EfficientNet) and modern variants of the Transformer architecture (such as LLaMA). The smooth, continuous nature of SiLU enables more robust optimization, allowing networks to converge to deeper and more optimal local minima. While slightly more computationally intensive than ReLU due to the evaluation of the exponential function, the performance gains often justify the overhead, making SiLU a premier choice for contemporary deep learning systems.

Part 5: Derivatives table + NumPy

To truly integrate these mathematical concepts into our micro-framework, we must translate the analytical formulas into robust, vectorized NumPy code. Furthermore, in a from-scratch environment, we cannot rely on faith; we must empirically verify that our analytical derivatives exactly match the true geometric gradients. We achieve this by comparing our calculated derivatives against numerical gradients computed using the finite differences technique, which approximates the derivative by measuring the change in the function’s output when the input is perturbed by a minuscule amount, $\epsilon$ .

The finite difference approximation for the derivative of a function $f$ at point $z$ is: $f'(z) \approx \frac{f(z + \epsilon) - f(z - \epsilon)}{2\epsilon}$

The table below collects every activation in this module with its derivative with respect to pre-activation $z$ . Where $A$ denotes the forward output $A = f(z)$ , the sigmoid and tanh rows use the compact forms you already derived.

Activation	Forward $A = f(z)$	Derivative $f'(z)$
Sigmoid	$\sigma(z)$	$\sigma(z)(1-\sigma(z)) = A(1-A)$
Tanh	$\tanh(z)$	$1 - \tanh^2(z) = 1 - A^2$
ReLU	$\max(0, z)$	$1$ if $z > 0$ ; $0$ if $z \le 0$
Leaky ReLU	$z$ if $z>0$ else $\alpha z$	$1$ if $z > 0$ ; $\alpha$ if $z \le 0$
ELU	$z$ if $z>0$ else $\alpha(e^z-1)$	$1$ if $z > 0$ ; $\text{ELU}(z)+\alpha$ if $z \le 0$
GELU (approx)	$0.5 z(1 + \tanh(\cdots))$	Chain rule on the tanh approximation (see code below)
SiLU	$z \cdot \sigma(z)$	$\sigma(z) + z\,\sigma(z)(1-\sigma(z))$

Below is a comprehensive Python script that implements the forward and backward passes for our discussed activation functions, culminating in a rigorous numerical verification test.

import numpy as np

# 1. Sigmoid
def sigmoid_forward(z):
    # np.clip to avoid overflow in exp for large negative values
    z_clipped = np.clip(z, -500, 500)
    return 1.0 / (1.0 + np.exp(-z_clipped))

def sigmoid_backward(z):
    sig = sigmoid_forward(z)
    return sig * (1.0 - sig)

# 2. Tanh
def tanh_forward(z):
    return np.tanh(z)

def tanh_backward(z):
    return 1.0 - np.tanh(z)**2

# 3. ReLU
def relu_forward(z):
    return np.maximum(0, z)

def relu_backward(z):
    return (z > 0).astype(float)

# 4. Leaky ReLU
def leaky_relu_forward(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

def leaky_relu_backward(z, alpha=0.01):
    return np.where(z > 0, 1.0, alpha)

# 5. ELU
def elu_forward(z, alpha=1.0):
    return np.where(z > 0, z, alpha * (np.exp(z) - 1.0))

def elu_backward(z, alpha=1.0):
    out = elu_forward(z, alpha)
    return np.where(z > 0, 1.0, out + alpha)

# 6. GELU (Approximate form)
def gelu_forward(z):
    inner = np.sqrt(2.0 / np.pi) * (z + 0.044715 * z**3)
    return 0.5 * z * (1.0 + np.tanh(inner))

def gelu_backward(z):
    # Utilizing the chain rule for the approximate GELU derivative
    inner = np.sqrt(2.0 / np.pi) * (z + 0.044715 * z**3)
    tanh_inner = np.tanh(inner)

    # Derivative of the inner term
    inner_deriv = np.sqrt(2.0 / np.pi) * (1.0 + 3.0 * 0.044715 * z**2)

    # Derivative of tanh(inner)
    tanh_deriv = 1.0 - tanh_inner**2

    # Applying product rule to 0.5 * z * (1 + tanh(inner))
    term1 = 0.5 * (1.0 + tanh_inner)
    term2 = 0.5 * z * (tanh_deriv * inner_deriv)

    return term1 + term2

# 7. SiLU (Swish)
def silu_forward(z):
    return z * sigmoid_forward(z)

def silu_backward(z):
    sig = sigmoid_forward(z)
    return sig + z * sig * (1.0 - sig)

# ==========================================
# Finite Differences Gradient Check
# ==========================================
def check_gradients():
    print("Running Finite Differences Gradient Checks...\n")
    np.random.seed(42)

    # Generate random input array, avoiding values exactly at zero for ReLU
    z = np.random.randn(5, 5) * 2
    epsilon = 1e-7

    functions = {
        "Sigmoid": (sigmoid_forward, sigmoid_backward),
        "Tanh": (tanh_forward, tanh_backward),
        "ReLU": (relu_forward, relu_backward),
        "Leaky ReLU": (leaky_relu_forward, leaky_relu_backward),
        "ELU": (elu_forward, elu_backward),
        "GELU": (gelu_forward, gelu_backward),
        "SiLU": (silu_forward, silu_backward)
    }

    all_passed = True

    for name, (forward_fn, backward_fn) in functions.items():
        # Analytical gradient
        analytical_grad = backward_fn(z)

        # Numerical gradient (Central difference)
        z_plus = z + epsilon
        z_minus = z - epsilon
        f_plus = forward_fn(z_plus)
        f_minus = forward_fn(z_minus)
        numerical_grad = (f_plus - f_minus) / (2.0 * epsilon)

        # Calculate maximum absolute difference
        max_diff = np.max(np.abs(analytical_grad - numerical_grad))

        status = "PASS" if max_diff < 1e-5 else "FAIL"
        print(f"[{status}] {name:12s} | Max Difference: {max_diff:.8e}")
        if status == "FAIL":
            all_passed = False

    print(f"\nOverall Status: {'SUCCESS' if all_passed else 'FAILURE'}")

if __name__ == "__main__":
    check_gradients()

When you execute this code, the shapes of all input arrays $z$ and the returned arrays representing the derivatives will perfectly match, maintaining strict dimensional consistency. The finite difference check will confirm that the analytical implementations exactly mirror the true mathematical gradients, providing absolute confidence in the foundational integrity of the micro-framework.

Part 6: Hidden vs output activations

A critical architectural distinction in deep learning is the selection of activation functions for the hidden layers versus the output layer. The hidden layers form the internal representational machinery of the network, and their primary mandate is to facilitate deep feature extraction while preserving robust gradient flow throughout the backpropagation process. Therefore, the activation functions utilized in hidden layers—such as ReLU, GELU, and SiLU—are explicitly chosen for their non-saturating properties, their computational efficiency, and their ability to mitigate the vanishing gradient problem, allowing the network to successfully train vast numbers of parameters across many hierarchical levels.

The output layer, however, has an entirely different responsibility. It must translate the abstract, high-dimensional representations generated by the hidden layers into a concrete, interpretable format that aligns mathematically with the chosen loss function and the specific objective of the machine learning task. The output activation function is essentially a formatting layer, coercing raw, unbounded numerical scores—often referred to as logits—into a structured probability distribution or a specific constrained range.

For instance, in a binary classification task, the network’s goal is to predict the probability that a given input belongs to the positive class. Probabilities must, by mathematical definition, reside strictly within the continuous range $[0, 1]$ . The ReLU function, which outputs values from $[0, \infty)$ , is wholly inappropriate for this task. Instead, we must employ the logistic sigmoid function at the output layer, as it mathematically guarantees that any raw logit, ranging from $(-\infty, \infty)$ , will be squashed perfectly into a valid probability score between $0$ and $1$ . The saturating nature of the sigmoid, which is detrimental in hidden layers, becomes an absolute necessity in the output layer to enforce this probabilistic constraint.

Similarly, in multi-class classification scenarios where the model must assign an input to exactly one of $K$ mutually exclusive categories, the output must form a valid categorical probability distribution. All $K$ output probabilities must be positive and they must sum precisely to $1.0$ . This requirement is satisfied by the Softmax function, which takes a vector of $K$ raw logits, exponentiates them to ensure positivity, and then normalizes them by dividing by the sum of all exponentiated logits. We will explore the Softmax function in meticulous mathematical detail in the upcoming Module A5, where we directly couple it with the cross-entropy loss function to derive the highly efficient gradient properties that drive multi-class learning. For the context of this module, the fundamental principle is that output activations are governed by the requirements of the task and the loss function, whereas hidden activations are governed by the requirements of optimization and gradient flow.

Part 7: Selection heuristics

Given the myriad of activation functions available, selecting the correct one for a novel architecture can appear daunting. However, empirical research and theoretical understanding have coalesced into a set of robust, practical heuristics that guide the modern deep learning practitioner. These are not arbitrary rules, but engineered decisions based on balancing computational complexity, gradient stability, and representational capacity.

1. Default to ReLU for simplicity and speed. If you are rapidly prototyping a standard Multilayer Perceptron (MLP) or a basic Convolutional Neural Network (CNN), start with the standard ReLU function. Its piecewise linear nature makes it the fastest function to evaluate, and its derivative of exactly $1.0$ for positive inputs provides excellent baseline resistance to the vanishing gradient problem. It establishes a strong performance benchmark with minimal computational overhead.

2. Upgrade to GELU or SiLU for deep, complex architectures. When training extremely deep networks, particularly modern architectures like Vision Transformers or Large Language Models, the smooth, continuous gradients provided by GELU or SiLU become crucial. The lack of a sharp discontinuity at zero allows advanced optimizers like Adam to navigate the loss landscape more effectively, often yielding slightly faster convergence and better terminal performance. The marginal increase in computational cost (due to the exponential or error functions) is usually vastly outweighed by the improvements in model quality. If you are aiming for state-of-the-art performance, GELU or SiLU should be your primary non-linearities.

3. Use Leaky ReLU to diagnose dead neurons. If you observe that your network utilizing standard ReLU activations is experiencing stunted learning or lower-than-expected capacity, you may be suffering from the dead ReLU problem due to high learning rates or unfortunate weight initialization. In this scenario, switching to a Leaky ReLU or Parametric ReLU (PReLU) can serve as a powerful diagnostic and remedial tool. The small negative slope ensures that gradients can still flow through inactive neurons, potentially reviving them and restoring the network’s learning capability.

4. Confine Sigmoid and Tanh to specific structural niches. The classical sigmoid function should almost never be used as a hidden layer activation in modern architectures due to its severe vanishing gradient issues. Its use is strictly reserved for the output layer of binary classification models, or within specialized gating mechanisms where a value must be explicitly constrained between 0 and 1 (such as the forget gate in an LSTM). The tanh function, while zero-centered and slightly better than sigmoid, is also largely deprecated for general hidden layers in MLPs and CNNs. However, it remains a standard choice for the hidden state updates within classical Recurrent Neural Networks (RNNs) and can be useful when the network outputs must explicitly map to the range $[-1, 1]$ .

Part 8: Extend the micro-framework

To solidify our progress and prepare for the derivation of backpropagation in Module A6, we will now extend our ongoing nn.py micro-framework. We will abstract the concept of an activation function into a unified interface, ensuring that our future network components can seamlessly swap between ReLU, GELU, or any other non-linearity without rewriting the underlying structural logic.

In your nn.py file, construct a base Activation class, and then implement the specific functions as derived subclasses. This object-oriented approach encapsulates the forward pass logic and the storage of intermediate inputs necessary for the subsequent backward pass.

# Extending nn.py with Activation primitives
import numpy as np

class Activation:
    def __init__(self):
        self.inputs = None

    def forward(self, x):
        raise NotImplementedError

    def backward(self, grad_output):
        raise NotImplementedError

class ReLU(Activation):
    def forward(self, x):
        # Store the input for use in the backward pass
        self.inputs = x
        return np.maximum(0, x)

    def backward(self, grad_output):
        # The local gradient is 1 where input > 0, else 0
        # By the chain rule, we multiply the local gradient by the incoming grad_output
        local_grad = (self.inputs > 0).astype(float)
        return grad_output * local_grad

class Tanh(Activation):
    def forward(self, x):
        # We can store the output directly, as the derivative
        # is expressed elegantly in terms of the output
        self.output = np.tanh(x)
        return self.output

    def backward(self, grad_output):
        # local gradient = 1 - tanh^2(x)
        local_grad = 1.0 - self.output**2
        return grad_output * local_grad

This structural enhancement is a critical stepping stone. By explicitly storing the inputs (or outputs) during the forward phase, the backward phase has immediate access to the necessary state to compute the exact analytical local gradient, which is then multiplied by the upstream grad_output dictated by the chain rule. This elegant flow of tensors forms the absolute bedrock of the backpropagation algorithm that we will construct from scratch in Module A6.

Did You Know?

The biological origins: The sigmoid function was originally favored because it closely modeled the all-or-nothing firing rate of biological neurons observed in neuroscience. However, it turns out the brain is far more complex, and artificial neural networks are fundamentally mathematical optimization engines, not accurate biological simulations.
The GELU connection to Dropout: The GELU activation was heavily inspired by the Dropout regularization technique. Instead of multiplying inputs by a discrete 0 or 1, GELU multiplies the input by a probabilistic expectation, smoothing out the regularizing effect and integrating it directly into the non-linearity.
Hardware optimization: On GPUs and TPUs, ReLU is often fused into GEMM kernels as a simple max(0, x) comparison—far cheaper per element than the exp() calls required by sigmoid, tanh, or SiLU, which is one reason ReLU dominated vision models before Transformers adopted GELU.
Two names, one curve: SiLU (Elfwing et al., 2017) and Swish (Ramachandran et al., 2017) refer to the same $x\sigma(x)$ non-linearity; papers and frameworks may use either label even though the papers arrived independently months apart.

Common Mistakes

Mistake	Problem	Solution
Using Sigmoid in Deep Hidden Layers	Rapidly induces the vanishing gradient problem, causing early layers to stop learning entirely as the gradients exponentially decay to zero.	Swap to ReLU or GELU for all hidden layers; restrict sigmoid usage exclusively to the output layer for binary probability generation.
Ignoring the Dead ReLU Problem	Utilizing excessively large learning rates can cause massive weight updates that push neuron pre-activations permanently into the negative domain, effectively “killing” large portions of the network capacity.	Lower the learning rate, implement careful weight initialization techniques (like He initialization), or switch to Leaky ReLU to allow gradient flow for negative inputs.
Applying Softmax in Hidden Layers	Softmax forces the activations of a layer to sum to 1, creating a severe bottleneck and enforcing an artificial competition between neurons that destroys independent feature extraction.	Never use Softmax in hidden layers; it is strictly a normalization function designed for interpreting mutually exclusive probabilities at the final output layer.
ReLU on the output of regression	ReLU clips all negative predictions to zero, which is wrong when targets can be negative or when you need an unbounded real-valued output.	Use a linear (identity) output activation for regression; reserve ReLU/GELU for hidden feature layers.
Mixing activation and loss	Applying sigmoid in the hidden stack and again at the output for binary classification double-squashes logits and worsens vanishing gradients.	Keep saturating squashes at the output head only; use ReLU, GELU, or SiLU between hidden layers.
Skipping gradient checks on GELU	The approximate GELU backward is easy to implement incorrectly because of nested tanh and polynomial terms.	Run a finite-difference check (as in Part 5) before trusting your `backward` in `nn.py`.
Treating dead ReLU as random noise	A layer where most units output zero forever is often a training-configuration issue, not bad luck.	Monitor active-unit fraction, reduce learning rate, fix initialization, or adopt Leaky ReLU.

Quiz

Why does stacking multiple linear layers without activation functions fail to increase a network’s representational capacity?

Answer
Because the composition of multiple linear transformations is mathematically equivalent to a single linear transformation. Without a non-linear activation function to break this equivalence, the network can only learn linear decision boundaries, regardless of its depth.
Explain the mathematical cause of the vanishing gradient problem in the sigmoid activation function.

Answer
The derivative of the sigmoid function, $\sigma(z)(1 - \sigma(z))$, has a maximum value of 0.25 at $z=0$ and rapidly approaches 0 for large positive or negative inputs. During backpropagation, these small local gradients are multiplied together across layers. If many layers are present, the repeated multiplication of values less than 1 causes the global gradient to decay exponentially towards zero, stalling the learning process in earlier layers.
How does the derivative of the ReLU function prevent the vanishing gradient problem for active neurons?

Answer
For any positive input, the local gradient of the ReLU function is exactly $1.0$. Therefore, when the chain rule is applied during backpropagation, the error signal passes through active ReLU neurons completely undiminished, preventing the exponential decay associated with saturating functions.
What is the primary architectural difference in the application of ReLU versus Sigmoid in modern deep learning models?

Answer
ReLU (and its variants like GELU) are universally applied in the hidden layers to facilitate robust feature extraction and prevent vanishing gradients. Sigmoid is almost exclusively restricted to the output layer of binary classification models to mathematically squash the final logit into a valid probability range of $[0, 1]$.
How do SiLU and GELU differ in origin, and why are both considered “smooth ReLU-like” activations?

Answer
GELU comes from Hendrycks & Gimpel (Gaussian CDF weighting); SiLU/Swish is $x\sigma(x)$ from Elfwing et al. and independently from Ramachandran et al. Both are smooth, non-monotonic near zero, and avoid the hard ReLU corner, which tends to help deep Transformers and CNNs optimize more smoothly than plain ReLU.
What practical signal suggests you should switch from ReLU to Leaky ReLU mid-project?

Answer
A rising fraction of neurons with zero activation and zero gradient on every mini-batch—especially right after aggressive learning-rate changes—indicates dead ReLU units. Leaky ReLU restores a small negative-slope gradient so those units can recover instead of staying permanently inactive.
How do you empirically validate analytical activation derivatives with numerical finite-difference gradients?

Answer
Perturb each input coordinate by a tiny $\epsilon$, evaluate the activation forward pass at $z+\epsilon$ and $z-\epsilon$, and form the central-difference estimate. Compare that numerical gradient to your analytical `backward` output; agreement within about $10^{-5}$ confirms the implementation before you chain activations into full backprop in Module A6.

Hands-On Exercise

Task: Extend the Activation interface in your nn.py micro-framework to include the modern GELU activation function using the approximate formulation. Work through these steps in order: (1) create class GELU(Activation) inheriting from your base class; (2) implement forward with the np.tanh approximation from Part 5 and store tensors needed for backward; (3) implement backward using the product and chain rules from the finite-difference script; (4) run forward and backward on a dummy array to confirm shapes match. You have succeeded when all of the following hold:

The forward method correctly computes the approximate GELU.
The backward method returns an array with the same shape as grad_output.
You empirically validate analytical GELU derivatives: numerical finite-difference gradients match backward within 1e-5.

Run this verification snippet after implementation:

# Assuming your class is implemented:
layer = GELU()
x = np.array([[-2.0, 0.0, 2.0]])
out = layer.forward(x)
print(f"Forward Output: {out}")
# Expected approximate output: [-0.045, 0.0, 1.954]

dummy_grad = np.ones_like(x)
grad = layer.backward(dummy_grad)
print(f"Backward Output: {grad}")
# Expected approximate gradients matching the finite difference check

Key Takeaways

Depth in neural networks is completely useless without non-linear activation functions; without them, deep networks collapse mathematically into simple linear regression models.
The sigmoid and tanh functions suffer from the vanishing gradient problem due to their saturating nature and derivatives that are strictly less than 1 (or quickly approach 0), making them unsuitable for deep hidden layers.
The ReLU function revolutionized deep learning by providing a derivative of exactly 1 for positive inputs, maintaining robust gradient flow and enabling the training of massive, multi-layered architectures.
Modern architectures frequently employ smooth, non-monotonic functions like GELU and SiLU to provide continuous gradient landscapes, facilitating more effective optimization by advanced algorithms.
The choice of activation function is strictly compartmentalized: non-saturating functions for hidden layer feature extraction, and specific saturating functions (like Sigmoid or Softmax) for formatting output probabilities.

Sources

Learner check

Depth without nonlinearity collapses to a single linear map; saturating tails shrink gradients, while ReLU and smooth variants like GELU preserve signal for deep stacks — see Activation Functions in Depth.

Next Module

The next module in this from-scratch arc is A5 — Loss Functions & Output Heads. Continue the sequence from the Deep Learning section index, where new modules appear as the wave lands.