Activation Functions and GELU

Activation Functions and GELU: The Bend That Makes Depth Work

A note on tone

Like my other notes, this is my own understanding written in plain words. For the full picture, take a proper deep learning course. Think of this as a friendly map, not the textbook.

Series: Neurons, Weights and Bias Explained → Neural Networks Explained → Activation Functions and GELU (this note) → Transformers Explained → Multi-Head Attention Explained.

In Neural Networks Explained we saw the punchline: without a bend after each layer, a hundred layers collapse into one. That bend is the activation function. It takes the neuron's raw sum $z$ and transforms it:

a = f (z)

It looks like a tiny detail, but choosing it well changed what neural networks could do. Let me walk through the main ones, ending at GELU, the one inside GPT-style models.

What makes a good activation?

Non-linear. Otherwise there is no point (see the collapse argument).
Cheap to compute. It runs billions of times.
Gives useful gradients. Training pushes the error signal back through $f$ . If its slope is almost zero, learning stalls.

The four classics

Sigmoid

σ (z) = \frac{1}{1 + e^{- z}}

Squeezes any number into 0 to 1, which reads nicely as a probability. It was the default for decades.

Problem: for large positive or negative $z$ , the curve is nearly flat, so the slope is almost zero. Stack several layers and the learning signal shrinks toward nothing. This is the vanishing gradient problem. Today sigmoid is mostly kept for the final "yes or no" output and for gates inside LSTMs.

Tanh

\tanh (z) = \frac{e^{z} - e^{- z}}{e^{z} + e^{- z}}

A sigmoid stretched to run from −1 to 1 and centered on zero, which helps. It still flattens out at both ends, so it still suffers from vanishing gradients. It lived on in older RNNs.

ReLU

ReLU (z) = max (0, z)

Dead simple: negative becomes 0, positive passes straight through. Its slope is exactly 1 for positive inputs, so gradients flow freely, and it is very cheap. ReLU made training deep networks practical around 2012 and dominated for years.

Problem: a neuron whose input is always negative outputs 0 with slope 0 forever, so it never learns again. This is the dying ReLU. Variants like Leaky ReLU ( $max (0.01 z, z)$ ) leave a tiny slope for negatives to avoid it.

GELU: a smoother ReLU

GELU (Gaussian Error Linear Unit, Hendrycks and Gimpel, 2016) is the activation used in BERT, GPT-2 and GPT-3.

GELU (x) = x \cdot Φ (x)

Here $Φ (x)$ is the chance that a standard bell-curve random number is smaller than $x$ . So GELU means: take the input $x$ and multiply it by the probability that it "deserves to pass".

For a big positive $x$ , $Φ (x) \approx 1$ , so the output is about $x$ .
For a big negative $x$ , $Φ (x) \approx 0$ , so the output is about 0.
Near zero, $Φ$ is in between, so it fades in smoothly instead of snapping.

Compare it with ReLU (the dashed line). ReLU has a sharp corner at zero. GELU rounds it off, and even dips slightly below zero (to about −0.17 at $x \approx - 0.75$ ) before returning to zero. That means a mildly negative input is not completely thrown away, and the smooth slope gives nicer gradients everywhere.

Some numbers

$x$	ReLU	GELU	Sigmoid	Tanh
−2	0	−0.045	0.119	−0.964
−1	0	−0.159	0.269	−0.762
0	0	0	0.5	0
1	1	0.841	0.731	0.762
2	2	1.955	0.881	0.964

At $x = 1$ , ReLU passes the whole thing but GELU passes about 84% of it. At $x = - 1$ , ReLU gives 0 but GELU gives a small negative. It behaves like a soft, probabilistic version of a gate.

In code

Exact version, and the fast approximation used in the original GPT-2 code:

import math

def gelu(x):                       # exact
    return 0.5 * x * (1 + math.erf(x / math.sqrt(2)))

def gelu_tanh(x):                  # approximation
    return 0.5 * x * (1 + math.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * x**3)))

for v in [-2, -1, 0, 1, 2]:
    print(v, round(gelu(v), 3), round(gelu_tanh(v), 3))

The two agree to about three decimals. Deep learning libraries include both (torch.nn.GELU, with an option for the tanh version).

Where does it live in a transformer?

In the feed-forward part of every transformer block: $Linear \to GELU \to Linear$ . You'll see it again in Transformers Explained.

What newer models use

GELU is not the end of the story. Many recent large models use a relative called SiLU (also called Swish):

SiLU (x) = x \cdot σ (x)

Same "input times a gate" idea, with a sigmoid as the gate. Combined with an extra learned gate (a GLU variant), it becomes SwiGLU, used in models like Llama and PaLM. So the modern trend is: smooth, gate-like bends.

A special one: Softmax

Softmax is not applied to one neuron. It takes a whole list of scores and turns them into probabilities that sum to 1:

softmax (z_{i}) = \frac{e^{z_{i}}}{\sum_{j} e^{z_{j}}}

It is used at the very end to pick one word out of a vocabulary, and inside attention to decide how much to listen to each word. We will meet it in Transformers Explained.

Who uses what

Key takeaway

An activation function is the bend that stops layers from collapsing into one. Sigmoid and tanh squash but fade out; ReLU is simple but has a hard corner; GELU smooths that corner by multiplying the input by the chance it should pass. Small change, but it trains a little better, and it sits inside every GPT-style block.

Try it yourself

Plot ReLU and GELU on the same axes with matplotlib and find the dip
Compute GELU(0.5) by hand using the erf formula
Train the XOR net from Neural Networks Explained with ReLU, Tanh and GELU. Compare
Take the derivative of sigmoid ( $σ (1 - σ)$ ) and see why it is at most 0.25
Try SiLU ( $x \cdot σ (x)$ ) and check how close it is to GELU

Next: Transformers Explained