Neural Networks: Many Neurons, One Big Function

In the last note we saw that one neuron can only draw a straight line. Real problems (faces, speech, language) are not separated by straight lines. The fix is beautifully simple: use lots of neurons, in layers.

Layers

A neural network is neurons arranged in columns called layers:

• Input layer: just holds your numbers (pixels, word vectors...).
• Hidden layers: neurons that find useful patterns. "Hidden" only means we don't directly see them.
• Output layer: the answer (a class, a number, the next word).

In a fully connected layer, every neuron takes input from every neuron in the previous layer.

This little network has 3 inputs, 4 hidden neurons and 2 outputs. Count the dials: the first layer has $3\times 4$ weights plus 4 biases, and the second has $4\times 2$ weights plus 2 biases. That is 26 learnable numbers. GPT-scale models have hundreds of billions, but they are counted the same way.

The forward pass: it is just matrix multiplication

Here is the part that connects to Matrix Multiplication Explained. All the neurons in a layer do their weighted sum at the same time. Stack the weights of each neuron into a row, and the entire layer becomes one matrix multiplication:

Look at the shapes in the picture. $W_1$ is $4\times 3$ because there are 4 neurons each looking at 3 inputs. Multiply by a $3\times 1$ input and you get $4\times 1$: one number per hidden neuron. Then $W_2$ ($2\times 4$) turns that into the 2 outputs.

import numpy as np

def forward(x, W1, b1, W2, b2):
    h = np.maximum(0, W1 @ x + b1)   # layer 1: multiply, add, bend (ReLU)
    return W2 @ h + b2               # layer 2: multiply, add

W1, b1 = np.random.randn(4, 3), np.zeros(4)
W2, b2 = np.random.randn(2, 4), np.zeros(2)
print(forward(np.array([1.0, 0.5, -1.0]), W1, b1, W2, b2))

And if you feed a whole batch of 1000 examples, the vectors become a $3\times 1000$ matrix and it is still a single multiplication. This is why GPUs are so good at this.

Why the bend matters

Look at the middle of each layer: the little f. That is the activation function, and it is not optional.

Without it, two layers give $W_2(W_{1}x)=(W_2{W}_1)x$. That is just one matrix in disguise. A hundred layers would still be equivalent to one layer, and one layer can only draw a straight line. The bend after every layer lets the next layer build something genuinely new on top. Depth only helps because of the bend. The next note goes deep on this: Activation Functions and GELU.

How a network learns

At the start, the weights are random, so the output is garbage. Learning is a loop of four steps:

1. Forward pass: run the input through and get a guess.
2. Loss: compare the guess to the right answer, and turn "how wrong" into a single number. For predicting numbers, mean squared error is common. For classes, cross-entropy is.
3. Backward pass (backpropagation): use the chain rule from calculus to work out how much each weight contributed to the error. This is the blame step. It was popularized by Rumelhart, Hinton and Williams in 1986.
4. Update: nudge every weight a little in the direction that shrinks the loss:

The $\eta$ is the learning rate, the size of the nudge. Too big and training jumps around and blows up. Too small and it takes forever. This method is called gradient descent: imagine standing on a hilly landscape in fog and always stepping downhill.

In practice you don't use all the data for each nudge. You use small random batches, and one pass over all the data is an epoch.

A complete tiny example

XOR is the classic problem one neuron cannot solve (no straight line separates it), but a tiny network can:

import torch, torch.nn as nn

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

model = nn.Sequential(nn.Linear(2, 4), nn.Tanh(), nn.Linear(4, 1), nn.Sigmoid())
opt = torch.optim.Adam(model.parameters(), lr=0.05)
loss_fn = nn.BCELoss()

for epoch in range(500):
    opt.zero_grad()
    loss = loss_fn(model(X), y)   # forward + loss
    loss.backward()               # backpropagation
    opt.step()                    # update the weights

print(model(X).round())           # usually [[0],[1],[1],[0]]

Things that go wrong

• Overfitting: the network memorizes the training examples instead of learning the pattern, and fails on new data. Fixes: more data, dropout, weight decay, stopping early.
• Vanishing gradients: in deep stacks, the blame signal can shrink to nothing on its way back, so early layers stop learning. This is a big reason activation functions and residual connections matter.
• Bad learning rate: loss goes to NaN (too large) or barely moves (too small).

The big picture

Try it yourself

• Run the forward pass above and change the random seed a few times
• Count the parameters of a 10 → 32 → 32 → 3 network by hand
• Run the XOR code, then remove the nn.Tanh(). Does it still solve XOR?
• Set the learning rate to 5, then to 0.00001, and watch the loss
• Draw a network by hand and label every weight matrix with its shape

Next: Activation Functions and GELU

Related: Neurons, Weights and Bias Explained, Matrix Multiplication Explained, State Machines Explained