How Neural Networks Learn: Loss, Gradient Descent and Backpropagation

A neural network starts with random weights and makes bad predictions. Training improves it step by step: measure how wrong it is, work out which way to nudge each weight to make it less wrong, nudge, and repeat thousands of times. Three ideas make this work: a loss function, gradient descent and backpropagation.

Loss: how wrong is the model?

The loss is a single number measuring the error over the training data. For yes/no predictions, the usual choice is cross-entropy (log loss), which punishes confident wrong answers heavily. For numbers, it is often the mean squared error. Training aims to make the loss as small as possible.

Gradient descent: which way to move

The gradient tells you, for every weight, how the loss changes if that weight increases a little. Gradient descent moves each weight a small step in the direction that lowers the loss. The size of the step is the learning rate.

Backpropagation: sharing out the blame

A network has many layers, so how do you know how much each weight contributed to the error? Backpropagation works backwards from the output: it calculates the error at the output, then uses the chain rule from calculus to pass it back through each layer, giving every weight its share. Frameworks such as PyTorch do this automatically.

The learning rate matters

This experiment trains the same network on the same data with three learning rates:

import numpy as np

def train(lr, epochs=300, seed=0):
    """Train the same small network on the circle data with a given learning rate; return the loss history."""
    rng = np.random.default_rng(seed)
    X = rng.uniform(-1, 1, size=(1000, 2))
    y = (X[:, 0] ** 2 + X[:, 1] ** 2 < 0.5).astype(float)
    W1, b1, W2, b2 = rng.normal(0, 1, (2, 8)), np.zeros(8), rng.normal(0, 1, (8, 1)), np.zeros(1)
    history = []
    for epoch in range(epochs):
        h = np.tanh(X @ W1 + b1)
        p = (1 / (1 + np.exp(-(h @ W2 + b2)))).ravel().clip(1e-9, 1 - 1e-9)
        history.append(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))
        d_out = (p - y)[:, None] / len(y)
        d_h = (d_out @ W2.T) * (1 - h ** 2)
        W2 -= lr * h.T @ d_out; b2 -= lr * d_out.sum(0)
        W1 -= lr * X.T @ d_h;   b1 -= lr * d_h.sum(0)
    return history

for lr in (0.01, 1.0, 50.0):
    h = train(lr)
    print(f"learning rate {lr:5}: loss at start {h[0]:.3f}, after 100 steps {h[100]:.3f}, "
          f"after 300 steps {h[-1]:.3f}, biggest jump up {max(0.0, float(max(np.diff(h)))):.3f}")

learning rate  0.01: loss at start 0.894, after 100 steps 0.751, after 300 steps 0.660, biggest jump up 0.000
learning rate   1.0: loss at start 0.894, after 100 steps 0.294, after 300 steps 0.158, biggest jump up 0.000
learning rate  50.0: loss at start 0.894, after 100 steps 12.579, after 300 steps 8.405, biggest jump up 11.504

• Too small (0.01): the loss falls steadily but very slowly. Training would take far too long.
• About right (1.0): the loss falls quickly and smoothly.
• Too large (50): the steps overshoot, the loss jumps up by more than 11 at one point, and the model ends worse than it started.

In practice, people use optimisers such as Adam, which adapt the step size for each weight, and learning-rate schedules that start larger and shrink over time.

Training in practice

• Mini-batches: compute gradients on small batches of data rather than the whole dataset, which is faster and adds helpful noise.
• Epochs: one pass over the training data. Training usually runs many epochs.
• Validation loss: watch it alongside training loss. If training loss keeps falling but validation loss rises, the network is overfitting; stop early or add regularisation such as dropout.

Common problems

1. Loss does not fall: learning rate too small, a bug in the data, or features on very different scales.
2. Loss explodes or becomes “nan”: learning rate too large.
3. Great training loss, poor validation loss: overfitting. See overfitting and data leakage.

Key takeaway

Training minimises a loss: backpropagation calculates how each weight affects it, and gradient descent moves the weights a step at a time, sized by the learning rate. Too small a rate is slow, too large is unstable. Next: the main network designs, in CNNs, RNNs and transformers.