How neural networks learn

Learning as optimisation#

Training minimises an objective over a parameter vector. Given a dataset $\mathcal{D} = {(x_i, y_i)}_{i=1}^{n}$ drawn i.i.d. from an unknown distribution $P(x, y)$, we want to minimise the true risk:

$$ R(\theta) = \mathbb{E}{(x,y) \sim P}\left[ \ell!\left(f\theta(x),\, y\right) \right] $$

Where:

• $\theta$ — all trainable parameters.
• $f_\theta$ — the network as a function.
• $\ell$ — the per-example loss function.
• $P$ — the data-generating distribution, which we never observe.

$P$ is unknown, so we minimise the empirical risk instead:

$$ \hat{R}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell!\left(f_\theta(x_i),\, y_i\right) $$

This substitution is called empirical risk minimisation (ERM), formalised by Vapnik (1991). The gap $R(\theta) - \hat{R}(\theta)$ is the generalisation gap, and controlling it is what regularisation, early stopping and held-out validation are all aimed at.

The update rule#

Parameters move by stochastic gradient descent:

$$ \theta_{t+1} = \theta_t - \eta_t \nabla_\theta \hat{R}_{\mathcal{B}_t}(\theta_t) $$

Where:

• $t$ — the iteration index.
• $\eta_t > 0$ — the learning rate at step $t$.
• $\mathcal{B}_t$ — a mini-batch sampled from $\mathcal{D}$.
• $\nabla_\theta \hat{R}_{\mathcal{B}_t}$ — the gradient of the loss on that mini-batch.

The mini-batch gradient is an unbiased estimator of the full-batch gradient: $\mathbb{E}{\mathcal{B}}[\nabla \hat{R}{\mathcal{B}}] = \nabla \hat{R}$. Its variance falls as $O(1/|\mathcal{B}|)$, which is the entire trade-off behind batch-size selection.

Cross-entropy, and why it pairs with sigmoid#

For binary targets $y \in {0, 1}$ and prediction $\hat{y} = \sigma(z)$ where $\sigma(z) = (1 + e^{-z})^{-1}$:

$$ \ell(\hat{y}, y) = -\left[ y \log \hat{y} + (1 - y) \log(1 - \hat{y}) \right] $$

Differentiating with respect to the pre-activation $z$, and using $\sigma'(z) = \sigma(z)(1 - \sigma(z))$:

$$ \frac{\partial \ell}{\partial z} = \hat{y} - y $$

The $\sigma'(z)$ factor cancels exactly. This matters practically: a squared-error loss on a sigmoid output retains that factor, and $\sigma'(z) \to 0$ when $z$ is large, so a confidently wrong prediction produces a vanishing gradient and the model stalls. Cross-entropy has no such failure mode, which is why it is the default for classification.

In production, compute this from logits with a numerically stable fused implementation — torch.nn.BCEWithLogitsLoss or torch.nn.CrossEntropyLoss — never by applying a softmax or sigmoid and then taking a log.

Convergence, stated honestly#

The loss surface of a deep network is non-convex, so there is no guarantee of reaching a global minimum. Practice is better than that pessimism suggests, and the reasons are partly understood:

• Dauphin et al. (2014), Identifying and attacking the saddle point problem, argue that in high dimensions saddle points, not poor local minima, are the dominant obstacle. A critical point needs every one of its Hessian eigenvalues positive to be a local minimum, which becomes vanishingly unlikely as dimension grows.
• Choromanska et al. (2015) give evidence that most local minima in large networks reach similar loss values.
• Zhang et al. (2017), Understanding deep learning requires rethinking generalization, show these networks can fit pure random labels to zero training error. Capacity alone therefore cannot explain generalisation, and this remains an open question.

Practical diagnostics#

Where to read next#

• Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors.
• Bottou, Curtis and Nocedal (2018), Optimization methods for large-scale machine learning — the careful survey of SGD theory.
• Smith (2017), Cyclical learning rates for training neural networks — the origin of the learning-rate range test.

What to learn next#

• Backpropagation — how the blame is shared out across every weight.
• Gradient descent — choosing the step size, and what goes wrong when you choose badly.
• Loss functions — picking the right way to measure "wrong".