Residual connection
In one sentence A residual connection adds a layer's input directly to its output, giving both information and gradients a shortcut past the layer.
Updated
A residual connection adds a layer's input straight onto its output, so the layer only has to learn the change it wants to make.
Think of editing an essay. One approach: retype the whole essay from memory at every revision — and risk losing good sentences each time. The other: keep the essay and mark corrections on top. Residual connections are the corrections approach. Each layer receives the running text, proposes an edit, and the edit is added to the original rather than replacing it.
In code the idea is one line: output = layer(x) + x. The + x is the residual (or "skip") connection.
Two big consequences follow. First, a layer can do no harm: if its best contribution is nothing, it can learn to output zeros and the input passes through untouched. That makes very deep networks safe to build. Second, during backpropagation the gradient flows backwards through the + x path without being multiplied by anything, which is the most direct cure for the vanishing-gradient problem.
The idea arrived with ResNet in 2015 (He et al.), which made 100+ layer CNNs trainable for the first time. Today every transformer block is built the same way: attention and feed-forward layers each wrapped with a residual connection and a layer-normalization.
Where to go next
- Full lesson: Transformers
- Related terms: vanishing-gradient, transformer, layer-normalization, cnn