The Vanishing Gradient Problem
Training deep neural networks (networks with many layers) using gradient descent was extremely difficult before 2015. Gradients, the signals used to update network parameters, shrink as they propagate backward through many layers, becoming negligibly small by the time they reach the early layers. This vanishing gradient problem prevented deep networks from learning effectively: early layers received almost no training signal.
The consequence was a ceiling on useful depth. In the early 2010s, most powerful networks had 5–10 layers. Deeper networks trained worse than shallower ones, despite theoretically having more capacity.
Residual Connections: The Solution
He et al. (2015) solved the vanishing gradient problem with residual connections, skip connections that allow gradient to flow directly from later layers to earlier ones without passing through all intermediate layers. A residual block adds its input to its output: output = f(x) + x. This identity shortcut ensures that even if the learned function f(x) has vanishing gradients, the identity path carries gradient directly through.
The result was dramatic: ResNet-152 (152 layers) trained successfully and outperformed much shallower networks. ResNet won the ImageNet competition in 2015 by a substantial margin. More relevantly for face recognition, it unlocked the training of much deeper face recognition networks, and depth correlates directly with representational richness.