What Is Overfitting?
Overfitting occurs when a machine learning model learns the training data so thoroughly, including its noise and idiosyncratic patterns, that it loses the ability to generalise to new data. The model has memorised rather than learned. On training examples it performs well; on new examples it performs poorly, because the patterns it has learned are specific to the training set rather than general to the underlying problem.
In face recognition terms: an overfit model might perfectly match every training celebrity, but when presented with a new user's face, it fails to place the embedding in the correct region of face space because its representation of face identity is too tightly coupled to the specific photos in the training set.
Why It Occurs
Overfitting becomes more likely as model capacity increases relative to training data size. A model with more parameters than training examples can potentially memorise every training example exactly. For face recognition, where models have millions of parameters trained on hundreds of thousands to millions of examples, overfitting is a real risk that requires active countermeasures.
It is also more likely when training proceeds for too many epochs: the model first learns general patterns (fast, early in training) and then progressively fits to idiosyncratic training data characteristics (slow, late in training). Validation curves typically show training accuracy continuing to increase while validation accuracy plateaus or decreases, a characteristic overfitting signature.
How Overfitting Is Prevented
Standard techniques include: Dropout, randomly zeroing neural activations during training, preventing any single pathway from being relied on exclusively. Weight decay (L2 regularisation), penalising large weights, pushing the model toward simpler representations. Data augmentation, artificially expanding the training set by transforming training images (flipping, rotating, colour jitter), making the model learn representations robust to these variations.
Ollie's training uses early stopping, monitoring validation accuracy and stopping when it begins to plateau, before overfitting to training data occurs. The training schedule uses CosineAnnealingLR with T_max=150, enabling thorough exploration of the loss landscape while avoiding late-training overfitting. The validation set acts as a held-out proxy for real-world generalisation.
Signs of Overfitting in Practice
At training time, overfitting is visible in the divergence between training and validation loss, when training loss continues falling while validation loss stops improving or rises. A well-trained model shows these two losses tracking each other closely throughout training.
In deployment, overfitting shows as poor generalisation: the model works well on photo conditions similar to its training data but degrades more than expected on novel conditions. This is one reason diverse training data is so important, a model trained only on high-quality controlled portraits will overfit to that condition and generalise poorly to real-world selfie photos.