Data Beats Cleverness
Early deep learning research spent a lot of effort on new network designs. Over time, a pattern became clear across vision, language and speech: once a model has enough capacity, the amount and quality of training data matter more than architectural tricks. Face recognition followed the same path.
A good example: Google's FaceNet in 2015 reached 99.63% on LFW with a fairly standard network, but trained on about 200 million images of 8 million people, far more data than academic groups had. The architecture wasn't the secret. The data was.
How Face Datasets Grew
| Dataset | Year | Images | People |
|---|---|---|---|
| CASIA-WebFace | 2014 | ~494,000 | 10,575 |
| VGGFace | 2015 | 2.6 million | 2,622 |
| MS-Celeb-1M | 2016 | ~10 million | 100,000 |
| VGGFace2 | 2018 | 3.31 million | 9,131 |
| MS1MV2 | 2019 | 5.8 million | 85,742 |
| Glint360K | 2021 | 17 million | 360,000 |
| WebFace260M | 2021 | 260 million (42 million after cleaning) | 4 million (2 million after cleaning) |
Each jump brought gains, especially on hard tests like IJB-C with strict false match limits. The WebFace260M team showed that a cleaned subset of 42 million images could train models that beat those trained on smaller sets, even with the same architecture.
Read more: VGGFace2 Explained: 3.3 Million Faces of 9,131 People
What Makes a Good Face Dataset
- Many identities. Margin losses like ArcFace and CosFace learn by separating people. More people means more practice at telling faces apart.
- Enough photos per person. At least a few dozen, so the model sees each person across conditions.
- Variety within each person. Different ages, lighting, angles, expressions and cameras, so the model learns what to ignore.
- Clean labels. A photo filed under the wrong name teaches the model to merge two people. Web-scraped sets are often 10% or more mislabelled before cleaning.
- Demographic balance. Groups that appear less get less practice, which shows up as higher error rates. See accuracy across demographics.
- No overlap with test sets. If benchmark people are in the training data, the scores are inflated.
Width vs Depth
Dataset designers face a trade-off between "wide" (many people, fewer photos each) and "deep" (fewer people, many photos each). Research by Masi and others found that both matter, but for the margin losses used today, width tends to help more, as long as each person has enough photos to define their cluster.
That's the logic behind MS1MV2, a cleaned version of MS-Celeb-1M prepared by the InsightFace team: about 85,742 people with an average of around 68 photos each. It became the standard training set for ArcFace-era models, so results are easy to compare.
Read more: ArcFace: The Loss Function That Makes Modern Face Recognition So Accurate
Get new guides by email
How Ollie's Training Data Was Used
Ollie's 20-layer network was trained from scratch on one of these public research datasets, with no pretrained weights, using CosFace loss. Before training, every person who appears in the LFW benchmark was removed from the data, so the 99.5% LFW score measures faces the model had never seen. Training took about 10 days on a single RTX 4060 Ti.
Training from scratch was a deliberate choice. The usual shortcut is transfer learning: start from a model someone else trained and fine-tune it. That's faster, but you inherit whatever the original model learned, including its biases and licence terms.
Ollie's celebrity search index is separate from the training data. It's built from freely licensed Wikimedia Commons photos and isn't used for training. See how the celebrity database was built.
The Ethics of Face Data
Most large face datasets were scraped from the web without the consent of the people in them. After criticism, Microsoft withdrew MS-Celeb-1M in 2019, Duke University withdrew its DukeMTMC dataset, and others followed. Copies of some datasets still circulate, and their use is a live debate in the research community.
Newer approaches try to reduce the problem: synthetic face datasets generated by AI, datasets collected with explicit consent, and stricter licences. Synthetic data has improved fast but still trails real data on hard benchmarks. Expect this to be one of the biggest changes in face recognition over the next few years.