Why a Single Network Is Not Enough
The intuitive approach to face recognition would be a classifier: a network that takes a face image and outputs the identity of the person. This works when you have a fixed, known set of identities with plenty of training images each. But it breaks down completely for the celebrity matching use case, you want to match against celebrities the network has never seen during training, including people who only became famous after the model was built.
What is actually needed is a network that learns a general notion of facial similarity, one that transfers to new identities it has never encountered. This requires a completely different training approach: instead of asking the network to name a face, you ask it whether two face photographs show the same person.
The Siamese Architecture
A Siamese network consists of two identical copies of the same neural network, sharing exactly the same weights, processing two face images simultaneously. Each copy independently produces an embedding for its input. A distance function then measures how far apart the two embeddings are in the embedding space. The network is trained by showing it labelled face pairs: same-identity pairs (the distance should be small) and different-identity pairs (the distance should be large).
The critical innovation is shared weights. Both copies are not just similar, they are literally the same network applied twice. This guarantees that both embeddings live in exactly the same mathematical space, making it meaningful to compute a distance between them. If you trained two separate networks independently, their embeddings would inhabit incompatible spaces and distance computation would be meaningless.
Training: Learning General Similarity
Training a Siamese network requires constructing pairs of face images with known identity labels. The loss function, typically contrastive loss or triplet loss, adjusts the network weights to pull same-identity embeddings together while pushing different-identity embeddings apart by at least a specified margin. Through millions of such pair comparisons, the network learns what makes faces similar at a structural level.
This pair-based training is what enables generalisation to completely new identities. The network has not learned to recognise any specific celebrity's face. It has learned a general structural notion of similarity, one that transfers immediately to any new face it encounters. When you upload a photo of yourself, the network produces an embedding based on the same learned notion of similarity, placing you in the same space as every celebrity it has ever processed.
Pre-computation and Efficient Search
The same trick applies to Ollie at inference time, even though its network was trained with the CosFace loss rather than with pairs: because one network embeds every face, celebrity embeddings only need to be computed once and stored. When you upload a photo, only your embedding needs to be computed. The search then compares your single embedding against thousands of pre-computed celebrity embeddings, a much more efficient process than running two networks for each comparison.
This pre-computation approach is why the search is so fast. The expensive neural network computation runs once (your photo); the subsequent similarity search in the pre-built index is very fast regardless of database size.