What 'Fooling' Means in Face Matching
In the context of face matching, 'fooling' refers to faces that produce unexpected or inconsistent matching behaviour, not faces that actively defeat the system, but faces that create ambiguity. There are two main types: highly central faces (embeddings near the mean of the distribution that match a disproportionate number of users at moderate confidence) and unstable faces (embeddings that shift noticeably between photo conditions because the training data for that identity was inconsistent).
High false-positive rates are a related issue: a face whose embedding sits in a dense region of space may produce high-confidence matches with many users, even when the visual resemblance is not strong. The confidence score reflects embedding distance, not visual obviousness.
Central vs Peripheral Embeddings
The distinction between central and peripheral embeddings is fundamental. A celebrity with proportions close to the mathematical average of the embedding distribution appears near many users. Their face does not look generically average to a human observer, in fact, averaged faces tend to look quite attractive, but mathematically they sit near the centroid of a dense cluster.
Celebrities with very unusual proportions occupy the periphery. They match fewer users overall, but when they do appear, the match typically reflects genuine, visually obvious geometric similarity. The embedding distance to their nearest neighbours is the same as for any celebrity; it is just that there are fewer near neighbours in the sparse peripheral region.
Interpreting Unexpected Results
When a match seems visually surprising, the explanation is usually one of two things: either the similarity is in dimensions not obvious to casual visual inspection (brow ridge depth, midface length, jaw curvature), or the celebrity has a central embedding and is appearing because of broad geometric proximity rather than specific resemblance.
Looking at multiple matches simultaneously helps disambiguate. If the top five results span a range of visually diverse celebrities who share one or two specific features, those shared features are likely driving the match. If the top results are all similarly average-looking celebrities with little strong resemblance, you may be in a central region of the embedding space.