Layers of Abstraction: From Pixels to Identity
A convolutional neural network (CNN) processes a face image through a sequence of layers, each extracting increasingly abstract information. The first layer operates directly on pixel values, detecting local gradients, where colour intensity changes sharply. These gradients correspond to visual edges: the boundary of an eyebrow, the edge of a nostril, the line of a jaw. At this level, the network has no concept of faces; it simply detects intensity patterns.
Successive layers combine the outputs of previous layers, building representations of progressively larger and more complex structures. By the fourth or fifth convolutional block, the network is responding to face-specific structures: the characteristic shadow pattern of an eye socket, the curvature of a cheek, the distinctive geometry of a nasal bridge. At this stage, the representations are no longer interpretable as image regions, they are distributed patterns encoded across many feature maps simultaneously.
What Convolutional Filters Actually Do
A convolutional filter is a small matrix of learned numbers, typically 3×3 or 5×5, that slides systematically across the image and computes a weighted sum at each position. The learned weights determine what pattern the filter detects: a filter with a particular weight arrangement will respond strongly to vertical edges, another to diagonal textures, another to the specific curvature found near a lip corner.
There are hundreds of filters in each layer, each responding to a different pattern. The combined output, a stack of filtered images called feature maps, becomes the input to the next layer, which learns to recognise combinations of the patterns found by the previous layer. This hierarchical composition of simple patterns into complex structures is what allows a finite network to encode the identity-relevant structure of thousands of distinct faces.
Pooling and the Path to the Embedding
After the convolutional stack, a technique called global average pooling collapses all spatial information into a single flat vector. This step is deliberately destructive: the exact positions of features in the image are discarded, retaining only information about what features are present. This pooling is what makes the network robust to small translations and spatial jitter, the network no longer cares exactly where in the image a particular feature appears, only that it appears.
A fully connected layer then maps this pooled vector to the final 512-dimensional embedding space, and L2 normalisation places the result on the unit hypersphere. At this point, all spatial structure from the original image is gone. What remains is a coordinate in a high-dimensional space that encodes identity, not appearance. Two photos of the same person, regardless of lighting or angle, will land near the same coordinate in this space.
What the Network Cannot See
Understanding what CNNs see also requires understanding what they cannot. The network's representations are distributed across all feature maps and layers simultaneously, there is no single neuron or layer that represents 'nose width' or 'eye spacing'. This distributed representation is powerful but opaque. It explains why deep learning models are often called black boxes: their decisions cannot be traced to any single interpretable feature.
This opacity has practical implications. When a match seems surprising, when you receive a result you did not expect, it typically reflects genuine geometric similarity in dimensions that are not intuitively obvious. The network may be responding to a combination of midface length, brow ridge depth, and jaw curvature that you would not consciously identify as a point of resemblance, but which falls out strongly from the embedding computation.