The Speed Challenge
A face recognition pipeline must complete in under 200ms for a real-time experience. The individual stages, detection, alignment, embedding extraction, search, each contribute to this budget. Training a model for accuracy and deploying it for speed are different engineering problems, and most of the effort in production systems is in the deployment stage.
The embedding extraction step (running the ResNet backbone) is typically the bottleneck on CPU. On a modern laptop CPU without optimisation, a full forward pass through ResNet-50 takes 80–200ms, consuming the entire latency budget. With optimisation, this can be reduced to 5–15ms.
Key Optimisation Techniques
Quantisation: Converting model weights from 32-bit float to 8-bit integer. This reduces model size by 4× and speeds up matrix multiplication (the dominant operation) by 2–4× on most hardware, with negligible accuracy loss for inference. ONNX export: Converting the model to the Open Neural Network Exchange format, enabling optimised inference runtimes (ONNX Runtime, TensorRT) to apply hardware-specific optimisations. TensorRT: NVIDIA's inference optimiser that fuses operations, optimises memory access patterns, and generates hardware-specific kernels for NVIDIA GPUs, typically achieving 2–4× speed improvement over unoptimised PyTorch on the same hardware.
FAISS quantisation: The celebrity embedding index can be compressed using product quantisation (PQ), reducing memory and search time at a small accuracy cost. For an index of tens of thousands of photos, like Ollie's, this is not necessary, but it becomes valuable at millions of entries. Batching: Processing multiple user photos simultaneously on a GPU to maximise hardware utilisation when many concurrent requests arrive.