This page is the complete record behind the claims made on the Models, Architecture, and Deployment pages — every test that was actually run, in the order it was run, with the raw numbers. Nothing on this page is extrapolated or assumed; each section states exactly what was measured and how.
Why this page exists: this codebase was scaffolded against a published spec, not real model weights, and several of those assumptions turned out to be wrong. Every fix documented here was found by actually running something — inspecting real ONNX graphs, running real photos through the real pipeline, building the real Docker image, and load-testing the real server — not by code review alone.

​
Final results (current state)

Everything below reflects the service after every fix in this document was applied — this is where the numbers landed, not a before/after comparison. The rest of the page is the journey that got here; this section is the destination.

​
Accuracy

Production embedder (arcface_w600k_r50.onnx, InsightFace buffalo_l), full production pipeline, 45 pairwise comparisons across 10 test photos: Clean bimodal separation, zero ambiguous pairs — every same-person pair scored above every cross-identity pair. Full data in §3c.

​
Correctness under concurrency

Full data in §6.

​
Throughput & latency (native, 10-core Apple Silicon, CPU-only)

Peak sustained throughput: ~1,944 RPM at concurrency≈16, scaling cleanly up to that point, then degrading gracefully (not erroring) beyond the 10-core ceiling. Extrapolated per-vCPU estimates for cloud sizing are in Models → Expected throughput — flagged there as extrapolation, not a measurement on real cloud hardware.

​
Docker parity

Confirms the containerized build is numerically equivalent to native (differences are float/compiler noise, not a regression) and, separately, that it runs standalone with the models baked in and zero volume mounts (see Deployment → Baking in models).

​
Build health

​
1. Model I/O verification

Goal: confirm the embedder’s actual tensor contract before trusting any code that depends on it. Method: loaded arcface_w600k_r50.onnx with the onnx and onnxruntime Python packages and printed the real input/output names, shapes, and dtypes directly from the graph. No mismatches. See Models → Embedder for the full contract table.

​
2. Local build (Phase 2)

Built from source on macOS (arm64): Homebrew cmake/opencv@5.0.0/jsoncpp, Drogon v1.9.7 from source, ONNX Runtime 1.19.2 (prebuilt release). One real compile bug found: FaceAligner.cpp failed with no member named 'estimateAffinePartial2D' in namespace 'cv' — OpenCV 5 moved that function (and boundingRect) into a new geometry module that its own umbrella opencv2/opencv.hpp doesn’t include by default, even though the module is built. Fixed with a version-gated include (#if CV_VERSION_MAJOR >= 5) so both OpenCV 4 (Docker) and OpenCV 5 (Homebrew) compile correctly. See Architecture → FaceAligner. Result: clean build, binary linked against all expected libraries (OpenCV 5.0.0, Drogon, ONNX Runtime, OpenSSL, jsoncpp).

​
3. Embedder validation — three models tested, two discarded

Method: a Python harness (scripts/run_similarity_test.py) mirroring the production pipeline exactly — detect → align (ArcFace 5-point template) → normalize → embed → L2-normalize → cosine similarity — run against a manual test set of 11 personal photos (10 usable; 1 failed detection), producing all 45 pairwise comparisons per model.

​
3a. VirtuoTuring (original embedder) — discarded

Same-person pairs scored lower on average than cross-identity pairs. No threshold separates anything in this distribution.
Diagnosis: detection, alignment, and normalization were each independently verified correct at this point (see Models → What was verified) — a noise-image sanity check confirmed the model itself was the problem: two unrelated random-noise images scored 0.9999 similarity under this model, and real same-person photos scored no higher than noise-vs-real comparisons (~0.87). Consistent with its provenance — a self-published, ~23,660-image, zero-download, unbenchmarked model.

​
3b. ONNX Model Zoo arcfaceresnet100-8.onnx — discarded

First replacement candidate, from the official high-trust onnx/models GitHub org — but an old export (opset 8, ~2019).
Even more compressed than VirtuoTuring — worse separation, not better.

​
3c. InsightFace buffalo_l / w600k_r50 — adopted

Actively-maintained, MIT-licensed, opset 11. Passed the same sanity check cleanly before being trusted:
Clean bimodal separation — this is what a working embedder looks like.
This is the embedder shipped in production. COMPARE_THRESHOLD (default 0.4) sits in the gap between the 0.52–0.67 same-person cluster and the 0.01–0.32 cross-identity cluster with margin on both sides.

​
4. Detection bug — full-resolution phone photos

How it was found: aligned crops from the initial VirtuoTuring test looked visually wrong (badly off-center, wrong zoom level) before the embedder was even implicated. Drawing the raw detector bbox/landmarks directly onto the pixel grid OpenCV actually sees (not a viewer-rendered preview, which can apply its own EXIF-rotation handling) isolated the cause to detection, not alignment. Root cause: YuNet’s accuracy degrades badly on raw phone-camera resolutions. At the production detectorScoreThreshold=0.8: Fix: detectFaces() now detects on a 640px-longest-side copy and maps bbox/landmarks back to full-resolution coordinates. See Architecture → InferenceEngine.

​
5. Docker build — three real bugs found by actually running the image

Building the image and confirming it compiles was not sufficient — it took running the container to surface all three of these.
1

Silently-broken apt-get install

Runtime stage listed libjsoncpp26 and libssl3 — neither exists on Ubuntu 24.04 (real names: libjsoncpp25, libssl3t64). A || fallback masked the failure, so the entire package install silently no-opped — including the correctly-named OpenCV packages. Container built “successfully” with zero OpenCV libraries, crashed on start: error while loading shared libraries: libopencv_imgcodecs.so.406: cannot open shared object file. Fixed the names and removed the fallback.
2

OpenCV too old for the detector model

Ubuntu 24.04’s apt OpenCV is 4.6.0; YuNet’s 2023mar model requires ≥4.8.0 — fails at runtime, not build time, with Layer with requested id=-1 not found. Switched the builder stage to compile OpenCV 4.10.0 from source. See Deployment → Why OpenCV is built from source for the full writeup.
3

Missing .dockerignore

Build context was uploading models/, third_party/, build/, and personal test photos on every docker build. Added .dockerignore.

​
Docker vs. native correctness check

Ran the same same-person/cross-identity /v1/compare cases against the containerized service over real HTTP after all three fixes: Differences are consistent with float/compiler noise across platforms, not a correctness regression.

​
6. Concurrency / load test — the most serious bug found

Context: asked to estimate expected RPM. Sequential single-client latency (~300ms p50) doesn’t answer that — a real concurrent load test was required, and it immediately surfaced a correctness bug, not just a capacity ceiling. Method: a Python harness firing concurrent POST /v1/embed requests at increasing concurrency (1→32 threads), 12 seconds sustained per level, against the native macOS build (10 cores).
Throughput collapsed under load instead of plateauing — a correctness bug, not a capacity limit.
Root cause: cv::FaceDetectorYN was shared as a single instance across all worker threads, and detectFaces() calls setInputSize() (mutates object state) immediately before detect() — concurrent requests raced on that shared state. The tell in the server logs:
One request’s 640×640 image processed with another concurrent request’s 40×40 internal state. Fix: each worker thread gets its own cv::FaceDetectorYN instance (thread_local in threadLocalDetector()). Ort::Session (the embedder) was left shared — Run() is genuinely documented thread-safe, unlike FaceDetectorYN. Verified by rerunning the identical load test (table above) and by rebuilding + smoke-testing the Docker image against the fix. See Models → Concurrency for the extrapolated RPM-per-vCPU estimates built on this data, and Architecture → InferenceEngine for the code-level explanation.

​
Summary: bugs found, by how they were caught

The pattern across all six: the bug was invisible until the specific condition that triggers it was actually exercised. This is the reasoning behind every “verify, don’t assume” note elsewhere in these docs.