--- license: cc-by-nc-4.0 library_name: pytorch pipeline_tag: depth-estimation tags: - depth-estimation - monocular-depth - face - coreml - apple-silicon - depth-anything-v2 base_model: depth-anything/Depth-Anything-V2-Large language: - en --- # FaceDepth Face-specialized monocular depth estimation. One photo in, sharp facial relief out, with no depth sensor. General depth models train on streets, rooms, and furniture, so they flatten a face into a smooth blob. A face holds its information in millimeters of relief, the bridge of the nose against the cheek, the small step from lid to eyeball, and that signal sits far below what a scene-trained model represents. Face relighting, avatar capture, and AR effects need the part those models throw away. FaceDepth is a Depth Anything V2-Large variant tuned to keep that detail. It resolves the eyelid crease, the nostril rim, the lip contour, and the hairline. Code and paper: [github.com/AristidesAI/FaceDepth](https://github.com/AristidesAI/FaceDepth) ยท Instagram: [@aristides.lab](https://instagram.com/aristides.lab) *Source on the left, FaceDepth on the right.* ## Results Measured on 500 held-out faces. | metric | baseline DA2-Large | FaceDepth | change | |---|---|---|---| | face-region SSI-MAE | 0.02764 | **0.00906** | **-67%** | | depth-edge F1 vs reference | 0.717 | **0.882** | **+23%** | | edge recall vs reference | 0.745 | **0.873** | +17% | | edge precision vs reference | 0.694 | **0.893** | +29% | Edge metrics are density-matched at 5% of in-face gradient pixels with a 2-pixel tolerance, so a blurry model cannot win by spreading weak gradients across the whole face. Recall and precision both rise, so the model finds real depth edges and stops inventing ones that are not there. ![input, reference, baseline, FaceDepth, and difference for six held-out faces](https://huggingface.co/a-ml/FaceDepth/resolve/main/figures/comparison.jpg) *Left to right: input, reference depth, baseline DA2-Large, FaceDepth, and the difference between the last two. Depth is normalized inside the face mask so relief stays visible. Compare columns three and four against column two.* ![a second set of six held-out faces in the same layout](https://huggingface.co/a-ml/FaceDepth/resolve/main/figures/comparison_2.jpg) *Six more held-out faces, same layout. The difference column concentrates on the face interior and hairline.* ## Files | file | format | size | notes | |---|---|---|---| | `FaceDepth_step15792.pt` | PyTorch | 2.5 GB | full checkpoint. Use the `ema_model` key. | | `coreml/FaceDepth_fp32.mlpackage` | Core ML | 1.2 GB | unquantized reference, 5.4 fps | | `coreml/FaceDepth_fp16.mlpackage` | Core ML | 668 MB | **realtime, 40.5 fps, Neural Engine** | | `coreml/FaceDepth_int8.mlpackage` | Core ML | 335 MB | 39.4 fps, smallest, GPU | | `video_depth.py` | script | | convert any video into a depth-map video | Every Core ML export correlates at 1.00000 against the PyTorch reference. Benchmarks use 392x518 input on an Apple-silicon laptop with `ComputeUnit.ALL`, averaged over 20 runs after warmup. On this hardware int8 buys a 2x size reduction rather than speed, so its advantage is the app bundle. The ranking may differ on iPhone, where the Neural Engine is relatively stronger, and that has not been measured. ## Quick start ### Core ML, realtime ```python import numpy as np, coremltools as ct from PIL import Image model = ct.models.MLModel("coreml/FaceDepth_fp16.mlpackage", compute_units=ct.ComputeUnit.ALL) img = Image.open("face.jpg").convert("RGB").resize((392, 518)) disp = np.asarray(model.predict({"image": img})["depth"]).reshape(518, 392) # inverse depth: larger = nearer ``` Input is a 392x518 portrait RGB image. ImageNet normalization is folded into the graph, so pass raw pixels. ### PyTorch ```python import torch, sys sys.path.insert(0, "third_party/DepthAnythingV2") from depth_anything_v2.dpt import DepthAnythingV2 m = DepthAnythingV2(encoder="vitl", features=256, out_channels=[256, 512, 1024, 1024]) ck = torch.load("FaceDepth_step15792.pt", map_location="cpu", weights_only=True) m.load_state_dict(ck["ema_model"]) m = m.to("mps").eval() ``` The checkpoint also carries a `conf_head`. Inference does not need it and the Core ML exports drop it. ### Normalizing the output for display The model returns relative inverse depth with an arbitrary scale. **Normalize inside the face, not across the frame.** A whole-frame min-max collapses the face's range the moment a distant background enters the shot, which reads as a black or washed-out face. This one detail causes most of the "the model looks broken" reports. ```python lo, hi = np.percentile(disp, 2), np.percentile(disp, 98) norm = np.clip((disp - lo) / (hi - lo), 0, 1) # 1 = nearest ``` ## Video `video_depth.py` converts any video ffmpeg can read, up to 4K, into a depth-map video. Frames stream through an ffmpeg pipe, so a long clip never lands on disk as a frame dump and memory stays flat. ```bash python video_depth.py --input clip.mov --output depth.mp4 --ckpt FaceDepth_step15792.pt python video_depth.py --input clip.mov --output sbs.mp4 --side-by-side --colormap magma python video_depth.py --input clip.mp4 --output out.mp4 --smooth-depth 0.3 --bf16 ``` Colormaps: inferno, magma, turbo, viridis, plasma, bone, gray. `--range-ema` smooths the near/far range across frames and is on by default with no ghosting. `--smooth-depth` smooths depth itself, which cuts residual jitter but ghosts behind fast motion. Needs ffmpeg on PATH and [Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2) cloned into `third_party/DepthAnythingV2`. ## Limitations This model optimizes single-image sharpness, so live video flickers more than a temporally stabilized model would. For offline video, `video_depth.py` damps that with `--range-ema` and `--smooth-depth`. Detail is capped by the reference the model learned from. The claim is sharp feature relief and boundaries, not sub-millimeter texture. Per-eyelash depth is beyond what any current monocular model resolves. Training data is centered, well-lit, and limited in pose, occlusion, and demographic diversity relative to in-the-wild use. Performance by demographic group has not been measured. Evaluate before deploying on populations or capture conditions that differ from the training distribution. ## License and provenance Released under **CC-BY-NC-4.0**, non-commercial research use. FaceDepth derives from [Depth Anything V2-Large](https://huggingface.co/depth-anything/Depth-Anything-V2-Large), which is CC-BY-NC-4.0. It builds on [Apple Depth Pro](https://github.com/apple/ml-depth-pro) and [CelebAMask-HQ](https://github.com/switchablenorms/CelebAMask-HQ), whose terms restrict use to non-commercial research and education. Honor the upstream terms of all three. ## Citation ```bibtex @software{facedepth2026, title = {FaceDepth: Face-Specialized Monocular Depth Estimation}, author = {Lintzeris, Aristides}, year = {2026}, url = {https://huggingface.co/a-ml/FaceDepth} } ``` Please also cite the work this builds on: Depth Anything V2 (arXiv:2406.09414), Depth Pro (arXiv:2410.02073), MiDaS (arXiv:1907.01341), DPT (arXiv:2103.13413), DINOv2 (arXiv:2304.07193), and CelebAMask-HQ (arXiv:1907.11922).