Fix Xcode 26 C++ build + iPhone 17 Pro depth visualization corruption#57
Open
wonmor wants to merge 4 commits into
Open
Fix Xcode 26 C++ build + iPhone 17 Pro depth visualization corruption#57wonmor wants to merge 4 commits into
wonmor wants to merge 4 commits into
Conversation
Depth CVPixelBuffer-backed Metal texture was created with MTLTextureUsage.shaderWrite but is actually used as the source of an MPSImageGaussianBlur (read-only). The A19 GPU on iPhone 17 Pro enforces this strictly, producing blocky purple/blue artifacts on the depth-coloring overlay; the A17 Pro silently tolerated the wrong flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
Collaborator
|
Thanks for the fix! I don’t see your first commit in here, does the build work for you without it? |
Upstream's commit 8c3bc69 picked between two orientation matrices based on whether intrinsicMatrixReferenceDimensions reports a widescreen sensor (>1.5 aspect ratio). On real-world iPhone 15 Pro Max devices, that heuristic mis-classified the sensor as widescreen at runtime and flipped the scan orientation. Switch to a hardware-id check via uname(): apply the new (negate-Y, negate-Z) matrix only on iPhone18,X — iPhone 17 Pro and iPhone 17 Pro Max. Every other iPhone, including 15 Pro Max, falls through to the original (swap-XY, negate-Z) matrix that was correct pre-upstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PoissonRecon previously returned a NULL solver on degenerate input clouds without propagation; SurfaceTrimmer's -1 return codes were discarded; SCMeshTexturing then read a possibly-empty output PLY. Net effect: a sparse cloud (e.g. from ICP diverging under head motion) produced an empty mesh or hung the meshing progress UI with no surfacing error. - MeshingOperation gains a failureReason property; main() stat-checks the Poisson and SurfaceTrimmer output PLYs and captures SurfaceTrimmerExecute's int return (already returned, just unused) to set a concrete reason. - SCMeshingOperation re-exports failureReason from the C++ wrapper. - SCMeshTexturing._meshPointCloud: refuses < 500-point inputs up front, checks operation.failureReason after run, stat-checks output existence, and rejects 0-vertex geometry rather than handing back an empty mesh. - _buildAPIError now sets NSLocalizedDescriptionKey alongside the existing NSDebugDescriptionErrorKey so the failure message reaches error.localizedDescription in the SwiftUI consumer. No upstream PoissonRecon source was modified -- detection is purely a thin shim around output stat + return-code check, per plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an opt-in ARKit + Kabsch capture path alongside the legacy AVCaptureSession one. Both pipelines live in the same build, gated by UserDefaults; users opt in via kScanUseARFacePipelineDefaultsKey. Phase B - ARFaceCameraManager + head-pose-prior accumulate overload. - New ARFaceCameraManager (StandardCyborgUI) drives capture from an ARSession with ARFaceTrackingConfiguration. Per frame it converts the YUV captured image to BGRA, pulls capturedDepthData + calibration, and computes a camera-in-face delta from ARFaceAnchor.transform * inverse of the prior frame's same transform. Confidence collapses ARFaceAnchor .isTracked + ARCamera.trackingState into [0, 1]. - SCReconstructionManager gains a sibling -accumulate... method carrying the delta + confidence. _IncomingFrameData carries it through the input queue; the model-queue worker converts simd_float4x4 -> Eigen::Matrix4f and forwards to PBFModel. - PBFModel::assimilate accepts an optional headPoseDelta + confidence. At confidence >= 0.9 it bypasses _runICP entirely and uses the prior as the frame transform -- this is what lets the pipeline fuse frames during head motion that plain ICP would reject. - ScanningViewController now constructs CameraManager OR ARFaceCameraManager based on the UserDefaults flag, conforms to both delegate protocols, and routes both into a unified _handleFrame. Phase C - reproject into head-anchor frame before fusion. - Folded into Phase B's delta math: because we feed camera-in-face deltas, the accumulated _extrinsicMatrix naturally lives in face-frame, so fused surfels stay registered to the head as the head rotates in world. No separate C++ change required. Phase D - Kabsch-based confidence validator. - KabschRefiner (StandardCyborgUI, Swift+simd, no external deps) runs RANSAC + Kabsch rigid alignment on a subsampled face-mesh vertex constellation and returns an inlier fraction in [0, 1]. Rotation is recovered via polar decomposition (Higham iteration) to avoid pulling BLAS/LAPACK. - ARFaceCameraManager wires it in behind kScanUseKabschRefinerDefaultsKey: when ARKit's per-frame delta disagrees with what its own face-mesh vertices say, confidence is linearly scaled down, which causes PBFModel to fall back to ICP for that frame. Phase E hook (UI lives in the parent app) - facePoseObserver callback. - ARFaceCameraManagerDelegate gains an optional arFaceCameraDidObserveFacePose method that fires every face-tracking frame with inv(camera) * face. - ScanningViewController exposes a facePoseObserver closure forwarding the pose; the parent app's bridge installs the closure to drive the Face-ID-style overlay. The OpticALLY parent app adds the overlay + UserDefaults wiring in a separate commit on the parent repo (depends on this submodule bump). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent fixes that affect anyone shipping a TrueDepth scanning app on current Apple hardware/toolchains:
C++ compat for Xcode 26 + static linking (671a57e) — addresses build failures in Xcode 26 with the modern C++ toolchain, and switches some targets to static linking to play nicer with downstream Swift apps.
iPhone 17 Pro depth visualization corruption (416bcfa) — the depth
CVPixelBuffer-backedMTLTextureinDepthColoringFilter._metalTexture(fromDepthBuffer:)was created withMTLTextureUsage.shaderWrite, but the texture is read (it's the source of anMPSImageGaussianBlur, never a destination). The A17 Pro (iPhone 15 Pro Max) silently tolerated the wrong flag; the A19 GPU on iPhone 17 Pro enforces it strictly and produces blocky purple/blue artifacts on the depth-coloring overlay during scanning. One-line fix:.shaderWrite→.shaderRead.Test plan
🤖 Generated with Claude Code