OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields

On seen camera types, OmniRay achieves a mean ray error of 0.14° and relative Chamfer distance of 0.0179, compared with 6.34° and 0.0760 for UniK3D and 33.03° and 0.1999 for Depth Anything 3. On the Nayar catadioptric camera — held out entirely from training — all baselines exceed 75° while OmniRay achieves 1.12° and 0.0131 CD; on the held-out GoPro fisheye, OmniRay achieves 5.53° versus 6.34° for UniK3D. On the unseen PanoCity dataset, OmniRay achieves 1.35° ray error versus 21.22° and 50.53° for UniK3D and DA3. On CO3Dv2, OmniRay's 2.97° ray error remains within a degree of both baselines.

Fig. 4 compares ray fields and camera-frame pointclouds on unseen PanoCity and CO3Dv2 images, with each ray field colored by its angular error. On the panorama, OmniRay keeps the road continuous while UniK3D breaks it across the image and DA3 fails to recover the ground structure. On the fisheye and GoPro rows, OmniRay keeps the road, pavement and lane markings straight where UniK3D bends them and DA3 also distorts the surrounding buildings. On the held-out catadioptric camera both baselines produce degraded reconstructions. On the CO3Dv2 pinhole example, OmniRay remains comparable to the baselines, demonstrating that the additional flexibility for wide-angle cameras does not come at the expense of pinhole imagery. On in-the-wild images, OmniRay also recovers coherent 3D structure, as shown in Fig. 6.

Table 2 — Main benchmark. Results on seen cameras, held-out cameras, and held-out domains. Ang. denotes mean ray error in degrees; CD denotes median relative Chamfer distance, normalized by each scene's mean point norm. Metrics are macro-averaged over test images, and all methods use per-image median scale alignment. Lower is better; best results are bold. † denotes exclusion from our training data; a dash indicates that the method does not predict scene geometry. Full per-camera and per-dataset results and additional metrics are provided in Tables 4–6 below.

Table 3 — Ray representation ablation. Mean angular ray error in degrees (lower is better) at native resolution. Seen camera families are evaluated on held-out configurations; † denotes configurations excluded from training. Only the ray module is trained and evaluated. Best results are bold.

The w/o AUG row of Table 2 isolates the random-lens sampling. Without it the seen cameras remain strong at 1.39°, but the held-out catadioptric and GoPro errors rise to 30.87° and 7.97°; restoring it recovers 1.12° and 5.53° and improves the seen cameras to 0.14°. Fig. 5a shows the same effect on a catadioptric example, where the unaugmented model exhibits larger ray-field and geometric errors. These results indicate that random-lens sampling is important for extending the composite representation beyond the camera configurations encountered during training. Adding ray guidance to each DPT scale (+ RG) leaves the metrics unchanged to within run-to-run variation; the injection is zero-initialized and could use the prompt if it helped, so we omit it.

Table 3 isolates and analyses different ray representations. The radial base performs well on fisheye and pinhole cameras (1.761° and 0.788°) but struggles on panoramas (21.825°), where a monotone radial profile is insufficient. The spherical base reduces panoramic error to 5.088°, while the free-form field performs best among single representations on panoramas, fisheye, and pinhole, but reaches 11.335° on the held-out catadioptric configuration. These results motivate combining a structured radial base with more flexible residuals.

Adding an SH residual to the radial base improves panoramic and held-out accuracy; staged training achieves 0.610° and 1.680°, respectively, and the best GoPro result (3.393°). Adding the free-form residual further improves the panorama (0.399°), fisheye (1.322°), and catadioptric results (1.538°) in the full staged model, which also outperforms both jointly trained three-term variants across all five configurations. On pinhole cameras, the full model remains competitive at 0.418°, ranking fourth behind the VGGT-initialized variant (0.330°) and the two radial-plus-free-form variants. While the additional spherical residual offers little benefit for this narrow-field geometry, it supports greater flexibility across wide-field cameras. Overall, these results support the staged radial → SH → free-form composition as a balance between geometric structure and representational flexibility.