Télécom Paris

From Reconstruction to Reasoning

Télécom Paris

From Reconstruction
to ReasoningStructured 3D Representations

Bingchen Gong

LIX, École Polytechnique

21 September 2026

Foundations for the research programme From Video to Animatable and Efficient 3D Worlds

Thank you for inviting me. I work on representations that connect images and video with geometry, semantic understanding and, increasingly, executable models. The research programme I submitted to you is called From Video to Animatable and Efficient 3D Worlds. This talk explains the foundations I would build it on. My central question is how visual observations can become something that we can understand, manipulate and improve through feedback. I will follow that question through a sequence of connected projects: first reconstruction, then semantic structure, and finally learning from judgments and evaluating agents that construct 3D objects. These are complementary studies with different inputs and evaluation settings, not stages of one system. At the end I will return to the programme and to what I would start first at Télécom Paris.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/CV_Gong_TelecomParis.tex: Research position and publications
Presenter affiliation and research context.
C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Line 31, project title
Programme title as submitted; the talk presents its foundations, not a demonstrated integrated system.

Motivation

Editing captured scenes

RecolorNeRF demonstration: editing a palette entry recolors the car; the active layer is shown at the right and the palette below — **RecolorNeRF**: edit colors through a palette

SeamlessNeRF demonstration: arms from another radiance field attached to a statue, before editing and with appearance propagated across the seam — **SeamlessNeRF**: compose parts and match appearance

Two early examples of this motivation, both joint work with Yuehao Wang, Xiaoguang Han and Qi Dou. RecolorNeRF decomposes a captured radiance field into a small learned palette and spatially varying layers, so the palette becomes the handle: in the demonstration, changing one entry turns the car magenta, purple or gold, and changing another turns the background to autumn tones, consistently across views. SeamlessNeRF edits structure instead of color: arms reconstructed as a separate radiance field are attached to an armless statue, and gradient-domain propagation makes their appearance match the marble across the seam. The handle there is a part boundary. The point I want to carry forward is the relationship between representation and operation. If an operation has no handle in the representation, controlling it becomes difficult. Recoloring and composing already need different handles; moving an articulated part or checking a collision needs others again. My later work brought in imaging variation and time: BilaRF handles inconsistent capture when building radiance fields, and Deform3DGS reconstructs dynamic surgical scenes from video. This talk follows the geometric and semantic handles, which the animation programme needs first. These examples show appearance and composition editing only, not articulated motion or collision checks.

Sources and evidence

C:/Linux/CNRSApplication/presentation-archive/2023/2023-03-22_RecolorNeRF_Color-Editing-of-3D-Scenes.pptx: Slide 20 demonstration video (media5.mp4), excerpt 4.7 to 25.5 s; slide 1 clips labelled Source, Edit 1, Edit 2 (media1, media3, media2); provenance and hashes in assets/reconstruction/evidence.json
Authentic RecolorNeRF palette-editing demonstration on the Toyota scene. Presentation canvas restyled white with dark headings; original photo, active-layer view and palette colors preserved. No generated research content. Processing details and hashes in assets/reconstruction/evidence.json.
C:/Linux/CNRSApplication/telecom-paris-dossier/publications_gong.bib: gong2023recolornerf
Authorship, equal contribution and publication venue.
C:/Linux/CNRSApplication/telecom-paris-dossier/publications_gong.bib: wang2024bilarf; yang2024deform3dgs
Spoken context only: radiance-field processing under inconsistent capture (SIGGRAPH 2024) and dynamic surgical-scene reconstruction from video (MICCAI 2024). No result from either paper is shown or claimed on the slide.
C:/Linux/CNRSApplication/presentation-archive/2023/2023-12-13_SeamlessNeRF_Stitching-Part-NeRFs.pptx: Slide 21 embedded demonstration (media1.mp4), statue example, excerpt 49.75 to 74.4 s; provenance and hashes in assets/reconstruction/evidence.json
Authentic SeamlessNeRF demonstration: arms from a second radiance field composed onto a statue, before editing, editing in progress and after editing; no generated content.
C:/Linux/CNRSApplication/telecom-paris-dossier/publications_gong.bib: gong2023seamlessnerf
Authorship, equal contribution and publication venue (SIGGRAPH Asia 2023).

Motivation

Representations for useful visual intelligence

Different scenes can explain the same images,
yet behave differently when edited.

Which structure must be explicit to constrain reconstruction,
target an operation and verify its consequences?

The editing examples point to an ambiguity that image quality alone cannot resolve. Different arrangements of geometry and appearance can explain the observed views, yet respond differently when we change a color, compose a part or move to an unseen view. So the representation is also a choice about which ambiguities to constrain and which operations to expose. With sparse observations, I control how much detail the model can fit. For surface reconstruction, I make geometric constraints explicit. For semantic interaction, I connect a concept to a location or region. And for feedback, I ask what evidence can tell us whether the result improved. These choices are task-dependent: a palette is useful for recoloring, but it does not define a surface or a part articulation. The research question is how to choose enough structure for the intended operation, together with a way to evaluate it.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Research overview and three proposed axes
Research synthesis: image agreement does not by itself establish geometric validity, semantic localization or successful editing. The question motivates complementary studies and the proposed programme.

Research programme

Structure, meaning and feedback

What makes a reconstruction geometrically reliable?

Image agreement leaves ambiguity; control complexity and make surface constraints explicit.

FourieRF · MILo · Spokes

How precisely can meaning be grounded in geometry?

Connect concepts to points and parts; evaluate localization beyond recognition.

ZeroKey · PatchAlign3D

When does feedback improve behavior?

Judgments can train a predictor; richer evidence does not guarantee better construction.

Is It a Good Match? · 3DHarnessBench

I will develop that question in three parts. First, geometric reliability. FourieRF controls when finer detail can be learned from sparse views. MILo couples image fitting to an explicit mesh, while Spokes gives Gaussians an oriented surface interpretation. The common issue is what constrains the geometry when the images leave it ambiguous. Second, semantic grounding. Recognizing a chair leg does not locate its endpoint. ZeroKey connects language to points, and PatchAlign3D learns dense part features. Here the useful question is how precisely meaning can be localized, and how that precision transfers to new shapes. Third, feedback. Is It a Good Match uses judgments during training to improve a visual predictor. 3DHarnessBench instead tests agents during executable 3D construction: giving them richer observations does not guarantee improvement. Together they separate the availability of a signal from the ability to use it. These are complementary studies with distinct evaluation settings. I will show the mechanism and evidence for each, then return to the proposed programme.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Prior work and research programme
Logical ordering of the research contributions.
/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/4_method.tex: Fourier Parameterization: 1D Features, 2D Features and Progressive Inclusion of Coefficients
Factor transforms, vector cutoff, circular planar mask and linear bandwidth schedule. b_t renames the paper's f_t to distinguish bandwidth from a factor.
C:\Linux\CNRSApplication\presentation-archive\supporting-assets\2025\2025_MILo_Presentation-Video.mp4: 00:41–01:00; poster00:44; silent excerpt
Original volume/mesh depth and normal comparisons
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Volume-to-Surface Consistency; Eqs. mesh_depth_consistency_loss, mesh_normal_consistency_loss and mesh_consistency_loss
Exact log-depth and normal-dot-product losses, with depth-derived Gaussian normals distinguished from expected Gaussian normals.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Figure 5, PDF page 12: bicycle column, GT / MILo / Ours rows
Selected qualitative thin-structure comparison. Original embedded images, with no synthetic enhancement. Exact extraction provenance in assets/reconstruction/spokes-source-manifest.json.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Sections 4 and 5: oriented Gaussians, wrapping constraints and mesh extraction
Learned orientation, normal consistency and densification; vacancy and normal estimates support surface extraction. Static reconstruction, not demonstrated motion generation.
https://diego1401.github.io/BlobsToSpokesWebsite/: Title, author list and ECCV 2026 label; verified 20 September 2026
Publication status and equal first authorship of Diego Gomez and Antoine Guédon.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Section 5.2 and Table 1 mean rows
Two debiased protocols; MILo 0.34/0.42 and Ours (2p) 0.48/0.53 mean F1 on six T&T scenes; legacy metric sensitivity to pivot count.
C:/Linux/CNRSApplication/presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 14; crop PDF points [231, 43, 438, 141]
Chair-corner localizations from Molmo and GPT-4o in the illustrated views
/Apps/Overleaf/[ICCV'25] Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models/sec/6_experiments.tex: GPT-4o baseline and setup
Tested localizer comparison and no 3D keypoint training
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/03_method.tex: 2D Dense Feature Extraction; Stage 1: 2D-to-3D feature distillation; Stage 2: 3D–text patch contrastive learning
Visible-view and patch averaging, cosine regression, final-block-only adaptation
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Table tab:ablation_stage; Figure fig:stage_features
50.2 joint, 50.5 stage 2 only, 56.9 sequential; independent PCA projections
presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: pages 26–27 and 35
Two-stage loss explanation and feature visualization
C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: Feature computation, action sampling, VLM reward
Source keypoints are given; REINFORCE learns target correspondence
C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/figures/pipeline_sketch_v2.png: pipeline figure
Illustration; aggregation network also trainable in geo-aware variant
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/main_result-update.tex: UNI3D column across4 harnesses
Selected Astra/Opus/Gemini aggregates; preliminaryAstra
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/sec/4_experiment.tex: Main Results
Originalseven-agentinterpretation

FourieRFPublished

FourieRF: controlling detail from sparse views

Held-out view on the left; training view on the right. Color and depth reveal different solutions.

Training views can fit well while unseen views fail.

With only a few views, a flexible field can fit the photographs using the wrong geometry. The dinosaur makes this visible: both methods fit the training view on the right, but the unregularized tensor field fails at the held-out camera on the left. Depth exposes what the RGB fit hides. FourieRF, joint work with Diego Gomez and Maks Ovsjanikov, controls which spatial frequencies the field can use during training. Diego is a doctoral student I co-mentor; across FourieRF, MILo and Spokes my role includes mentoring, shared code and the GRAPHDECO collaboration. The idea is to establish coarse structure before allowing fine detail. This example motivates that prior; it does not establish the correctness of unseen surfaces.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/images/overfitting_figure.pdf: page 1; complete figure
Train/test rendering and depth comparison
https://arxiv.org/html/2502.01405v1: Motivation section
Sparse-view overfitting motivates frequency control
C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: Mentoring and open-source software
Documented roles.
C:/Linux/CNRSApplication/telecom-paris-dossier/Publications_Gong.tex: Three selected publications and software
ZeroKey conception/implementation/evaluation and MILo contributor role.

FourieRFPublished

Frequency control inside the representation

A compact 3D field sums learned line × plane factors.

Original FourieRF Figure 2: a learned line and plane are Fourier transformed, clipped by a 1D cutoff or a circular 2D mask, and inverse transformed into smoothed features. Red arrows show gradient flow. — Original method illustration · FourieRF, Fig. 2

1D: frequency cutoff.
2D: centered disk.

b_t = b_t−1 + Δ

Increase bandwidth
as training progresses.

f̃_t = F⁻¹(M_bₜ ⊙ F(f))

Render with the filtered factors; image error trains them through the mask.

The underlying TensoRF representation builds a three-dimensional feature field from sums of line-and-plane products. We sample a line at one coordinate and a plane at the other two, multiply them, and sum across components and orientations. FourieRF filters each learned factor before the renderer uses it. We transform into the frequency domain, keep the coefficients allowed by a mask, then transform back. The equation describes that operation; b is bandwidth and f is a factor. In the paper’s figure, the upper branch clips a 1D spectrum and the lower branch applies the circular 2D mask. The red arrows show gradient flow. The bandwidth increases on a prescribed schedule, gradually making finer variation available. Image-loss gradients pass through the inverse transform and mask to train the factors. We therefore control the capacity available during fitting without changing the image supervision. The full tensor equation and optimization details are in the appendix.

Sources and evidence

/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/4_method.tex: Fourier Parameterization: 1D Features, 2D Features and Progressive Inclusion of Coefficients
Factor transforms, vector cutoff, circular planar mask and linear bandwidth schedule. b_t renames the paper's f_t to distinguish bandwidth from a factor.
/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/4_method.tex: Preliminaries: Vector-Matrix (VM) Decomposition
VM factor interpretation: a 3D feature field is represented by sums of learned line-plane products.
https://arxiv.org/pdf/2502.01405v1: Figure 2, PDF page 2; complete diagram, excluding article text and caption
Original method illustration showing 1D and 2D Fourier coefficient clipping, inverse transforms and gradient flow. High-resolution PDF rendering; exact crop and hashes in assets/reconstruction/evidence.json.

FourieRFPublished

Coarse structure before fine detail

RGB and depth over training, from top to bottom. FourieRF on the left; TensoRF on the right. — Training proceeds downward. Left: FourieRF. Right: TensoRF.

Releasing detail gradually changes the geometry learned from the same images.

Read the flower example from top to bottom. FourieRF begins with a smooth explanation and adds detail as the bandwidth expands. TensoRF sharpens the image early, but its depth stays fragmented. This illustrates why the training path matters: allowing all the detail immediately can lock the model into an explanation that fits the photographs without coherent geometry. The schedule ablation in the appendix shows the tradeoff. Releasing frequencies too quickly weakens regularization; releasing them too slowly can underfit the available training budget. The result supports controlling capacity over time, while leaving genuinely unobserved geometry uncertain.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/images/learning_low_frequencies.pdf: page 1; complete training progression
Original RGB/depth learning sequence
https://arxiv.org/html/2502.01405v1: Motivation and progressive frequency schedule
Coarse-to-fine interpretation

FourieRFPublished

A measured speed–quality tradeoff

5.57× faster training than ZeroRF,
with 0.212 dB lower PSNR.

Method	Relative training time ↓	4-view Blender PSNR ↑
FourieRF-VM	0.930	21.728 dB
ZeroRF	5.181	21.940 dB

Reported Blender setup: 10,000 iterations, RTX 4090. Training time normalized to TensoRF = 1.

Frequency control regularizes the field. Geometric use also needs an explicit surface.

The quantitative result is a speed-quality tradeoff. FourieRF takes 0.930 relative training time against 5.181 for ZeroRF, giving a ratio of 5.57. Their four-view Blender PSNR values are 21.728 and 21.940 decibels, so FourieRF gives up 0.212 decibels. This is the reported ten-thousand-iteration setting on an RTX 4090, with time normalized to TensoRF. The conclusion is competitive reconstruction with a lightweight frequency prior, without a learned external prior. It is not a universal quality lead: FreeNeRF has higher PSNR on the reported LLFF settings. Finally, frequency control regularizes a field, but collision, simulation and geometric editing need a surface. MILo and Spokes address that surface question through mesh supervision and oriented Gaussian geometry.

Sources and evidence

https://arxiv.org/html/2502.01405v1: Main Blender metrics table and normalized training-time table
FourieRF .93; ZeroRF 5.181; FreeNeRF35.71; four-view PSNR21.728/21.94
C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/sec/X_suppl.tex: Implementation details
RTX4090 and10k iterations
https://arxiv.org/html/2502.01405v1: Computed:5.181 / 0.93 = 5.570967741935484;21.940 - 21.728 =0.212 dB
5.57× is a runtime ratio;0.212dB is the associated 4-view PSNR difference
https://arxiv.org/html/2502.01405v1: Main LLFF table and conclusion
Metric-dependent tradeoffs and unobserved geometry limitation

MILo + SpokesPublished

MILo and Spokes: surfaces from Gaussian splatting

How can Gaussians fitted to images describe a usable surface?

MILo · TOG / SIGGRAPH Asia 2025

From Blobs to Spokes · ECCV 2026

MILo teaser crop: bicycle and bench, with extracted mesh on the left and rendered appearance on the right

Spokes teaser crop: reconstructed bicycle and garden scenes with detailed wheel spokes and plant geometry; multiview input diagram omitted

Let the extracted mesh
constrain training.

Give each Gaussian
an outward surface orientation.

MILo and From Blobs to Spokes address the same problem: turning an image-trained Gaussian scene into an explicit surface that other tools can use. MILo makes an extracted mesh participate in training. Spokes, the follow-up, changes the geometric interpretation of the Gaussians themselves. These are distinct reconstruction methods pursuing the same surface goal. MILo is joint work with Antoine Guédon, Diego Gomez, Nissim Maruani, George Drettakis and Maks Ovsjanikov. Spokes is with Diego, Antoine, Nissim and Maks; Antoine and Diego share first authorship on both. My MILo contributions include code and the LIX and GRAPHDECO collaboration. The left shows MILo’s bicycle mesh and rendering. The right shows Spokes’ bicycle and garden reconstructions, including wheel spokes and plant details. These views illustrate each method; the matched comparison comes later. I will explain the mesh constraint first, then the oriented representation, and compare their surfaces under a common evaluation.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/main.tex: Author block and title
Complete authors, equal contribution and publication metadata
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/images/teaser.pdf: page 1; upper-left bicycle mesh/render crop from the original teaser
MILo bicycle and bench surface/render illustration; benchmark insets and statue example omitted from this introductory crop.
C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: Mentoring and open-source software
Documented roles.
C:/Linux/CNRSApplication/telecom-paris-dossier/Publications_Gong.tex: Three selected publications and software
ZeroKey conception/implementation/evaluation and MILo contributor role.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Sections 4 and 5: oriented Gaussians, wrapping constraints and mesh extraction
Learned orientation, normal consistency and densification; vacancy and normal estimates support surface extraction. Static reconstruction, not demonstrated motion generation.
https://diego1401.github.io/BlobsToSpokesWebsite/: Title, author list and ECCV 2026 label; verified 20 September 2026
Publication status and equal first authorship of Diego Gomez and Antoine Guédon.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Figure 1, PDF page 1; reconstruction output montage cropped from the original teaser
Spokes bicycle and garden surface reconstructions; multiview inputs, arrows and Gaussian Wrapping banner omitted. Exact display crop is recorded in assets/reconstruction/spokes-source-manifest.json.

MILo + SpokesPublished

MILo: the extracted mesh constrains training

Gaussian rendering and extracted mesh rendering linked by depth and normal losses — The same cameras render both representations. Mesh losses update the parameters that produce the surface.

ℒ = ℒ_vol + ℒ_mesh + ℒ_reg

Image fitting and Gaussian normals + mesh depth/normal agreement + surface regularization

MILo keeps two views of the scene during training. The upper branch renders the Gaussians and fits the captured images. The lower branch extracts a mesh and rasterizes its depth and face normals with the same camera. Their agreement makes the surface affect learning. The objective combines volumetric image fitting and normal regularization, Gaussian-to-mesh consistency, and surface regularizers. The key technical step is to connect a mesh vertex continuously to the Gaussian parameters. We place adaptive sample points around selected Gaussians, then interpolate a surface between points with opposite signs. This gives mesh losses a gradient path back to the representation.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/figures/src/trailer.pdf: page1; complete pipeline
Bidirectional volume/surface training relationship
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Method overview and mesh supervision
Depth and normal coupling
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Overview; Mesh-in-the-Loop Optimization; Eq. final_loss
Full objective decomposes into volumetric, mesh-consistency and regularization losses.

MILo + SpokesPublished

MILo: surface vertices follow the Gaussians

Original 2D schematic of Gaussian-aligned pivots with signed scalar values — Nine pivots in 3D: the Gaussian center and eight aligned box corners.

p_k,i = μ_k + R_k(s_k ⊙ b_i)

Position μ, rotation R and scale s place each pivot p.
b_i: unit-box center or corner.

A learned scalar f marks each pivot as inside or outside.

v = (fₐpᵦ − fᵦpₐ) / (fₐ − fᵦ)

Opposite signs locate the zero crossing: a mesh vertex v.

Mesh losses update both the signed values and the Gaussian geometry.

Marching tetrahedra needs samples on both sides of the surface. Each selected Gaussian supplies nine pivots: its center and eight aligned box corners. The first equation shows how position, rotation and scale place those pivots; b selects the unit-box center or a corner. Each also carries an independently learned signed scalar, rather than a calibrated distance. Delaunay tetrahedra connect them. Along an edge with opposite signs, the second equation interpolates the zero crossing, which becomes a surface vertex. Changing the scalar moves the crossing; changing the Gaussian moves the pivots. Mesh losses can therefore update both. Connectivity is discrete and refreshed every five hundred iterations, while mesh extraction and rendering happen each iteration. The complete gradient derivation and selection variants are in the appendix.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/figures/src/gaussians_tetrahedralization.pdf: page1; original schematic
Signed pivot illustration
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Sampling Delaunay Vertices from Gaussians; Computing Signed Distance Values
Nine-pivot formula, rendering-importance sampling and independently learned signed scalars. Base and dense selection behavior.
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Differentiable Marching Tetrahedra; Eq. marching_tetrahedra
Exact interpolation equation and two gradient paths through signed scalars and pivot coordinates. Editable edge diagram is conceptual.
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/X_supplemental.tex: Implementation Details: Delaunay triangulation
Connectivity is non-differentiable and recomputed every 500 iterations; mesh extraction is per iteration.

MILo + SpokesPublished

MILo: depth and normals align volume and surface

Gaussian and mesh depth and normal renderings during training — Gaussian and mesh renderings from the same camera

Depth locates the visible surface

ℒ_MD = ∑_i log(1 + |D(i) − D_M(i)|)

D: Gaussian depth · D_M: mesh depth

Normals align its orientation

ℒ_MN = ∑_i [1 − Ñ(i) · N_M(i)]

Ñ: normal from Gaussian depth
N_M: rasterized mesh-face normal · i: pixel

ℒ_mesh = λ_MDℒ_MD + λ_MNℒ_MN. Both renderings should describe the same surface.

The consistency clip now has a precise objective behind it. At every pixel, the depth term compares the Gaussian depth with the depth of the extracted mesh. Its logarithmic penalty discourages discrepancies without giving very large errors unlimited influence. The normal term compares the mesh face normal with a normal computed by finite differences from the Gaussian depth map. Their dot product is one when the directions agree, which gives zero loss. This normal target is worth distinguishing from the expected Gaussian normal used in the separate volumetric normal regularizer. Both mesh terms send gradients through the rendered surface and its construction. RGB fitting anchors the Gaussian scene to the observations, while these terms encourage its extracted mesh to occupy and face the same visible surface. Additional regularizers preserve selected Gaussian centers inside the surface and suppress internal artifacts; their equations are in the appendix. The depth and normal terms describe visible geometry, so occluded regions still need these additional constraints. Spokes pursues the same geometric goal by giving each Gaussian an explicit outward orientation.

Sources and evidence

C:\Linux\CNRSApplication\presentation-archive\supporting-assets\2025\2025_MILo_Presentation-Video.mp4: 00:41–01:00; poster00:44; silent excerpt
Original volume/mesh depth and normal comparisons
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Volume-to-Surface Consistency; Eqs. mesh_depth_consistency_loss, mesh_normal_consistency_loss and mesh_consistency_loss
Exact log-depth and normal-dot-product losses, with depth-derived Gaussian normals distinguished from expected Gaussian normals.

MILo + SpokesPublished

Spokes: oriented Gaussians define a surface boundary

Learn outward normals; encourage the Gaussians to wrap the surface.

Original Spokes method diagram: oriented Gaussians define normal and vacancy estimates for tetrahedral extraction and optional Primal Adaptive Meshing — Original method figure: normal and vacancy estimates support extraction.

Surface fields

Normal estimates describe orientation.
Vacancy estimates distinguish outside from inside.

Mesh extraction

Two pivots per Gaussian locate the boundary.
Optional PAM adapts mesh resolution.

MILo learns signed values on pivots. Spokes derives fields from oriented Gaussians.

Spokes changes the meaning of the primitive. A symmetric density Gaussian does not identify an outward side. We learn an outward normal for each Gaussian and use normal consistency plus targeted densification to encourage a complete surface wrapping. The resulting representation supplies normal and vacancy estimates. Vacancy describes whether a query lies outside the reconstructed object; the theoretical construction assumes consistent wrapping, which real scenes only approximate. For the initial mesh, each Gaussian supplies its center and an outward-displaced pivot. Marching tetrahedra uses the estimated vacancy to locate the boundary. Primal Adaptive Meshing, or PAM, is an optional refinement that can choose mesh resolution independently of Gaussian count. This is the key distinction from MILo: learned signed pivot values coupled to mesh losses there, oriented Gaussian fields here. The two methods share the surface-reconstruction objective, rather than forming successive stages of one training pipeline.

Sources and evidence

https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Figure 3, PDF page 7; Sections 4 and 5
Oriented fields and two-pivot Marching Tetrahedra; optional Primal Adaptive Meshing. Practical fields are estimates under model assumptions.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Sections 4 and 5: oriented Gaussians, wrapping constraints and mesh extraction
Learned orientation, normal consistency and densification; vacancy and normal estimates support surface extraction. Static reconstruction, not demonstrated motion generation.
https://diego1401.github.io/BlobsToSpokesWebsite/: Title, author list and ECCV 2026 label; verified 20 September 2026
Publication status and equal first authorship of Diego Gomez and Antoine Guédon.

MILo + SpokesPublished

Thin structures reveal the difference

Input photograph of the bicycle wheel — Input photograph

MILo reconstruction of the same bicycle wheel — MILo

Gaussian Wrapping reconstruction with finer bicycle spokes — Spokes / Gaussian Wrapping

Bicycle scene: input photograph and surface-normal visualizations.

Oriented Gaussian wrapping recovers more of the wheel’s thin spokes in this example.

Here both methods reconstruct the same bicycle wheel. The left panel is the input photograph; the other two are surface-normal visualizations from the Spokes paper. MILo recovers much of the structure but leaves fragmented spokes. Gaussian Wrapping preserves more continuous spokes in this selected example. This shows what the two methods recover, but it does not establish a gain for every object or every region. To assess the methods more broadly, we need to separate output complexity from geometric accuracy. I will first show MILo’s resource result, then evaluate both methods with the same surface-sampling protocols.

Sources and evidence

https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Figure 5, PDF page 12: bicycle column, GT / MILo / Ours rows
Selected qualitative thin-structure comparison. Original embedded images, with no synthetic enhancement. Exact extraction provenance in assets/reconstruction/spokes-source-manifest.json.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Sections 4 and 5: oriented Gaussians, wrapping constraints and mesh extraction
Learned orientation, normal consistency and densification; vacancy and normal estimates support surface extraction. Static reconstruction, not demonstrated motion generation.
https://diego1401.github.io/BlobsToSpokesWebsite/: Title, author list and ECCV 2026 label; verified 20 September 2026
Publication status and equal first authorship of Diego Gomez and Antoine Guédon.

MILo + SpokesPublished

Mesh complexity and surface accuracy

Surface sampling evaluates geometry independently of vertex density.

Method	Mesh vertices ↓ Mesh-rendering evaluation	Uniform F1 ↑	Virtual-scan F1 ↑
GOF	11.63 M	0.31	0.38
RaDe-GS	10.06 M	0.35	0.39
MILo	4.36 M	0.34	0.42
Spokes	5.80 M	0.48	0.53

Tanks & Temples. Vertices: mesh-rendering evaluation (Table 2). F1: six-scene surface evaluation (Table 1); Spokes F1 uses two pivots, without PAM.

MILo has the fewest reported vertices; Spokes has the highest F1 among these methods.

The table separates mesh complexity from surface accuracy. All numbers come from the Spokes paper, but the columns summarize two separately reported evaluations. The mesh-rendering experiment reports 4.36 million vertices for MILo and 5.80 million for Spokes, compared with 10.06 million for RaDe-GS and 11.63 million for GOF. MILo has the smallest mesh by vertex count among these methods. For surface evaluation, we avoid the legacy metric that can reward denser tessellation. Uniform sampling draws points over the mesh surface; virtual scanning renders depth from the input viewpoints to approximate acquisition visibility. Across six Tanks and Temples scenes, MILo scores 0.34 and 0.42 mean F1; two-pivot Spokes scores 0.48 and 0.53. These F1 values exclude optional PAM. The paper does not identify the MILo variant in either table, so this is a summary of reported results, not a verified same-checkpoint size-versus-accuracy curve. Original MILo storage and timing details remain in the appendix.

Sources and evidence

https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Table 1, PDF page 13, mean rows; full-scene GOF, RaDe-GS, MILo and Ours (2p) columns
Uniform F1: 0.31/0.35/0.34/0.48; virtual-scan F1: 0.38/0.39/0.42/0.53 across six T&T scenes. Spokes uses two pivots; PAM excluded.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Table 2, PDF page 14, right-hand Tanks & Temples #Verts column
Reported mesh-rendering evaluation vertex counts in millions: GOF 11.63, RaDe-GS 10.06, MILo 4.36, Ours 5.80. These are separately reported evaluations from Table 1; identical checkpoints and MILo variants are not established.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Sections 5.1–5.2, PDF pages 11–12
Baselines evaluated under common protocols; uniform surface sampling removes vertex-density bias, virtual scanning approximates acquisition visibility.

Reconstruction → semanticsPublished

Usable geometry still needs semantic handles

MILo meshes used in downstream simulation.

A surface can support interaction.

But geometry alone does not identify a hinge, a wingtip, or the part a user wants to edit.

The next question is how language can locate meaningful structure.

Together, MILo and Spokes connect Gaussian reconstruction to explicit surfaces: through mesh supervision in MILo and oriented fields in Spokes. The demonstration here uses MILo’s static meshes in water and smoke simulations. It shows exported geometry entering existing tools; recovering motion from video is a separate problem. Neither a mesh constraint nor an outward orientation identifies which location is a hinge, which point is a wingtip, or which part a user intends to edit. For those operations we need semantic handles. ZeroKey asks whether multimodal models can locate them through language and known camera geometry. It consumes renderable shapes; this is a connection between research questions, rather than a shared pipeline.

Sources and evidence

C:\Linux\CNRSApplication\presentation-archive\supporting-assets\2025\2025_MILo_Presentation-Video.mp4: 01:32–01:38; poster01:35; silent excerpt
Static exported meshes used in water/smoke demonstrations
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/5_conclusions_and_limitations.tex: Future directions
Dynamic scenes remain future work

ZeroKeyPublished

ZeroKey: connecting names to 3D points

Locate named 3D points through agreement across rendered views.

ZeroKey locates named keypoints on an unseen airplane mesh

Query pretrained image models on a known mesh.
Zero-shot: no task-specific 3D keypoint training.

Bingchen Gong, Diego Gomez, Abdullah Hamdi, Abdelrahman Eldesokey,
Ahmed Abdelreheem, Peter Wonka and Maks Ovsjanikov

A mesh gives us coordinates, but language refers to meaningful locations: a wingtip, a nose, or a chair-back corner. ZeroKey connects a point description to a location on a known 3D shape. This is my first-author project; I conceived, implemented and evaluated the method. We transfer the image-pointing ability of pretrained multimodal models into 3D without training a new 3D keypoint detector. A name selects the target, while agreement across rendered views reconciles the spatial hypotheses. This makes pretrained image models usable on a known mesh; the geometry supplies the common 3D reference. Zero-shot refers to that task-specific training requirement, not to an absence of foundation-model pretraining. The central question is which locations language can identify precisely enough to become repeatable spatial references.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Title, author list, abstract and Sections 3–4
Task: named 3D keypoints on a known shape, without task-specific 3D keypoint training.
C:/Linux/CNRSApplication/telecom-paris-dossier/figs/zerokey_teaser.pdf: page 1, complete original figure
Project teaser and qualitative task illustration
C:/Linux/CNRSApplication/telecom-paris-dossier/Publications_Gong.tex: Three selected publications and software
ZeroKey conception/implementation/evaluation and MILo contributor role.

ZeroKeyPublished

Language queries become 3D point hypotheses

Original ZeroKey pipeline: a text query produces 2D points in rendered views, which are lifted and aggregated in 3D

Language → pixels
GPT-4o proposes names.
Molmo points in 26 rendered views.

Pixels → 3D hypotheses

x_j = π_j⁻¹(u_j, D_j(u_j))

u_j: predicted pixel; x_j: 3D candidate.
Known camera π_j; cached mesh depth D_j.

The mesh supplies geometry; the model supplies the point’s meaning.

GPT-4o proposes candidate names, such as nose or wingtip. Molmo answers each query with image coordinates in rendered views. We treat its predictions as a black box; the attention graphic in the original figure does not mean that we extract an attention map. The camera and cached mesh depth lift each pixel to a visible 3D surface point, as the compact inverse-projection equation shows. A small pointing error can jump across a silhouette or another surface, so these are hypotheses that need reconciliation. The default uses 26 views. Automatic naming is one mode of the system; KeypointNet evaluation instead uses human-written descriptions of its annotated keypoint IDs. Those descriptions specify what to find, while geometric aggregation decides which predictions agree.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Figure 2 and Sections 4.1–4.3
GPT-4o candidate naming, Molmo image coordinates, known-camera/depth lifting and multiview aggregation. Molmo used as a black box.
https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Section 4, Setup and Dataset
Benchmark uses manually written text descriptions for semantic keypoint IDs; this differs from automatic candidate generation.
https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 6.4, Aggregation ablation
Default 26 views.

ZeroKeyPublished

A name selects the target; 3D agreement stabilizes it

Original ablation plot: relative IoU for specific versus global prompts, soft versus binary back-projection and HDBSCAN versus direct averaging — Relative IoU; full ZeroKey = 1 at each threshold.

Local uncertainty
Lift an h × h neighborhood; use Gaussian support weights.

Agreement in 3D
HDBSCAN separates consistent groups and rejects isolated detections.

One name can denote several corners. Preserve separate spatial hypotheses.

Global prompts, binary weights and direct averaging all reduce performance in this experiment.

The method has to stabilize both the image-to-surface mapping and the agreement between views. First, it lifts a small neighborhood around each predicted pixel, excludes background pixels and averages the valid 3D locations. Gaussian weights then measure support from contributing image neighborhoods; they are not calibrated confidence scores from Molmo. Second, HDBSCAN finds dense groups among the pooled 3D candidates. A query can refer to several valid corners, so a single global mean could fall between them. The ablation shows why the language and geometry choices both matter: replacing specific names with a global salient-points prompt hurts, and direct averaging without clustering is especially damaging. Binary voting also loses accuracy. The vertical axis is relative IoU, normalized separately at each tolerance. Agreement here means consistency across rendered views, not demonstrated temporal or deformation stability.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 4.3, equations 3–4
Neighborhood depth back-projection, Gaussian support weights and HDBSCAN over multiple views. Distinguish multiple valid instances from outliers.
https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 6.4 and Figure 7
Ablation normalized to default ZeroKey at each geodesic threshold; point-specific prompting, soft voting and clustering contribute.
https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Sections 6–7 and Table 3
Additional patchify/clustering discussion; binary back-projection baseline is 54.64 at geodesic threshold 0.10.

ZeroKeyPublished

Semantic grounding still leaves a precision gap

KeypointNet: airplane, chair and table. IoU (%) at geodesic distance thresholds.

Method	0.01	0.05	0.10
GPT-4o localizer	0.48	6.04	20.73
B2-3D · few-shot, 62 views	20.29	57.72	70.57
ZeroKey · 26 views	13.16	56.60	79.43

Benchmark queries are human-written descriptions of keypoint IDs.
ZeroKey requires no task-specific 3D keypoint training.

Names help identify the intended structure; tight tolerances expose localization errors.

This comparison separates semantic grounding from exact localization. Replacing the Molmo localizer with GPT-4o gives 20.73 IoU at the 0.10 tolerance, compared with 79.43 for ZeroKey. In this tested pipeline, the point-capable image model supplies much more useful localization evidence. That comparison does not isolate every difference between the two pretrained models. ZeroKey is competitive with few-shot B2-3D at a moderate tolerance, while B2-3D is stronger at 0.01 and 0.05. The displayed B2-3D row uses 62 views; its matched 26-view result is 56.90 at 0.10. The benchmark uses human-written descriptions of KeypointNet IDs for airplane, chair and table. This leaves a deeper question: are all surface points equally easy to communicate through language?

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Table 1 and Sections 6.1–6.3
Published values: GPT-4o 0.48/6.04/20.73, B2-3D 20.29/57.72/70.57, ZeroKey 13.16/56.60/79.43 at thresholds 0.01/0.05/0.10.
https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Sections 4–5 and Table 3
Human-written benchmark prompts; B2-3D 62-view row and matched 26-view result 14.39/44.11/56.90. 54.64 belongs to binary back-projection, not the final ZeroKey row.

ZeroKeyPublished

Can language recover the point it describes?

Nameability test: marked point p → short description ℓ → recovered point p̂
Keep the most repeated description; query new views with the marker removed.

Original salient and non-salient examples: describe the green marked point and retrieve the blue point; some arbitrary points yield no prediction — Human saliency: Schelling-point selection density.

Original retrieval-accuracy plot: salient points are more accurately recovered than non-salient points across the tested absolute 3D error thresholds — Retrieval stability: recovery within a 3D error tolerance.

Human-salient points are easier to recover from their descriptions.

Interpretation: distinctive names can give 3D points shared, repeatable identities.

This experiment tests the connection between points and language in both directions. Saliency comes from human Schelling-point annotations: locations people independently choose receive high density. We select a highest-saliency point or a zero-saliency point, mark it in several views, and ask Molmo for a short description. We keep the most repeated description, remove the marker, and ask ZeroKey to recover the point from new views. The graph measures retrieval accuracy at different absolute 3D error tolerances; it is not the geodesic IoU from the previous slide. Human-salient points are consistently easier to recover in this experiment. Arbitrary locations may receive an ambiguous description, lead to another instance, or produce no point. Nameability concerns how a description identifies a location; retrieval stability concerns whether that description leads back to it. Their relationship with human saliency is empirical, not an equivalence or a proof of causation. My interpretation is that language gives distinctive points shared identities and geometry tests whether their localizations agree. These sparse handles leave much of the surface unnamed. PatchAlign3D addresses that gap by learning language-aligned local features across the shape.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Sections 5.1–5.2 and Figures 4–5
Human Schelling-point density defines salient/non-salient samples; qualitative and quantitative results show more successful recovery for salient points across tested thresholds.
https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Section 3, Algorithm 1, PDF page 2
Highest-saliency versus zero-saliency points; marked views produce short names, most repeated prompt retained; new unmarked views used for recovery; absolute 3D error.

PatchAlign3DPublished

PatchAlign3D: language-aligned local features

Encode the shape once; query regions with part names.

Point-cloud features compared with text-defined parts — A trained 3D encoder matches local features to text embeddings.

Several part names can describe the same patch.

Souhail Hadgi, Bingchen Gong, Ramana Sundararaman, Emery Pierson,
Lei Li, Peter Wonka and Maks Ovsjanikov

ZeroKey asks where a named point is. PatchAlign3D asks what language can describe throughout a shape. Instead of repeatedly asking an image model to point, we train a 3D encoder to produce local features that can be compared with many text queries. This is joint work with Souhail Hadgi and the collaborators listed here. A point cloud is encoded once, and part names such as cap, body or lens select regions through feature similarity. Find3D also uses feed-forward inference; our contribution is how the local representation is learned. The important change in supervision is that a patch can support several overlapping descriptions. Fractional support and multiple positive names preserve this overlap in the contrastive objective. The reusable representation learns from visual features and pseudo-part masks, not from ZeroKey outputs. Regional grounding complements named point localization, but does not establish ZeroKey’s human-saliency or point-retrieval stability result.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Abstract, Sections 3.1–3.3 and Figure 1
Language-aligned local patch features, two-stage pretraining and text-based segmentation. Find3D is also feed-forward.
C:/Linux/CNRSApplication/telecom-paris-dossier/figs/patchalign3d_teaser.pdf: page 1, complete original figure
Project teaser and qualitative task illustration
C:/Linux/CNRSApplication/telecom-paris-dossier/publications_gong.bib: hadgi2026patchalign3d
Complete project author list and publication venue

PatchAlign3DPublished

Visual distillation, then local language alignment

Original PatchAlign3D training pipeline, with frozen visual and text encoders and the trainable last transformer block highlighted — 2,048 XYZ points · 128 overlapping patches of 32 points · 12 transformer blocks

Stage 1 · visual structure
Match 3D patch features to frozen DINOv2 targets
with a cosine loss.

Stage 2 · language alignment
Freeze the first 11 blocks; train the last block
and projection into OpenCLIP’s text space.

The input is 2,048 XYZ points. Farthest-point sampling chooses 128 centers, each with 32 nearest neighbors. PointNet encodes each neighborhood, a centroid embedding provides position, and a 12-block transformer shares context across patches. Stage one distills a frozen visual teacher: DINOv2 features are averaged over visible views and then over patch points. The cosine loss trains the encoder to reproduce the direction of each visual target d_i. Each online patch uses the target of its nearest cached patch center. Stage two initializes from those weights, freezes the earlier layers, and trains only the last block and a text projection against OpenCLIP embeddings. At inference, the visual teacher and renderer are absent. Each point receives the prediction of its nearest patch center. This separation lets fine visual structure guide initialization before coarser part names shape the representation.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Figure 2; Sections 3.2–3.3, equations 1–3; Section 4.1 implementation
Patch construction, cosine distillation, cached visual targets, frozen early layers, OpenCLIP text targets and inference.
https://arxiv.org/pdf/2601.02457v2: Appendix A, Table 9 and accompanying paragraph
The correct setting unfreezes the last block and head; the table row label says “Freeze last block” but its caption and method clarify the intended setting.

PatchAlign3DPublished

A patch can match several part descriptions

Training labels: 10 rendered views → SAM regions → Gemini part names → 3D pseudo-labels.

Fractional target y_ij
Share of patch i annotated with name j.

Name	Annotated points	y_ij
Nose	24 / 32	0.75
Muzzle	32 / 32	1.00
Ear	0 / 32	0.00

Illustrative patch with overlapping annotations.
The targets need not sum to one.

“Nose” and “muzzle” can both be valid positives.

Original text-similarity maps for ear, nose, muzzle and face: Find3D compared with PatchAlign3D — Text-similarity maps; yellow means higher similarity.

The language signal comes from an automatic data engine, following Find3D. SAM segments rendered images, Gemini gives short part names, and back-projection maps those masks onto the shape. Multiple views can assign overlapping descriptions to the same point. We retain those labels. This illustrative 32-point patch has 24 points annotated as nose and all 32 annotated as muzzle, while ear has no support. Both nose and muzzle are positive descriptions, and their fractions sum to more than one. A boundary patch also naturally receives fractional support instead of an arbitrary majority label. The maps on the right are separate qualitative evidence: related queries activate overlapping regions. They are similarities, not calibrated probabilities. ZeroKey highlighted the value of a distinctive point name. Here language also describes nested or overlapping regions, so the learning problem is many-to-many rather than one patch paired with one exclusive word.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Section 3.2 training data; Section 3.3 fractional labels; Appendix B, Figure 6
SAM/Gemini pseudo-labels, retained overlap, fractional patch targets and qualitative text response. The 24/32 example is explanatory, not a measured sample.

PatchAlign3DPublished

Contrastive learning needs the right negatives

s_ij = ⟨q_i, t_j⟩ / τ + b, p_ij = sigmoid(s_ij)

L_text = −∑_i,j [y_ij log p_ij + (1 − y_ij) log(1 − p_ij)]

q_i = h_text(z_i): projected patch feature; t_j: text embedding; τ: learned temperature; b: learned bias.

Independent patch–text pairs
Sigmoid allows several positive names.

Negatives within each shape
Use its valid part names with y_ij = 0.

ShapeNetPart · mIoU (%)

Negative sampling	mIoU
Within shape	56.9
Also across shapes	45.3

A “leg” on another shape may be a false negative, not a different concept.

The contrastive unit is a patch and a part-name embedding within one shape. q_i is the text-projected patch token. A learned temperature and bias turn similarity into a sigmoid score, and binary cross-entropy fits the fractional target. This is the paper's SigLIP-style objective; OpenCLIP supplies the text embeddings. Each pair is scored independently, so nose and muzzle can both match the same patch. The contrast comes from other valid names in that shape with zero annotated overlap. We do not automatically make labels from other shapes negatives: a leg on another chair can express the same concept. Incomplete pseudo-labels also make absence of an annotation weak evidence of semantic difference. The supplement tests this choice: adding cross-shape negatives reduces ShapeNetPart mIoU from 56.9 to 45.3. This supports the negative-sampling decision in this setting; it does not prove that every cross-shape pair is harmful or that within-shape labels are noise-free.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Section 3.3, equations 4–5
Paper objective: projected patch/text similarities with temperature and bias, independent sigmoid BCE, fractional targets and within-shape negatives.
https://arxiv.org/pdf/2601.02457v2: Appendix A, Table 12
Within-shape negatives 56.9 mIoU; adding cross-shape negatives 45.3, on ShapeNetPart.

PatchAlign3DPublished

Visual structure first, language alignment second

DINOv2, Stage 1 and Stage 1 plus Stage 2 local features visualized with independent PCA — Independent PCA projections; colors do not match across columns.

Training order matters

ShapeNetPart · mIoU (%)

Training	mIoU
Language alignment only	50.5
Both losses jointly	50.2
Visual → language	56.9

Preserve local visual structure while learning what its regions are called.

Point-cloud baselines: COPS 25.6; Find3D 23.3 mIoU. Reported runtime: PatchAlign3D and Find3D both 0.4 s/shape.

The visual target describes more local variation than a small set of part names. The two-stage design first learns that structure, then adapts its final block to language. On ShapeNetPart, language alignment alone reaches 50.5 mIoU, joint optimization 50.2, and sequential training 56.9. This is evidence for the training schedule; interference between the two signals is the paper's interpretation, not a separately measured causal mechanism. The feature plots use independent PCA projections, so a color in one column does not mean the same numerical feature in another. The final model scores 56.9 versus COPS at 25.6 and Find3D at 23.3 under the reported point-cloud protocol. It retains Find3D's reported 0.4-second inference time. The benefit comes with limits: a fixed patch resolution and nearest-center propagation restrict boundaries, while real scans still produce a low 22.7 mIoU on ScanObjectNN.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Figure 5; Tables 1, 3, 5 and 6
Independent PCA, training schedule ablation, segmentation baselines, runtime and remaining real-scan limitations.

PatchAlign3DPublished

Patch features can support ZeroKey’s point proposals

Semantic support for voting
Use patch–text similarity to modulate ZeroKey’s soft-voting weights.

Local features for clustering
Add learned feature cues when grouping the 3D point proposals.

Reported KeypointNet comparison · IoU (%) at geodesic distance thresholds

Method	0.01	0.05	0.10
ZeroKey	13.16	56.60	79.43
PatchAlign3D + ZeroKey	10.42	58.87	81.27

IoU improves at 0.05 and 0.10, but falls at 0.01.

The hybrid still uses ZeroKey’s multiview pipeline.

The connection is already tested in the updated paper. PatchAlign3D supplies local features to ZeroKey in two places: patch-to-text similarity modulates soft-voting weights, and local feature cues enter clustering. We retain ZeroKey's rendered-view point proposals. The supplement describes these two interventions but does not give a complete hybrid weighting formula, so I do not invent one here. Its comparison reports 58.87 versus 56.60 IoU at 0.05 and 81.27 versus 79.43 at 0.10. At the tight 0.01 threshold, the score falls from 13.16 to 10.42. Learned regional semantics therefore provide useful complementary evidence, without guaranteeing more precise coordinates. The table comes from the arXiv v2 supplement's direct comparison; its standalone keypoint experiment has a different table and should not be merged into this one. This hybrid uses a trained part encoder, while neither component is trained on target keypoint labels in this setting. Next, can a model judge proposed matches and use those judgments to train features that transfer?

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Appendix C, “Integration with ZeroKey”, PDF pages 15–16 and Table 14
Actual hybrid: patch/text similarities modulate soft voting and local feature cues aid clustering. Only ZeroKey and hybrid rows reproduced; improvement at 0.05/0.10, deterioration at 0.01.

Learning from feedbackOngoing work

Is It a Good Match? A VLM as the training judge

Given two images and a source point, learn its target match.

ZeroKey
Query an image model for points during inference.

Good Match
Judge visual proposals during training; infer without a VLM.

Bingchen Gong · Viktoria Ehm · Emery Pierson · Maks Ovsjanikov

ZeroKey asks an image model where a named point is in rendered views of a known mesh. Here the task is 2D correspondence: given a source point in one image, find the same semantic part in another. DINO proposes the target location and a frozen VLM scores the marked pair during training. REINFORCE transfers this feedback into a small visual adapter. At test time, matching uses the adapted visual features without a VLM call. The source point is supplied; we do not learn a keypoint detector or train ZeroKey from these judgments. PatchAlign3D learned local features from visual targets and part names; this project asks whether evaluations of proposed matches can instead guide adaptation. This is joint work with Viktoria Ehm, Emery Pierson and Maks Ovsjanikov. I led the scientific direction, experiments and writing.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/0_abstract.tex: Abstract
Training-time frozen VLM judgment; visual-only inference
/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: Problem Formulation; Figure pipeline; REINFORCE Training
Given source keypoint, sampled target correspondence, frozen judge and LoRA update
https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Sections 4.1–4.3
ZeroKey queries image models and geometrically reconciles points in rendered views of a known shape.
C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: supervision table, Viktoria Ehm
Personal contribution

Learning from feedbackOngoing work

A one-step policy over target pixels

Context sImages I₁, I₂ and source point p

→

Action qOne sampled target pixel

→

Reward r ∈ [0, 1]“Same semantic part?”A frozen VLM scores the marked pair.

Feature distance → policy

D(q) = ‖f₁(p) − f₂(q)‖₂

π_θ(q | s) = exp(−D(q)/τ) / Σ_q′ exp(−D(q′)/τ)

Shared encoder: f₁ = g_θ(I₁), f₂ = g_θ(I₂).
τ = 0.1 controls sampling diversity.

Adapt only the LoRA weights

W′ = W + (α / r_L) BA

W stays frozen. A and B learn.
Low-rank updates to ViT Q/K/V projections.

LoRA rank r_L = 16; scale α = 8.

Train: sample q ∼ π_θ. Test: take argmax_q π_θ(q | s), without a judge.

Each image pair and supplied source point define a context. We choose one target pixel and receive an immediate score, so this is a one-step reinforcement-learning problem, also called a contextual bandit. The policy comes from dense feature matching: compare the source feature with every target feature, then apply softmax to negative Euclidean distances. Temperature controls how broadly we explore the candidate locations. Sampling provides the stochastic action whose log probability is used by REINFORCE. At inference we choose the maximum-probability location. Both images share the adapted DINOv3 encoder. The pretrained weights remain frozen while the low-rank A and B matrices change the Q, K and V projections. I use r_L for LoRA rank to distinguish it from the reward r. The optional diffusion-feature fusion branch is detailed in the appendix; this slide explains the plain LoRA setting used in the main comparison.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: Problem Formulation; Feature Computation; Keypoint Selection and Action Sampling; VLM/LLM Reward Function
One action and immediate score; categorical policy, Euclidean distance, default temperature and LoRA QKV updates
/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Experimental Setup / Vision backbone
Rank 16 and scaling 8

Learning from feedbackOngoing work

REINFORCE learns from a non-differentiable score

Reward above the recent baseline reinforces the sampled match.

ℒ_t = −(1/B) (r_t − b_t) log π_θ(q_t | s_t)

Advantage: r_t − b_t

Frozen judge returns r_t ∈ [0, 1].
Use b_t before adding the current reward.

b_t+1 = 0.99 b_t + 0.01 r_t

Accumulate B = 10 samples per update.

Gradient through the policy

Differentiate log π through the adapter.
Hold reward and baseline fixed.

Positive advantage: increase probability.
Negative advantage: decrease probability.

Keep valid zero scores; skip invalid responses.

No gradient through the VLM; no target-coordinate annotation in the reward.

The judge does not need to be differentiable. We hold its score and the running baseline fixed, and differentiate the log probability of the sampled action through the LoRA-adapted encoder. Under gradient descent, positive advantage increases the probability of this match; negative advantage decreases it. The baseline begins at zero. We first compute r_t minus the old b_t, then update the exponential moving average with rate 0.01. The one-over-B factor averages accumulated gradients over ten samples before an Adam step; gradients are clipped at norm one. A valid zero score must be retained: once the baseline is positive, it gives negative advantage and discourages that action. Invalid API or parsing responses are skipped. Source keypoints are still given, the visual backbone is pretrained, and annotations are used for evaluation. The missing supervision in this reward is the ground-truth target coordinate.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: VLM/LLM Reward Function; REINFORCE Training, Equation loss
Score parsing and range, pre-update EMA advantage, loss and B=10 accumulation
/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/appendix.tex: Gradient accumulation and clipping
Adam, norm clipping, invalid-response skipping and valid zero-score retention

Learning from feedbackOngoing work

VLM reward improves matching across 18 categories

SPair-71k, all 18 categories · Peak validation PCK@bbox 0.10 (%)

Method	PCK ↑
Frozen DINOv3	54.20
DINOv3 + LoRA	67.83
Geometry-aware + LoRA	71.20

Both trained models use REINFORCE
with Gemini Pro reward.

+13.63PCK points
LoRA adaptation

+17.00PCK points
with geometry-aware features

Both gains are relative to frozen DINOv3.

Training and evaluation span the same 18 categories.

Source points are given. Target-coordinate labels do not enter the reward.

These results train and evaluate across all eighteen SPair categories. Frozen DINOv3 reaches 54.20 PCK. With Gemini Pro reward, REINFORCE adapts its LoRA parameters and reaches 67.83, a gain of 13.63 percentage points. This is the plain-backbone comparison. Adding the geometry-aware stack reaches 71.20, or 17 points above frozen DINOv3. That larger gain combines reinforcement learning with Stable Diffusion features and their aggregation, so it should not be attributed to the RL update alone. Both trained rows use the thirty-six-thousand-epoch regime and report peak validation PCK at a bounding-box threshold of 0.10. They evaluate categories included in training, rather than unseen-category transfer. The table reports no seed uncertainty for these rows. Source points remain given, and the target-coordinate annotation does not enter the reward. The earlier three-seed Kimi cat experiment and optimizer comparison remain in the appendix. The trained matcher needs no VLM call during inference.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Experimental Setup; Table main_results, SPair all column, 36k-epoch Gemini Pro rows
All 18 categories trained and evaluated, peak validation bbox PCK@0.10: frozen DINOv3 54.20; plain LoRA with Gemini Pro reward 67.83 (+13.63); geometry-aware LoRA 71.20 (+17.00, includes SD/fusion architecture). No seed uncertainty supplied for these rows.
/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/multi_vlm_table.tex: Gemini 3.1 Pro Preview, All-categories DINOv3 column
Confirms plain DINOv3 + LoRA all-category score 67.83.

Learning from feedbackOngoing work

Reward precision and geometric feedback

Right part ≠ exact location

Qwen3-VL-32B-Thinking scores plateau within ≈20 px in a 126-keypoint cat probe.

Separate calibration probe with a different judge.

Separate cat-only transfer test
Cat-trained plain LoRA gains only +0.02 mean PCK across six other categories.

The 3D pilots do not establish successful transfer.

What changes under feedback?

Good Match · training
Propose a match → score → update visual weights.

3DHarness · inference
Execute a program → inspect renders or measurements → revise the program.

Feedback must distinguish the geometric errors the task needs to correct.

The learning rule can optimize only distinctions that its reward exposes. In a separate Qwen3-VL-32B-Thinking calibration experiment on 126 cat keypoints, scores plateau for localization errors around twenty pixels or less. This is evidence about that probe, not a universal resolution limit or a measurement of the Gemini judge used in the all-category result. A separate experiment uses a different model trained only on cat: cat-trained plain LoRA gains only 0.02 PCK on average across the six other tested categories. The 3D pilots do not establish a successful extension of this training recipe. This motivates examining what evidence feedback contains. In Good Match, a scalar judgment updates visual model parameters during training. In 3DHarness, an existing agent executes and inspects a 3D program, obtains renders or permitted geometric measurements, and revises the program during inference. The benchmark studies this interaction process; it does not train the agent with REINFORCE. Its reconstruction metrics evaluate the result and are not automatically agent-visible rewards. We now ask whether richer access to geometric evidence produces better program revisions.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Judge calibration; Cross-category and cross-dataset transfer
Qwen32BThinking N126, approximate 20px plateau; plain-LoRA non-cat mean +0.02
/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/appendix.tex: Cross-Modality Pilots
Named-keypoint and nonrigid pilots do not establish successful 3D transfer
/Apps/Overleaf/3DV2027/sec/3_benchmark.tex: Benchmark formulation; Target Access Settings
Agent reconstructs and revises executable programs with tool observations during inference; not a parameter-training result

Executable 3D hypothesesOngoing work

3DHarnessBench: a program as a 3D hypothesis

What does the agent believe
the object is made of?

Parts and their dimensions
Transforms, symmetry and repetition
Geometry and material procedures

Ling Liu · Bingchen Gong
Amal Dev Parakkat · Maks Ovsjanikov

P → E(P) = Ĝ

Blender Python → Executed 3D object

Bird reconstructed by an executable Opus 5 program

Execution makes the inferred structure observable and measurable.

100 targets from 3DCodeBench, one per category. Original source programs are withheld.

Good Match used feedback to change visual model weights. Here an existing agent writes a program that constructs a three-dimensional object. Blender Python exposes choices about parts, dimensions, transforms, repetition and appearance. Execution maps the program to an object whose geometry can be measured and whose renders can be compared with the target. This is reconstruction through a program, not recovery of the original source. We sample one hundred targets from 3DCodeBench, one per category, and withhold their original procedural programs. The benchmark evaluates existing agents without new model training. This is joint work with Ling Liu, Amal Dev Parakkat at Télécom Paris, and Maks Ovsjanikov. My contribution includes experimental guidance, project coordination and evaluation infrastructure. We ask which evidence lets an agent construct and revise a faithful 3D hypothesis.

Sources and evidence

/Apps/Overleaf/3DV2027/sec/3_benchmark.tex: Benchmark Formulation
Executable P and Blender execution E(P); explicit geometry, dimensions, transforms and relations
/Apps/Overleaf/3DV2027/sec/4_experiment.tex: Benchmark data
100 targets, one per category, original source withheld

Executable 3D hypothesesOngoing work

Code expresses repetition, symmetry and named parts

Excerpts from the generated Blender Python

Repeat feathers along a row

for i in range(n):
    s = i / float(n - 1)

Mirror the wing geometry

wf.mirror_y()

Create a named component

wf.object(PFX + "wing_feathers")

Opus 5 reconstruction under Full 3D Interaction

Construction rules become explicit operations the agent can revise.

Opus 5, Full 3D Interaction. Placement calculations and coordinate tables omitted.

This follows the target, readable code and result layout from the presentation prepared for Maks, but uses the current paper's reconstruction and supplied Opus program. The displayed lines are literal excerpts from the wing-feather construction. A loop distributes feathers along each row; omitted calculations interpolate position and direction, with per-feather lengths and a shared profile. The mirror operation produces the other side, and the final call creates the named wing-feathers object. These are explicit construction decisions that can be inspected and changed. The full script also contains extensive measured section and coordinate tables, so this example does not establish that generated programs are always compact or clean symbolic models. Full 3D access supplies measurements, while direct target copying is prohibited. We evaluate the executed reconstruction. Semantic editability and program abstractions remain questions for future evaluation.

Sources and evidence

/Apps/Overleaf/3DV2027/script_examples/bird_opus5_full.py: Wing feather construction, lines 580–591
Literal discontiguous excerpts: repetition loop, mirror_y and named wing_feathers object; dense coordinate tables elsewhere
/Apps/Overleaf/3DV2027/imgs/bird_visualization.pdf: Target and Opus 5 Full 3D row
Source of paired target and executed-reconstruction renders
/home/wenri/pCloudDrive/A_journey_across_representations__3D_data_and_deep_learning_over_the_years.pdf: Page 57
Target / readable code / result layout; early 3DCodeBench evidence not reused as current benchmark evidence

Executable 3D hypothesesOngoing work

Observe, write, execute, inspect and revise

Fixed views: three refinement rounds. Interactive settings: agent-directed tool use.

The representation stays executable across all four settings. Single-view gives one fixed target image, and Multi-view gives four. Both already execute and render the program and provide three rounds of visual refinement. Active Visual lets the agent choose target views, while separate Blender processes and an allowlisted target interface block geometry access. Full 3D adds queries for dimensions, transforms and distances through broader Blender tools. Here the target and reconstruction share a process, and the prohibition on copying is an instruction. Each observation can change the next program, whose execution produces a new object to inspect. This is inference-time program revision by existing agents, not reinforcement learning of model weights. Interface and interaction budgets also differ, so this is not a compute-matched information-only ablation. We measure the executed object's appearance, geometry and topology, together with the interaction required.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/sec/3_benchmark.tex: Target Access Settings
Four target access interfaces
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/evaluation_protocol.tex: protocol table
Fixed-view three rounds versus interactive refinement
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/imgs/3DHarnessBench_pipeline.pdf: pipeline figure
Original benchmark schematic

Executable 3D hypothesesOngoing work

Different evidence produces different executable structures

Single view bird reconstruction — CD 0.017

Four fixed views bird reconstruction — CD 0.014

Active visual bird reconstruction — CD 0.005

Four separate completed runs. Normalized, rotation-aligned Chamfer ↓ for this object.

Opus 5: richer evidence helps recover proportions, wings and legs.

Here is one concrete example, using Opus from the original seven-agent study. The target is at the left, followed by reconstructions produced under the four access conditions. Fixed images support a recognizable bird, but some proportions and structural details remain uncertain. Active inspection and geometric access lead to closer part dimensions and a more faithful rear wing and leg structure in this example. The Chamfer values under the images are per-object values from the original figure, not dataset averages. The displays are separate completed runs under different harnesses, not successive steps in a single trajectory. That distinction matters: we are showing the effect of a changed interaction setting, not inventing a temporal improvement sequence. The full seven-agent comparison is available in the appendix.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/imgs/bird_visualization.pdf: Opus5 column; target crop
Per-objectCD .017/.014/.005/.001 acrosssettings; nottrajectory

Executable 3D hypothesesOngoing work

Useful observations matter more than access alone

100-target comparison

Agent	4 views	Active	Full 3D
Opus 5	.748	.871	.927
Gemini 3.1 Pro	.753	.691	.783

Uni3D 3D–3D similarity ↑
Active view selection helps one agent and hurts another.

Controlled viewport ablation

.3722 → .6148

MiniMax Uni3D after normalizing its viewport screenshots.

10 targets, agent still chooses its views.
Four fixed views on this subset: .6200.

Program revision depends on obtaining evidence that reveals what to change.

Code creates a place to express a correction, but it does not tell the agent which correction is needed. On all one hundred targets, Opus improves from 0.748 to 0.871 Uni3D when it chooses its own views, while Gemini declines from 0.753 to 0.691. Full 3D reaches 0.927 and 0.783 respectively. These are learned shape similarities, not percentages of accurate geometry. A separate ten-target intervention helps explain the observation problem. MiniMax retains its own view choices, but viewport normalization improves its Uni3D from 0.3722 to 0.6148, close to the 0.6200 fixed-view result on that subset. Normalization changes shading, overlays and selection state to make screenshots usable. Reconstruction therefore depends on an observation policy as well as code generation. These comparisons do not isolate a universal best policy, and the aggregate and ablation use different target counts.

Sources and evidence

/Apps/Overleaf/3DV2027/tables/main_result.tex: Opus and Gemini Uni3D rows
Original seven-agent study, 100-target aggregates
/Apps/Overleaf/3DV2027/tables/2ablation_why_gemini.tex: MiniMax rows
10-target .3722 Active, .6148 normalized, .6200 multiview
/Apps/Overleaf/3DV2027/sec/X_suppl.tex: Observation-interface ablation
Normalization preserves agent choices and controls viewport state

Executable 3D hypothesesOngoing work

Measurements constrain the geometry expressed in code

Target evidence	Chamfer ↓	Uni3D ↑
Images	.0150	.8133
+ global	.0103	.8088
+ parts	.0099	.8557
+ global and parts	.0084	.8296

10 targets · Opus 5
Fixed multiview input plus measurements

44%lower normalized
Chamfer distance

Global: center, bounds and dimensions.

Parts: anonymous centers, dimensions and repetition statistics.

Parts alone give the best Uni3D.
Combining cues does not improve every metric.

A program can express measured dimensions and relations as construction parameters.

Does explicit geometry help the agent choose its construction parameters? This controlled ablation keeps Opus and the fixed multiview input and adds target measurements on ten objects. Global evidence includes the bounding box, center and dimensions. Part evidence contains anonymous part centers, dimensions and component or repetition statistics extracted by a measurement program. Combining both reduces normalized Chamfer from 0.0150 to 0.0084, a forty-four percent reduction computed from the reported values. Part measurements alone give the highest Uni3D, and global measurements alone slightly reduce it, so the cues are not uniformly additive. This supports the usefulness of explicit geometric evidence. It does not prove that every dimension is correctly recovered: Chamfer removes translation and absolute scale and aligns over twenty-four rotations. The measurements are available target evidence; final evaluation metrics are not automatically online agent rewards.

Sources and evidence

/Apps/Overleaf/3DV2027/tables/3ablation_measurement.tex: 10-target Opus measurement ablation
CD .0150 to .0084, calculated 44% reduction; part-only Uni3D .8557
/Apps/Overleaf/3DV2027/sec/X_suppl.tex: Experimental Settings: metrics and ablations
Anonymous measurement definitions; normalized aligned Chamfer; distinct evaluation metrics

Executable 3D hypothesesOngoing work

Code provides a state that can be revised and re-executed

First program versus final program

Agent	Initial	Final
Opus 5	.914	.927
Qwen 3.8 Max	.796	.857
MiniMax M3	.566	.679

Full 3D · Uni3D ↑ · 100 targets
Larger repair gains need not yield a better final object.

What the representation enables

Inspect a construction choice.
Change a part, parameter or procedure.
Execute again and assess the result.

Next: learn which observation and code revision will fix an error.

Proposed follow-up: train on verified interaction traces and test semantic edits.

This benchmark evaluates reconstruction and interaction. Program quality and semantic editability remain open.

The first executable hypothesis and the ability to improve it are different aspects of performance. Under Full 3D, Opus starts at 0.914 Uni3D and finishes at 0.927. Qwen moves from 0.796 to 0.857, while MiniMax improves more, from 0.566 to 0.679, but still ends lower. Headroom matters, so the largest gain is not by itself the strongest repair policy. Code gives us a persistent object description with explicit construction decisions that can be revised and re-executed. The benchmark measures objects and interactions. It does not measure clean program abstractions, robustness to semantic edits, or superiority to meshes. Next we can use verified observation-and-revision traces to train smaller models and test whether useful structure survives deliberate edits. This connects to Good Match's learning from feedback, while remaining a proposal: 3DHarnessBench itself does not train agents with RL. The synthesis next connects this executable representation with the earlier geometric and semantic representations.

Sources and evidence

/Apps/Overleaf/3DV2027/tables/self-correction.tex: Full 3D initial and final Uni3D
Opus .914/.927, Qwen .796/.857, MiniMax .566/.679
/Apps/Overleaf/3DV2027/sec/5_conclusion.tex: Future Work
Fine-tuning smaller models from agentic traces is proposed
/Apps/Overleaf/3DV2027/sec/2_relatedwork.tex: Executable Inverse Graphics
Code externalizes reconstruction as evaluation interface; no code-quality/editability benchmark

Research synthesis

Five ingredients, one programme

FourieRF held-out view and depth comparison — Sparse-view geometry

Gaussian surface reconstruction examples from From Blobs to Spokes — Usable surface

ZeroKey semantic point localization examples — Semantic handles

Good Match visual matching and VLM reward pipeline — Learned feedback

3DHarnessBench target and executable reconstruction examples — Executable objects

Research question	Useful structure	Evidence or feedback
Reconstruct with limited evidence	Controlled feature frequencies	Held-out views and geometry
Recover a usable surface	Mesh in the loop; oriented Gaussians	Depth and normal consistency; debiased surface protocols
Locate meaningful elements	Semantic points and part features	Localization and segmentation
Improve a predictor	Adapted visual features	Training-time model judgments
Construct and repair 3D objects	Executable programs	Geometry, appearance and interaction

The evidence comes from separate projects. The programme tests whether these structures can work together.

We can now return to the central question with concrete ingredients. The strip at the top is a visual reminder of the five pieces: frequency control, explicit surfaces, semantic handles, learned feedback and executable objects. These are examples from the presented projects, not a result from one combined system. Frequency control constrains what an underdetermined reconstruction can learn. Explicit surfaces expose geometry to supervision and downstream tools. Semantic points identify the elements an operation should act on. A learned judge provides a training signal whose precision and transfer must be measured. Executable programs expose construction decisions and permit tests. The programme I submitted makes one wager on top of these: that representations and operations can be learned together, from images and video, with checks we can trust.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Research hypothesis
The combination is a future research direction, not a demonstrated end-to-end pipeline.
C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Line 31, project title; research hypothesis
Programme title as submitted. The combination remains a future research direction, not a demonstrated pipeline.
assets/reconstruction/fourierf-overfitting.png: prepared figure
Published FourieRF sparse-view comparison used as a visual anchor.
assets/reconstruction/spokes-teaser.png: prepared figure
Published Spokes surface reconstruction used as a visual anchor.
assets/semantics/zerokey-teaser.png: prepared figure
Published ZeroKey semantic localization used as a visual anchor.
assets/feedback/goodmatch-pipeline.png: prepared figure
Good Match training-time feedback pipeline used as a visual anchor.
assets/feedback/harness-bird.png: prepared figure
3DHarnessBench executable object examples used as a visual anchor.

Future programmeProposed work

Articulation exposes testable constraints

“Can the drawer open without hitting the chair?”

Identify the moving part and its relation to the frame.
Execute the permitted motion.
Check contact, collision and appearance consistency.

SeamlessNeRF before and during a part composition edit — Prior part composition edit; articulation and collision checks remain proposed.

Conceptual scene description

part: drawer
parent: cabinet
joint: prismatic
axis: local x
range: bounded
constraint: no collision

Illustrative schema for the proposed representation

Let me make the programme concrete with one edit. An articulated edit gives us a precise task for evaluating structure. Consider the question of whether a drawer can open without colliding with a chair. A semantic representation can identify the drawer and cabinet, but a successful answer also needs motion constraints, dimensions and the surrounding geometry. The small image recalls SeamlessNeRF's existing part-composition edit. It motivates the move from choosing a part to executing a constrained motion; it is not evidence that the proposed drawer task is solved. An executable representation lets us perform the motion and inspect the resulting state. I would first separate perception from reasoning: supply correct geometry to evaluate the operation, then replace it with reconstructed geometry to measure perception errors.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Axis 2: 3D programs as reasoning tools
Drawer-collision example and separation of perception from reasoning.
assets/reconstruction/opening-composition-wings-poster.png: prepared SeamlessNeRF demonstration poster
Prior part-composition edit shown as motivation; it does not demonstrate articulated motion or collision avoidance.

Future programmeProposed work

Explicit structure with neural appearance

Published components motivate a proposed hybrid.

MILo pipeline with Gaussian reconstruction and mesh supervision — Explicit surface
MILo: mesh in the loop

From Blobs to Spokes oriented Gaussian surface extraction — Oriented geometry
Spokes: Gaussian surface field

FourieRF sparse-view reconstruction examples — Neural appearance
FourieRF: image-based detail

These building blocks were studied separately. Their integration is proposed work.

Test the hybrid against pure-program, mesh and Gaussian representations.

To support that edit, the representation must combine executable scene structure with explicit surfaces and neural appearance. The three images show the research ingredients already established separately: MILo couples Gaussian reconstruction to mesh supervision, Spokes extracts surfaces from oriented Gaussian fields, and FourieRF controls image-based detail from sparse views. The proposed combination is not an existing result. Programs can express dimensions, part relationships and articulation; surfaces support contact and occlusion; neural appearance can carry residual detail. The hybrid must earn its complexity through comparisons with simpler representations. I would begin with static objects and simple articulated structures, measuring reconstruction quality and whether intended edits preserve the rest of the object.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Axis 1: structured, animatable scenes
Proposed program/surface/Gaussian combination and nonprocedural fallback.
assets/reconstruction/milo-pipeline.png: prepared MILo pipeline figure
Published mesh-in-the-loop building block.
assets/reconstruction/spokes-method.png: prepared Spokes method figure
Published oriented Gaussian surface building block.
assets/reconstruction/fourierf-teaser.png: prepared FourieRF teaser
Published neural reconstruction building block; no hybrid result is implied.

Future programmeProposed work

Compact models can learn from verified interaction

Existing evidence shows the two ends of the loop.

3DHarnessBench program revision and executed reconstruction comparison — 3DHarnessBench: inference-time observe, revise, execute

Good Match VLM reward and visual feature adaptation pipeline — Good Match: training-time judge and feature adaptation

Verified trajectoriesPrograms, observations and repairs

→

Compact modelHeld-out edits and task families

The transfer study is proposed. Match information access and compute.

The feedback study suggests a way to transfer expensive model judgments into a compact model that runs independently at inference. The benchmark suggests that interaction quality and information access also matter. The two images make the distinction visible: 3DHarness revises an executable program at inference time, while Good Match changes visual features during training with a VLM score. I would combine those lessons in a controlled learning study. A stronger agent produces construction and repair trajectories. Execution and geometry checks filter them, with learned judgments supplying information that is difficult to express analytically. We then compare learning from the final program, learning from the full interaction, and learning from feedback. The resulting model must transfer to new operations and task families rather than merely reproduce familiar scripts.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Post-training compact models on verified trajectories
Proposed comparison of final scripts, trajectories and reinforcement feedback.
C:/Users/User/Downloads/research/papers/2026_keypoint_matching_v4.pdf: Method and limitations
Existing 2D feedback learning motivates, but does not establish, 3D transfer.
assets/feedback/harness-refinement.png: prepared Harness refinement figure
Published inference-time program revision example.
assets/feedback/goodmatch-pipeline.png: prepared Good Match pipeline figure
Published training-time VLM feedback example; 3D transfer remains proposed.

Future programmeProposed benchmark

4DHarnessBench would evaluate operations over time

Proposed tracks, illustrated by prior examples

MILo mesh used in downstream simulation — **Articulation**
parts, axes and motion

SeamlessNeRF part composition edit — **Constrained edit**
preserve unedited structure

RecolorNeRF edit shown across views — **Motion / video**
temporal appearance and geometry

3DHarnessBench interactive inspection example — **Spatial query**
answer under executable tests

Visual anchors from prior work; none is a 4DHarnessBench result.

Vary representation and feedback while controlling evidence and compute.

4DHarnessBench is a proposed extension of the static benchmark. The visual board uses prior examples to make each proposed track concrete, while the caption prevents a common misreading: these are not temporal benchmark results. Articulation recovery measures parts, joint axes and motion. Constrained editing measures whether a requested change preserves unrelated content. Motion reconstruction adds temporal geometric and appearance consistency. Spatial questions test whether execution supports a correct and calibrated answer. Procedural scenes can provide exact parts and motion for initial tests. I would establish static and articulated subsets first, then compare programs, meshes, Gaussian representations and hybrids under comparable evidence and budgets.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: 4DHarnessBench tasks, tracks and metrics
Proposed benchmark, not an existing release or demonstrated result.
assets/reconstruction/milo-simulation-poster.png: prepared MILo simulation poster
Prior static surface entering simulation; visual motivation only.
assets/reconstruction/opening-composition-poster.png: prepared SeamlessNeRF poster
Prior part-composition edit; not a 4DHarnessBench result.
assets/reconstruction/opening-editing-poster.png: prepared RecolorNeRF poster
Prior multiview appearance edit; not temporal reconstruction evidence.
assets/feedback/harness-inspection.png: prepared Harness inspection figure
Prior spatial inspection example; not a 4D benchmark result.

Future programmeProposed work

Structure may reduce the cost of dynamic content

FourieRF coarse-to-fine training comparison of RGB and depth — Published FourieRF evidence: release detail gradually

FourieRF held-out and training view comparison — A frequency prior can change held-out geometry

Proposed extension

Structured base → residual bands → streamable detail

Count programs, meshes, coefficients, residuals and client computation.

FourieRF motivates frequency control. Compression gains for dynamic content remain a hypothesis to test.

The efficiency direction asks whether explicit structure can reduce storage and computation while preserving the operations we care about. The two FourieRF images show the evidence we actually have: progressive frequency control changes the reconstruction path and can improve held-out geometry. The proposed extension is to let programs, meshes and rigs form a persistent base, with residual geometry and appearance encoded progressively. That extension is not demonstrated by FourieRF. The evaluation must count every component, including programs, assets, coefficients, residuals and decoding or rendering cost. Compression gains remain a hypothesis until the full system is measured.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: FourierGaussian and frugal, streamable 4D media
Proposed spectral coding and full-system rate-distortion evaluation.
assets/reconstruction/fourierf-curriculum.png: prepared FourieRF curriculum figure
Published coarse-to-fine reconstruction evidence.
assets/reconstruction/fourierf-overfitting.png: prepared FourieRF held-out comparison
Published geometry/appearance comparison; no dynamic compression result implied.

Future programmeProposed work

A first project with measurable milestones

Verified trajectories for compact models that reconstruct and edit articulated objects

Milestone	Deliverable	Decision criterion
Establish the task	Static and articulated cases with exact checks	Reliable measurements and reproducible baselines
Learn from interaction	Final-program and trajectory-trained models	Transfer at a matched compute budget
Test useful operations	Held-out edits and spatial queries	Success beyond reconstruction examples

Initial versus final Uni3D improvement for agents in 3DHarnessBench — Prior baseline: initial quality and repair gains
Future work tests transfer to new operations.

I would focus the first project on verified trajectories for compact models that reconstruct and edit static or simply articulated objects. The figure on the right is a prior Harness diagnostic: it separates initial reconstruction quality from repair gain. The first project should preserve that distinction while adding held-out edits and task families. The first milestone is a reproducible task with exact checks and usable baselines. The second is a controlled comparison between final-program supervision and interaction trajectories. The third tests new edits and spatial questions that were absent from training. This ordering gives us informative results even if the larger hybrid representation is difficult.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Year 1, risks and controlled comparisons
Proposed milestone sequence. Staffing and outcomes are not claimed as secured.
C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Line 172, Year 1
Chair-funded Ph.D. recruitment and 4DHarnessBench v1 release are stated year-one plans; outcomes are not claimed as secured.
assets/feedback/harness-refinement-uni3d.png: prepared initial/final Uni3D scatter
Prior 3DHarnessBench baseline used to motivate separate initial quality and repair evaluation; future transfer remains proposed.

Future programmeProposed evaluation

The programme should survive difficult tests

Hypothesis	Test	Evidence against it
Explicit structure improves control	Held-out edits with geometric constraints	No gain over simpler representations
Verified interaction teaches reusable skills	Unseen task and program families	Imitation without transfer
Structure reduces system cost	Count all assets and client computation	Gains disappear under full accounting

A useful negative result should change the representation, supervision or scope.

The programme should be organized around tests that can change the research direction. If explicit structure does not improve constrained edits over simpler representations, we should identify which operations actually require it. If a model reproduces construction traces but fails on new task families, we should treat that as imitation rather than transferable reasoning. If compression gains disappear after counting all assets and client computation, then the representation has not delivered the proposed efficiency benefit. These are productive outcomes because they isolate the assumptions behind the programme. The earlier studies already show why this discipline matters: strict localization thresholds expose limits in semantic points, feedback can fail on geometry-sensitive tasks, and more information does not benefit every agent equally. Those observations should shape the next experiments.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Risks and mitigations; Open science and evaluation
Falsifiable tests derived from the proposed research hypotheses.

Télécom Paris

Complementary expertise for the research programme

Connection	Research opportunity	Relationship
Amal Dev Parakkat IMAGES team	Executable 3D construction and evaluation	Existing 3DHarnessBench collaboration
Kaouther Messaoud	Geometry-grounded motion and video generation	Proposed collaboration
Stephan Alaniz · Jhony Giraldo	Fine-grained grounding and relational structure	Proposed collaboration
Enzo Tartaglione · Xiaoran Jiang	Compact models and Gaussian compression	Proposed collaboration
Jean Le Feuvre	GPAC streaming and end-to-end delivery cost	Proposed collaboration
Kiwon Um · S2A group IDS Department	Differentiable simulation and audio-driven animation	Proposed collaboration
Hi! PARIS	Seminars, summer school and internal calls	Chair attached to the post

Télécom Paris offers complementary expertise for each unresolved part of this programme. My existing collaboration with Amal Dev Parakkat on 3DHarnessBench is the concrete starting point in the IDS Department. In the Multimedia team, the video and animation side of the programme connects most directly to Kaouther Messaoud's work on motion forecasting and physically plausible video generation, which would ground the geometry-conditioned video generation and the motion priors of Axes 1 and 2. Stephan Alaniz and Jhony Giraldo bring fine-grained grounding and relational structure. Enzo Tartaglione and Xiaoran Jiang bring compact models and Gaussian compression, and Jean Le Feuvre's GPAC work lets us measure what these representations cost in a complete streaming system. Beyond the team, Kiwon Um's differentiable simulation and the S2A group's audio work extend the animation axis. These are proposed connections, not agreed projects. Through the Hi! PARIS chair I would contribute to its axes through seminars, the summer school and internal calls, while keeping my geometry collaborations with Maks Ovsjanikov and George Drettakis. The full attribution table and my teaching plan are in the appendix.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Integration in the Multimedia team and IDS Department
Existing Parakkat collaboration and proposed team connections.
C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: Institutional collaborations
Existing 3DHarnessBench collaboration.
C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Lines 119, 133, 164 to 166
K. Um differentiable simulation and K. Messaoud trajectory priors (Axis 1); geometry-grounded video generation with K. Messaoud (Axis 2); team, IDS (S2A) and Hi! PARIS integration as submitted.

Conclusion

Representations and feedback must support the task

Usable geometry exposes structure.
Semantic features identify what to change.
Execution and feedback test whether it worked.

The programme: From Video to Animatable and Efficient 3D Worlds.
The first project: verified trajectories for compact models that reconstruct and edit articulated objects.

The next step is to learn representations and operations together, with checks we can trust.

Bingchen Gong
bingchen.gong@lix.polytechnique.fr · gongbingchen@gmail.com

The common theme is the relationship between representation, operation and feedback. Reliable reconstruction needs constraints matched to the observations. Useful geometry needs explicit structure, and the benchmark that measures it needs scrutiny too. Semantic understanding provides the handles for choosing what to change. Learned judgments can train efficient predictors, but geometric competence has to be measured. Executable models make operations and checks concrete. That is the foundation for the programme I submitted, From Video to Animatable and Efficient 3D Worlds: learn representations and operations together, from images and video, with checks we can trust. At Télécom Paris I would start with verified trajectories for compact models on articulated objects, the first release of 4DHarnessBench, and the collaborations I just described. Thank you to the collaborators credited throughout, and thank you for listening. I am happy to discuss the individual methods, the attribution of each project, my teaching, or the programme.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Overall research programme
Closing synthesis and proposed direction.

FourieRFPublished

A 3D feature field built from lines and planes

T(x,y,z) = ∑_r [v_r^x(x)M_r^yz(y,z) + v_r^y(y)M_r^xz(x,z) + v_r^z(z)M_r^xy(x,y)]

T: feature grid · r: component index. Sum components in all three orientations.

Density and appearance come from compact factors. FourieRF controls the detail stored in each factor.

Before introducing the frequency mask, let me explain what it acts on. TensoRF stores a three-dimensional feature field as a sum of simpler components. In the first component, we sample a line at the x coordinate and a plane at the y and z coordinates, then multiply their values. We repeat this with several learned components and with the other two orientations. This is the vector-matrix, or VM, decomposition. The diagram shows one component and the equation shows the complete field. The density and appearance branches use this compact representation, and the renderer evaluates it along camera rays. At a fixed number of components, the factor storage grows quadratically with grid resolution instead of cubically. The important point for FourieRF is that a line or plane can still contain arbitrarily sharp variations. We therefore control its spatial frequencies during training.

Sources and evidence

/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/4_method.tex: Preliminaries: Vector-Matrix (VM) Decomposition
VM tensor equation and factor interpretation. Diagram redraws one line-plane product. Display uses a shared component index for readability.

FourieRFPublished

FourieRF: bandwidth and optimization

b_t = b₀ + tΔ, Δ = (1 − b₀) / N

Conceptual linear schedule in the main method. N is its duration.

What changes

Filter every 1D and 2D tensor factor before field evaluation.
Use the prescribed mask at iteration t while differentiating the image loss.
Remove positional encodings from feature and view-direction decoder inputs.

What optimizes the field

Training-image supervision and total variation on density and appearance features.
Synthetic scenes: AdamW weight decay 0.2.
Real scenes: density-feature L1 penalty weighted 10⁻⁴, with zero weight decay.

The source has conflicting initial-bandwidth values and an inconsistent synthetic increment. Use the released implementation for exact reproduction.

The curriculum changes the feature representation presented to the renderer. Its main method writes a linear schedule that expands bandwidth from an initial value to full capacity over N iterations. Here b replaces the paper's threshold symbol to avoid confusion with a tensor factor. The factors continue to learn through image supervision, with total variation on density and appearance features. The supplement uses weight decay for synthetic scenes and a density-feature L1 penalty for real scenes. The decoder also omits positional encodings for both features and view directions, reducing another route to sparse-view overfitting. There is a reproducibility detail worth keeping in this technical backup: the current source reports inconsistent initial bandwidths between main text and supplement, and its printed synthetic increment equates one over two thousand with two times ten to the minus three. Those are different values. The main talk explains the algorithm and measured curve without treating these prose values as a verified configuration.

Sources and evidence

/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/4_method.tex: Progressive Inclusion of Coefficients; View Dependence
Conceptual linear schedule and decoder restriction.
/Apps/Overleaf/3DV2025/author-kit-3DV2025-v1/sec/X_suppl.tex: B. Details on Implementation and settings
AdamW, TV, synthetic weight decay and real-scene L1 settings. Conflicting bandwidth and increment descriptions remain unresolved.

FourieRFPublished

The rate of detail release matters

Blender, six input views. The frequency-increment axis is logarithmic.

Rapid release weakens regularization. Very slow release can underfit the fixed budget.

This ablation varies the frequency increment while keeping the experiment budget fixed. The curve is not monotonic. Releasing the full bandwidth too rapidly weakens the regularization we wanted, while an overly restrictive schedule can leave the representation too coarse at the end. The best region in this displayed experiment lies near an increment of five times ten to the minus four. I would treat that as evidence about the balance rather than as a universal parameter for every scene. The archived draft contains inconsistent notation for the exact initialization and increment, so the general mechanism and measured curve are firmer claims than a single recipe copied from its prose. The substantive lesson is that representation capacity is something we can schedule deliberately, guided by the amount and structure of the available evidence.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/images/delta_vs_psnr_log.pdf: page 1; full plot, no values digitized
Frequency-increment sensitivity
C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/sec/X_suppl.tex: Implementation details; archived notation differs from main text
Exact recipe should follow released implementation rather than merge inconsistent draft values

FourieRFPublished

FourieRF: protocol and scope

Evaluation	Input views	FourieRF PSNR
Blender: 8 scenes	4 / 6	21.728 / 23.927 dB
LLFF: 8 scenes	3 / 6 / 9	19.303 / 23.595 / 25.011 dB

Blender follows ZeroRF’s evaluation setup; LLFF follows RegNeRF’s.
10,000 iterations; VM tensor factors in the main comparison.
No claim of recovering truly unobserved surfaces.

These are the dataset and view-count settings behind the main claims. The synthetic experiments use eight Blender scenes with four or six training views. The forward-facing real-scene experiments use eight LLFF scenes with three, six or nine views. The method uses the vector-matrix representation in the main comparison and trains for ten thousand iterations. The evaluation protocols follow the cited ZeroRF and RegNeRF settings, respectively. The averages should remain attached to those settings. FourieRF does not lead every metric: ZeroRF has higher synthetic PSNR, and FreeNeRF has higher LLFF PSNR at each listed view count. The argument is that a simple feature-frequency curriculum provides a competitive and efficient prior, rather than a complete solution to the ill-posedness of sparse-view reconstruction.

Sources and evidence

https://arxiv.org/html/2502.01405v1: Main experimental tables and protocol
Dataset counts,view counts,PSNR values and protocol attribution

FourieRFPublished

FourieRF: interpreting the timing ratio

Method	Relative training time ↓	4-view Blender PSNR ↑
TensoRF	1.000	—
FourieRF-VM	0.930	21.728 dB
ZeroRF	5.181	21.940 dB
FreeNeRF	35.710	—

5.57× training speedup over ZeroRF in this setup, with 0.212 dB lower PSNR.

10,000-iteration Blender timing comparison; TensoRF normalized to 1. RTX 4090.

The speed statement is an explicit calculation from the normalized timing table. ZeroRF is listed at five point one eight one and FourieRF at zero point nine three, so their ratio is approximately five point five seven. A ratio of training times should not be translated into rendering frame rate, sample efficiency or a universal hardware-independent advantage. The associated four-view image-quality comparison remains visible in the main slide precisely because the methods do not reach identical PSNR. The archive also contains conflicting schedule notation and supplementary entries, which is why this presentation uses the main reported comparison and does not invent one reconciled training recipe. Reproducing the experiment should use a pinned released implementation and its configuration rather than infer parameters from inconsistent draft expressions.

Sources and evidence

https://arxiv.org/html/2502.01405v1: Main Blender metrics table and normalized training-time table
FourieRF .93; ZeroRF 5.181; FreeNeRF35.71; four-view PSNR21.728/21.94
C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/sec/X_suppl.tex: Implementation details
RTX4090 and10k iterations
https://arxiv.org/html/2502.01405v1: Computed:5.181 / 0.93 = 5.570967741935484;21.940 - 21.728 =0.212 dB
5.57× is a runtime ratio;0.212dB is the associated 4-view PSNR difference

FourieRFPublished

FourieRF: real-scene comparison

LLFF example: ground truth, FourieRF, FreeNeRF, ZeroRF. The reference has no ground-truth depth.

Competitive reconstruction does not imply the best PSNR or verified depth.

This LLFF comparison shows rendered color and predicted depth. Ground-truth depth is unavailable, so the depth panels illustrate the inferred structure without measuring its accuracy. Across the reported LLFF table, FreeNeRF has higher PSNR at each input-view count. FourieRF should therefore be understood as an efficient regularization strategy with metric-dependent tradeoffs. The frequency prior cannot establish the correctness of completely unobserved surfaces. The main talk connects that limitation to the separate question of explicit surface reconstruction.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf.old/3DV2025/author-kit-3DV2025-v1/images/llff_comparison.pdf: page1; complete figure
Depth reference unavailable; selected qualitative comparison
https://arxiv.org/html/2502.01405v1: Main LLFF table and conclusion
Metric-dependent tradeoffs and unobserved geometry limitation

MILoPublished

MILo: losses and geometric evaluation

Ablation	Tanks & Temples F1 ↑
Baseline	0.41
+ Mesh depth	0.46
+ Mesh normals	0.44
+ Anti-erosion	0.44
+ Interior regularization	0.47
GOF integration	0.49

Normal supervision improves selected surfaces visually even when rounded F1 decreases.

The ablation is a useful reminder that a geometric metric and visual surface quality can rank variants differently. Mesh depth gives a clear increase over this baseline. Adding normal consistency reduces the reported aggregate F1 from zero point four six to zero point four four, even though selected surfaces become smoother and visually cleaner. The anti-erosion result is unchanged at the table’s two-decimal precision. Interior regularization brings the full configuration to zero point four seven, while the GOF integration reaches zero point four nine. I therefore describe the loss roles and show their numerical behavior, rather than claiming that every added component monotonically improves every metric. These results also belong to the MILo paper’s evaluation protocol and should not be merged with later Gaussian Wrapping evaluations that use different surface-sampling procedures.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/tables/ablation_components.tex: Complete ablation table
Exact displayed F1 values
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/figures/ablation_depth_only_vs_mesh_only.tex: Caption and cited Barn examples
Normal supervision visual motivation

MILoPublished

MILo: optimization and evaluation boundaries

Iterations	Training stage
0–3,000	Photometric fitting and aggressive densification
3,000–8,000	Normal regularization; Gaussian refinement
8,000–18,000	Mesh supervision and selected pivots

Delaunay update: 500 iterations. Occupancy-label update: 200 iterations.

Mesh-rendering quality complements direct geometry metrics; it does not certify unseen surfaces.

The full optimization schedule contains eighteen thousand iterations. The first three thousand emphasize photometric fitting and densification. Normal regularization follows, and mesh supervision begins at eight thousand iterations for the final ten thousand. Connectivity updates every five hundred iterations, while visibility-based occupancy labels refresh every two hundred. The paper evaluates surfaces with direct metrics where reference geometry is available and with mesh-based novel-view synthesis for larger scenes. In the latter evaluation, a learned TensoRF color field assigns appearance independently of vertex density before the mesh is rendered. This helps separate surface complexity from color resolution, but the resulting PSNR, SSIM and LPIPS are still image-consistency measures. They depend on view coverage and appearance fitting and should not be interpreted as proof of correct hidden geometry or global watertightness.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/X_supplemental.tex: Optimization schedule and update intervals
18k total;3k/8k boundaries;500/200 updates
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/4_experiments.tex: Mesh-based rendering metric
TensoRF color field and dense-view evaluation limitations

MILoPublished

MILo: preserving the surface and cleaning its interior

Preserve selected Gaussian centers

ℒ_erosion = ∑_{g∈G_Del} max(0, f_μg)

G_Del: Gaussians selected for triangulation
f_μg: signed value at a Gaussian center

Penalize positive center values. Keep a negative sample inside thin structures.

Suppress unsupported interior surfaces

ℒ_interior = ∑_p o_p H(σ(−f_p), o_p)

o_p = 1 when pivot p lies behind all visible mesh depths
H: binary cross-entropy · σ: sigmoid

Supervise only interior-labeled pivots. Refresh labels every 200 iterations.

ℒ = ℒ_vol + λ_MDℒ_MD + λ_MNℒ_MN + λ_erosionℒ_erosion + λ_interiorℒ_interior

λ_MD = λ_MN = 0.05 · λ_erosion = λ_interior = 0.005 in the reported implementation.

The two regularizers address failure modes that visible depth and normals cannot resolve alone. A thin part can disappear when all nearby signed values become positive. After disappearance, rasterization supplies little gradient to recover it. The anti-erosion term therefore penalizes positive values at the centers of selected Gaussians. It leaves the corner values free, so they can still define a boundary around the center. The interior term handles hidden cavities and internal fragments. We render the current mesh from the training cameras and label a pivot as interior when it lies behind the visible surface in every camera that contains it. Only those interior-labeled pivots receive the cross-entropy penalty. Their signed values are encouraged to be negative, reducing extra zero crossings inside the object. These labels refresh periodically and depend on the observed scene. They improve the reported reconstructions without guaranteeing the geometry of every unseen region. The equation below collects the terms used during the mesh-supervised stage.

Sources and evidence

/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Regularization: Erosion and Interior artifacts; Eqs. anti_erosion_loss, interior_loss and final_loss
Center-only anti-erosion loss, visibility-derived interior labels and occupancy-weighted cross-entropy.
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/X_supplemental.tex: Implementation Details: Optimization
Reported consistency and regularization weights.

MILoPublished

A zero crossing gives a differentiable surface vertex

v = (fₐpᵦ − fᵦpₐ) / (fₐ − fᵦ)

Moving either pivot or changing its signed value moves the extracted surface.

Mesh losses→Surface vertices v→

Signed values f
Gaussian geometry μ, R, s

Arrows show the gradient path. Discrete Delaunay connectivity updates every 500 iterations.

Consider one edge of a tetrahedron. Its endpoints are pivots and their signed values have opposite signs, so the surface crosses this edge. Linear interpolation locates the zero crossing. Changing an endpoint position moves the vertex, and changing either signed value slides the crossing along the edge. Marching tetrahedra connects the crossings into triangles. When we rasterize those triangles, the mesh losses can update the signed values and the pivot positions. Because the positions depend on Gaussian means, rotations and scales, the loss also reaches those geometric parameters. That is the gradient path shown below. The continuous calculation works within a fixed connectivity. Delaunay tetrahedralization is a separate discrete operation, refreshed every five hundred iterations. Mesh extraction and rendering still happen at every iteration. This is how surface supervision influences the volumetric representation without requiring differentiation through the triangulation algorithm. Rendering importance selects which Gaussians supply pivots. The base variant removes unselected Gaussians; the dense variant keeps them for rendering while using only the selected subset for the mesh.

Sources and evidence

/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Differentiable Marching Tetrahedra; Eq. marching_tetrahedra
Exact interpolation equation and two gradient paths through signed scalars and pivot coordinates. Editable edge diagram is conceptual.
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/X_supplemental.tex: Implementation Details: Delaunay triangulation
Connectivity is non-differentiable and recomputed every 500 iterations; mesh extraction is per iteration.
/Apps/Overleaf/[SIGGRAPH Asia'25] MILo: Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/3_method.tex: Sampling Delaunay Vertices from Gaussians; Computing Signed Distance Values
Nine-pivot formula, rendering-importance sampling and independently learned signed scalars. Base and dense selection behavior.

MILoPublished

Inspecting recovered thin structures

Original supplementary video: exported bicycle and bench mesh in Blender.

Changing the viewpoint exposes details that a single attractive rendering can conceal.

Here we inspect the exported geometry directly in Blender. The bicycle frame, wheels and repeated bench slats are useful stress cases because thin structures can disappear or become entangled during extraction. Moving the viewpoint helps distinguish a surface from an appearance effect in one image. These are selected qualitative results, so the demonstration should complement the benchmark rather than replace it. The reconstruction is also not uniformly clean everywhere: the surrounding ground and background vegetation contain irregular geometry. That limitation is visible and worth preserving. The useful result is that detailed meshes can be recovered for full captured scenes with substantially reduced complexity compared with several dense extraction baselines. The main section separates that compactness claim from training cost and geometric accuracy.

Sources and evidence

C:\Linux\CNRSApplication\presentation-archive\supporting-assets\2025\2025_MILo_Presentation-Video.mp4: 02:21–02:40; poster02:31; silent excerpt
Direct Blender inspection of bicycle/bench and surrounding geometry
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/4_experiments.tex: Qualitative results
Selected detail preservation evidence

From Blobs to SpokesPublished

Spokes: oriented fields and surface extraction

Original paper diagram: oriented Gaussians supply normal and vacancy fields, with pivot-based Marching Tetrahedra and optional Primal Adaptive Meshing

Two pivots per Gaussian
Use the center and an outward-displaced point for tetrahedral surface extraction.

Primal Adaptive Meshing (PAM)
Resample and project vertices to control mesh resolution independently of Gaussian count.

Tanks & Temples · mean F1 ↑ · six scenes

Method	Uniform sampling	Virtual scan
MILo	0.34	0.42
Spokes (2 pivots)	0.48	0.53

Both methods evaluated with the Spokes paper’s protocols.

Vacancy and normal fields are estimated in practice. The table reports the two-pivot variant; PAM is optional.

The learned orientation supplies an outward direction for each Gaussian. Normal consistency and densification encourage the Gaussians to form a complete shell. From this representation we obtain a normal-field estimate and a vacancy estimate, which describes whether a point is outside the reconstructed object. The theoretical construction assumes a consistently wrapped opaque surface; practical scenes only approximate that model.

The initial extractor uses two pivots per Gaussian: its center and a point displaced along the outward normal. Estimated vacancy distinguishes their sides of the surface, and Marching Tetrahedra extracts the boundary. Primal Adaptive Meshing is an optional second step. It samples and projects vertices, filters poor candidates, and reconstructs connectivity so the mesh resolution can be chosen independently of the number of Gaussians.

The table compares MILo with the two-pivot Gaussian Wrapping variant on six Tanks and Temples scenes. The mean F1 increases from 0.34 to 0.48 with uniform surface sampling, and from 0.42 to 0.53 with virtual scanning. Both rows use this paper's evaluation protocols. These differ from the original MILo evaluation, so the absolute scores should be interpreted within this table. The improvement does not imply smaller meshes than MILo in every setting or correct recovery of unobserved geometry.

Sources and evidence

https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Figure 3, PDF page 7; Sections 4 and 5
Oriented fields and two-pivot Marching Tetrahedra; optional Primal Adaptive Meshing. Practical fields are estimates under model assumptions.
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Table 1, PDF page 13, mean rows: uniform sampling and virtual scanning; MILo and Ours (2p)
Uniform mean F1 0.34/0.48 and virtual-scan mean F1 0.42/0.53 on six Tanks and Temples scenes. Same-paper comparisons, not original MILo legacy scores or PAM results.
https://diego1401.github.io/BlobsToSpokesWebsite/: Author list and ECCV 2026 label
Publication metadata.

MILoPublished

MILo: fewer vertices and smaller meshes

Method	Mesh vertices ↓	Mesh size ↓	Training time ↓
GOF	16.49 M	600.0 MB	93 min
RaDe-GS	14.75 M	592.0 MB	42 min
MILo base	4.36 M	179.6 MB	50 min
MILo dense	6.89 M	276.1 MB	110 min

Tanks & Temples resource table; RTX 4090. Base/dense use the RaDe-GS backbone. Legacy vertex-based F1: MILo 0.47 (base) and 0.49 (dense); GOF 0.46, RaDe-GS 0.40.

Mesh size and geometric accuracy are separate claims. Compare accuracy with a common surface-sampling protocol.

MILo’s resource result concerns the exported mesh. Its base variant uses 4.36 million vertices and 179.6 megabytes, versus roughly 15 to 16 million vertices and 600 megabytes for RaDe-GS and GOF. Training takes 50 minutes, against 42 and 93 respectively in this setup. The denser MILo variant trades more training and storage for accuracy. The original vertex-based metric gave MILo 0.47 or 0.49 mean F1. Spokes later showed that this metric rewards dense tessellations, so those values should not be read as a fair comparison across extraction strategies. The resource advantage remains a separate observation. The joint results slide in the main section uses the later Spokes paper’s reported mesh counts and surface evaluations. Its MILo row is not identified as the same variant as either resource row here.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/tables/resource_requirements.tex: All rows; slide selectsGOF,RaDe-GS,base,dense
Vertices,mesh sizes and training times copied without recomputation
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/tables/surface_metric_tandt.tex: Mean row, per-scene columns and Time row
F1 .47 (RaDe-GS base, 50 min) / .49 (GOF base 150 min, RaDe-GS dense 110 min); GOF .46, RaDe-GS .40.
C:/Users/User/Dropbox/Apps/Overleaf/[SIGGRAPH Asia'25] MILo_ Mesh-In-the-Loop Gaussian Splatting for Detailed and Efficient Surface Reconstruction/sections/X_supplemental.tex: Hardware
RTX4090 24GB
https://diego1401.github.io/BlobsToSpokesWebsite/static/pdfs/gaussian_wrapping.pdf: Section 5.2, Legacy Evaluation; Table 4 caption
The standard vertex-based T&T protocol biases scores toward denser tessellations; extracting the follow-up meshes with 9 pivots instead of 2 inflates the score.

ZeroKeyPublished

ZeroKey: localization thresholds and comparison scope

Method	0.01	0.05	0.10
B2-3D · few-shot, 62 views	20.29	57.72	70.57
B2-3D · few-shot, 26 views	14.39	44.11	56.90
GPT-4o localizer	0.48	6.04	20.73
CLIP-DINOiser	1.41	9.80	25.56
StablePoints	5.80	19.91	38.22
Binary back-projection	5.84	29.81	54.64
ZeroKey · 26 views	13.16	56.60	79.43

KeypointNet: airplane, chair and table. IoU (%) at geodesic thresholds.
StablePoints receives the ground-truth keypoint count; benchmark text descriptions are manually assigned.

This table preserves the broader comparison and its protocol boundaries. The official ICCV supplement reports both B2-3D view counts and explicitly distinguishes the binary back-projection ablation from the full soft-voting method. At a 0.10 tolerance, replacing soft weights with constant weights reduces ZeroKey from 79.43 to 54.64 IoU. This measures that ablation in the reported setup. It is separate from the saliency experiment, which uses a point-description-point loop and absolute 3D recovery error. StablePoints receives the number of ground-truth keypoints and is not driven by the same free-form point query. B2-3D uses few-shot reference information. The benchmark evaluates only airplane, chair and table, with human-written descriptions of semantic keypoint IDs.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Sections 4–5 and Table 3
Exact published threshold results, view counts, binary back-projection row and human-written query protocol.
https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 6.2, StablePoint
Ground-truth keypoint count supplied to StablePoints.

ZeroKeyPublished

ZeroKey: distinguish dense agreement from isolated votes

Ordinary distance treats an isolated pair like a well-supported cluster.

d_m(a,b) = max{core_k(a), core_k(b), ‖a − b‖}

core_k(a)
Distance from candidate a to its kth nearest neighbor.

Isolated candidates have large core distances, even if two happen to be close.

HDBSCAN on pooled 3D candidates

Use the Gaussian soft-voting weights.
Find stable dense clusters across views.
Fix minPts k = 10 in the experiments.

Spatial density separates repeated semantic instances from inconsistent predictions.

The clustering metric is mutual reachability. It takes the largest of the Euclidean separation and the two local core distances. The core distance is the distance to the kth nearest neighbor, with k fixed to ten in the reported experiments. This makes an isolated candidate less attractive to a dense cluster, even when its raw Euclidean separation from another isolated point is small. HDBSCAN builds a density hierarchy using these distances and selects stable clusters. The ZeroKey method additionally incorporates the Gaussian voting weights derived from the image neighborhoods. These weights measure local supporting evidence; they are not a calibrated probability from Molmo. Pooling across views is useful both for rejecting outliers and for distinguishing several valid instances of a semantic name, such as table corners. The exact clustering mechanism is independent of the camera-depth lifting operation given in the depth-lifting appendix.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 4.3, equations 3–4
Published Gaussian support weights and mutual-reachability distance with minPts=10.
/Apps/Overleaf/[ICCV'25] Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models/sec/4_method.tex: Filtering and Refinement, equation equ:mutual_reachability
Mutual-reachability distance, core-distance definition, Gaussian weights and minPts=10
presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 13
Earlier talk explanation of soft voting and weighted HDBSCAN

ZeroKeyPublished

A named point is a localization problem

“Point to the corners of the chair back.”

Input: a known mesh and a text query
Output: named locations on its surface
No task-specific 3D keypoint training

Semantic recognition still needs precise spatial grounding.

The same chair query in multiple rendered views

Recognizing that an object is a chair is easier than locating the corners of its back. Even identifying the whole back as a region does not tell us which surface points should serve as handles. We give the system a mesh that can be rendered and a language query, and ask for coordinates on that mesh. The example compares the localizations produced with different image-language models. It motivates the use of a model trained to point at image locations. The experiment is specific to the tested models and prompts, so it does not establish that every model without pointing data must fail. Also, zero-shot here means no task-specific 3D keypoint training. The foundation models already contain learned visual and language knowledge. We separate naming, pointing, and geometric aggregation to make that transfer possible.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/supplemental/Gong_ZeroKey_Point-Level_Reasoning_ICCV_2025_supplemental.pdf: Figure 3 and Section 4
Model comparison and manually written benchmark descriptions.
C:/Linux/CNRSApplication/presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 14; crop PDF points [231, 43, 438, 141]
Chair-corner localizations from Molmo and GPT-4o in the illustrated views
/Apps/Overleaf/[ICCV'25] Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models/sec/6_experiments.tex: GPT-4o baseline and setup
Tested localizer comparison and no 3D keypoint training

ZeroKeyPublished

Use known depth to lift an uncertain image point

Image-language predictions for the same chair-corner query — A small image error can cross a silhouette or a depth discontinuity.

Cache the mesh depth while rendering.
Each valid pixel maps to a visible 3D surface point.

x_j(u) = R_j^T[D_j(u) K_j⁻¹ũ − t_j]

u: pixel; ũ: homogeneous pixel
D: camera-depth map; K, R, t: camera parameters

Lift an h × h neighborhood, reject background pixels, then average its valid 3D points.

The mesh and camera are known, so this is a grounding step rather than a triangulation problem. Rendering already gives us the depth of the visible surface at each pixel. In this equation, the camera convention is x_camera = R x_world + t, and D is camera-z depth. Multiplying the homogeneous image pixel by the inverse intrinsic matrix gives a camera ray; depth selects the visible surface point, and the inverse rigid transform returns world coordinates. This is an explicit pinhole-camera expansion of the paper's inverse-projection notation. Near an object silhouette, a slight pointing error may land on the background or a different surface. ZeroKey therefore lifts an h by h neighborhood around each Molmo prediction, discards pixels that miss the mesh, and averages the remaining 3D points into a candidate. The neighborhood average stabilizes a detection, but it does not guarantee that the mean lies exactly on the original surface. We still need multi-view agreement to reject inconsistent candidates.

Sources and evidence

https://openaccess.thecvf.com/content/ICCV2025/papers/Gong_ZeroKey_Point-Level_Reasoning_and_Zero-Shot_3D_Keypoint_Detection_from_Large_ICCV_2025_paper.pdf: Section 4.3
Cached depth and neighborhood lifting; camera expansion uses x_camera = R x_world + t.
/Apps/Overleaf/[ICCV'25] Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models/sec/4_method.tex: Zero-Shot 3D Keypoint Detection / Back-Projection of 2D Keypoints; Soft Voting Weights
Cached raster depth, h × h neighborhood, background rejection and mean lifted candidate; displayed camera equation expands the stated pinhole model
presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: pages 11–13
Style reference: pipeline followed by explicit lifting and aggregation mechanisms

ZeroKeyPublished

Sparse semantic handles leave gaps

Selected named keypoints across different shapes

What represents the surface between the points?

Repeated model calls make dense querying costly.
Ambiguity and symmetry remain.
Editing and segmentation need local features across the shape.

The next step is to learn language-aligned features directly in 3D.

ZeroKey gives us a flexible interface to selected, nameable locations. That is valuable for a sparse set of handles, but many applications need a representation everywhere: a region to select, a part boundary to segment, or a local feature to compare across shapes. Repeating the full rendering and language-model pipeline for many queries would be expensive. Symmetric duplicates and ambiguous descriptions also remain difficult. These limitations motivate a different computational choice. Rather than call a large image-language model every time we need a local answer, we can transfer 2D foundation-model knowledge into a 3D encoder during training. The encoder should preserve geometric locality while making its features comparable with text. This is the transition to PatchAlign3D. It shares the goal of localized 3D semantics with ZeroKey, but moves the cost and the supervision into an offline learning stage.

Sources and evidence

C:/Linux/CNRSApplication/presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 17; crop PDF points [34, 45, 255, 221]
Selected keypoint detections on salient shape features
C:/Linux/CNRSApplication/presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 21
ZeroKey limits and transition to dense features

PatchAlign3DPublished

PatchAlign3D: where the two training signals come from

Pseudo-part annotations

32,052 Objaverse shapes: 28,827 train / 3,225 validation.
10 views per shape → SAM masks → Gemini 1.5 part names.
Back-project to 3D; retain overlapping labels.
Over 2 million annotations, 761 categories.

Training and representation

Stage 1: frozen DINOv2 visual targets.
Stage 2: OpenCLIP ViT-bigG-14 text embeddings.
100 epochs per stage; batch size 32.
Learn τ and b, initialized at 0.1 and −10.

Random rotations, translations, scaling and point jitter support generalization.

The pseudo-part data uses the Find3D data-engine procedure, applied to 32,052 curated Objaverse shapes. Each shape is rendered from ten views. SAM produces multi-scale masks, and Gemini 1.5 assigns a short part name to each masked region. Back-projecting those masks creates per-point labels, including overlaps when a point receives different valid descriptions from different views. The dataset contains more than two million annotations over 761 categories, split into 28,827 training and 3,225 validation shapes. These are automatic labels with noise, not target-benchmark manual annotations. The visual teacher is DINOv2; the text encoder is OpenCLIP ViT-bigG-14. SigLIP names the objective family, not the selected text encoder. Both training stages run for one hundred epochs with batch size 32. Temperature and bias are learned from initial values 0.1 and minus ten. The point cloud is augmented with random rotations, translations, scaling and jitter. Thus zero-shot refers to evaluation on downstream part tasks without category-specific fine-tuning.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Sections 3.2–3.3 and 4.1
32,052 curated shapes, SAM/Gemini annotations, two stages and the OpenCLIP text encoder.
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/03_method.tex: Training data; Stage 2: 3D–text patch contrastive learning
Find3D-style pseudo-annotation pipeline, counts, fractional multi-label supervision, learned temperature/bias and augmentations
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Implementation settings
Teacher/text encoder, 100 epochs per stage, batch size 32
presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 28
Earlier talk training-data explanation

PatchAlign3DPublished

PatchAlign3D: build a stable visual target before training

Rendered feature mapsBicubic upsampling to image resolution

→

Visible surface pointsAverage observations across views

→

Cached patch targetsAverage features within each patch

d(x) = 1/|V(x)| ∑_{r ∈ V(x)} F_r(π_r(x))

d_i = 1/k ∑_{x ∈ P_i} d(x)

V(x): views observing x; F_r: DINOv2 feature map; π_r: camera projection; k: points per patch.

L_2D = 1/G ∑_i [1 − cos(h_2D(z_i), d_i)]

z_i: patch token; h_2D: visual projection; G: number of patches.

Match each online patch center to the nearest cached center before applying the cosine loss.

The first stage uses a fixed teacher target that is prepared before encoder training. Dense DINOv2 image features are upsampled to the rendering resolution, then mapped to the visible surface using known geometry. For a point visible in multiple images, d of x is the arithmetic mean of the corresponding view features. Unobserved points receive features through nearest-neighbor interpolation from observed points. The second equation averages these point features over a patch. We cache the patch centers, memberships and feature targets for each shape. Online patch sampling need not reproduce the cached centers exactly, so each online center is matched to the nearest cached center and uses its target feature. The displayed d_i in the main cosine loss therefore represents the matched cached target. This detail prevents supervision from being assigned by an arbitrary token index when sampling changes. The two averages have different purposes: views combine observations of one surface point, while patches form the local target that the 3D encoder predicts. The final cosine loss aligns each projected patch token with its matched target. G is the number of patches, z_i is the encoder token, and h_2D maps it to the visual teacher space.

Sources and evidence

/Apps/Overleaf/CVPR2026_souhail (Arxiv)/03_method.tex: 2D Dense Feature Extraction; Stage 1: 2D-to-3D feature distillation
Visible-view averaging, unobserved-point interpolation, patch averaging, cached-center matching and cosine loss

PatchAlign3DPublished

PatchAlign3D: benchmark scope and exceptions

Benchmark	PatchAlign3D mIoU	Comparator mIoU	Qualification
ShapeNetPart	56.9	COPS 25.6	Laptop: 61.1 vs COPS 63.3
FAUST	67.8	Find3D 63.2	Torso: 49.7 vs Find3D 52.0
PartNetE	41.4	COPS 27.0	PartSLIP reports 36.4 cIoU
ScanObjectNN	22.7	Find3D 18.8	Noise and clutter remain difficult
Objaverse unseen	35.61	Find3D 34.6	Author-defined 14-category split

Different datasets and metric definitions support different conclusions.

This table keeps the broader evidence available without making the main talk a benchmark catalogue. The gains are substantial on ShapeNetPart, but laptop is an exception against COPS. FAUST improves overall while the torso part remains below Find3D. PartNetE includes incompletely annotated points, and the model assigns a body label to otherwise unlabeled points in the reported protocol. PartSLIP's listed aggregate is category IoU, so it should not be silently compared with the mIoU column here. ScanObjectNN demonstrates the remaining difficulty of real scanned objects. The Objaverse-General experiment uses one hundred validation shapes and an author-defined split containing fourteen unseen categories because the original split was unavailable. Its unseen-category gain over Find3D is modest. These qualifications describe the evaluation scope and help separate an improved representation from a claim of universal part understanding.

Sources and evidence

/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Tables tab:shapenetpart, tab:faust, tab:partnete_scanobjectnn, tab:objaverse_general
Exact results, metric qualifications and category-specific exceptions

PatchAlign3DPublished

PatchAlign3D: selected segmentation examples

Original ShapeNetPart examples with all displayed baselines

Spatial coherence
Many predicted regions follow recognizable parts.

Boundary limitations
The car remains a useful counterexample.

Qualitative examples illustrate behavior. Aggregate tables determine the measured gain.

The examples compare ground truth, COPS, Find3D and PatchAlign3D on an airplane, car, chair, lamp and cap. The proposed features often produce spatially coherent parts, particularly on clearly separated structural components. The car is a useful reminder that a coherent region can still be incorrectly bounded. This is consistent with a fixed-resolution patch representation and nearest-center label propagation. The examples are selected illustrations and do not provide per-object error bars or establish that every instance improves. They should be read alongside the quantitative benchmark and its category-specific exceptions. The training signal itself comes from noisy pseudo-labels, so visual coherence is not equivalent to exact ground-truth fidelity. This distinction also motivates later work on feedback that evaluates whether an apparently plausible local prediction actually corresponds to the intended target.

Sources and evidence

C:/Linux/CNRSApplication/presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 30; crop PDF points [35, 39, 279, 224]
Original GT, COPS, Find3D and PatchAlign3D selected ShapeNetPart comparisons
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Qualitative results and ShapeNetPart table
Selected comparisons and category-level exceptions

PatchAlign3DPublished

Stronger part segmentation at feed-forward speed

ShapeNetPart, point-cloud methods. Reported IoU (%) and inference time.

Method	mIoU	cIoU	Seconds / shape
COPS	25.6	32.2	1.38
Find3D	23.3	23.9	0.4
PatchAlign3D	56.9	53.1	0.4

+31.3 percentage points mIoU over COPS. The reported runtime matches Find3D.

mIoU averages instances. cIoU averages categories. Timings refer to the paper’s evaluation.

On ShapeNetPart, the learned patch representation improves segmentation substantially over the listed point-cloud baselines. The main metric is instance-averaged mIoU: 56.9 for PatchAlign3D, 25.6 for COPS, and 23.3 for Find3D. The improvement over COPS is 31.3 percentage points, rather than a relative percentage. Category-averaged IoU is reported separately. The runtime is 0.4 seconds per shape in the paper, equal to Find3D, while COPS takes 1.38 seconds. The useful result is improved representation quality while retaining feed-forward efficiency. These timings should stay attached to the reported evaluation protocol rather than become a universal real-time guarantee. Mesh-based methods receive different inputs and are excluded from this compact comparison. Even within the point-cloud comparison, one category, laptop, remains below COPS. The next question is how far these gains transfer beyond clean synthetic objects.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Sections 3.2–3.3; Tables 1–5
Published architecture, inference and benchmark details retained for discussion.
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Tables tab:shapenetpart and tab:inference_speed
Exact main point-cloud benchmark and runtime values

PatchAlign3DPublished

Transfer improves, but domain shift remains

FAUST ground truth — A FAUST pose, with visible boundary errors

PatchAlign3D prediction — A FAUST pose, with visible boundary errors

FAUST	mIoU (%)
COPS	30.4
Find3D	63.2
PatchAlign3D	67.8

Real scanned objects are harder.
ScanObjectNN: 22.7 mIoU versus Find3D’s 18.8.

Improvement does not remove the robustness gap.

The method also transfers to non-rigid human shapes. On FAUST, PatchAlign3D reaches 67.8 mIoU, compared with 63.2 for Find3D and 30.4 for COPS. Find3D is the stronger baseline here and should remain visible in the comparison. The gain over it is 4.6 percentage points. The example shows a coherent body segmentation, but the torso boundary still extends differently from the ground truth. In the part-level table, Find3D actually has higher torso IoU. Real scanned objects are more challenging: on ScanObjectNN, PatchAlign3D reaches 22.7 versus Find3D's 18.8. That is an improvement at a low absolute score, not a solved robustness problem. The representation transfers useful structure, but noisy input, partial observations and shifts in object distribution still expose limitations. We can also inspect what the local features mean by changing the language query.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Sections 3.2–3.3; Tables 1–5
Published architecture, inference and benchmark details retained for discussion.
/Apps/Overleaf/CVPR2026_souhail/figs/faust_new/tr_scan_003/GT.png: Original PNG; white alpha composite; whitespace crop (336, 15, 597, 893)
FAUST scan 003 ground truth or segmentation output
/Apps/Overleaf/CVPR2026_souhail/figs/faust_new/tr_scan_003/PB.png: Original PNG; white alpha composite; whitespace crop (336, 15, 597, 893)
FAUST scan 003 ground truth or segmentation output
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Tables tab:faust and tab:partnete_scanobjectnn
Correct FAUST baseline and ScanObjectNN result

PatchAlign3DPublished

PatchAlign3D: patch encoding and point-level inference

Point cloud to farthest-point centers and overlapping local neighborhoods — 2,048 XYZ points → 128 FPS centers → 32 nearest neighbors per patch

Local encodingPointNet(P_i) + MLP(c_i)P_i: patch points; c_i: center

→

Context across patches12 transformer layers128 output patch tokens

→

Text projectionCompare each patch with part namesOpenCLIP ViT-bigG-14 text embeddings

ŷ(x) = arg max_j ⟨z_i(x), t_j⟩, i(x) = arg min_i ‖x − c_i‖

z: text-aligned patch feature; t: part-name embedding. Each point inherits its nearest center’s label.

The model consumes XYZ coordinates directly. Farthest-point sampling selects 128 centers, and each center gathers 32 nearest neighbors from the 2,048-point input. These neighborhoods overlap. A PointNet describes the local geometry; an MLP embeds the center position so that identical local patches at different locations remain distinguishable. A 12-layer transformer lets patch tokens exchange context, and the text head maps them into the OpenCLIP text space. At inference, each patch compares its feature with the requested part names. The equation shows how that patch prediction becomes a dense labeling: choose the nearest center for each point, then use the highest-scoring part of that center. The notation z here denotes the feature after the text projection. No rendering or 2D teacher is needed in this inference path. The nearest-center rule is efficient, but it also explains why very fine part boundaries can be difficult at a fixed patch resolution.

Sources and evidence

https://arxiv.org/pdf/2601.02457v2: Sections 3.2–3.3; Tables 1–5
Published architecture, inference and benchmark details retained for discussion.
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/03_method.tex: Data and Architecture / Architecture; Inference
PointNet patch encoding plus centroid MLP, 12-layer transformer, nearest-center point-label propagation
/Apps/Overleaf/CVPR2026_souhail (Arxiv)/04_experimental.tex: Implementation settings
2,048 XYZ points, 128 patches of 32 points, OpenCLIP ViT-bigG-14
presentation-archive/2026/2026-01-28_IMAGINE-Seminar_Language-Grounded-3D-Shapes.pdf: page 25
Style reference: explicit architecture linked to inference equations

Appendix · feedback learningOngoing work

What is trained, and what counts as supervision?

VLM-reward setting

Source points are provided.
DINOv3 is pretrained and frozen apart from LoRA.
The VLM scores sampled target matches.
Target annotations are used for evaluation, not the reward.

Separate diagnostic settings

Ground-truth PCK reward is supervised.
Geometry-aware fusion changes the feature stack.
Cat-only and all-category training are different regimes.
Peak validation is not held-out test performance.

This slide makes the supervision boundary precise. In the VLM-reward experiment, the source point is known and the learning signal for the proposed target comes from a frozen judge. The DINO backbone and optional diffusion features contain substantial pretraining information. Consequently, label-free refers to the absence of target-correspondence labels in the reward, rather than learning from no prior information. A separate diagnostic replaces the judge with a ground-truth PCK reward. That diagnostic tests the capacity and optimization of the visual policy, but is supervised and cannot substantiate a label-free claim. The manuscript also contains both cat-only and all-category training, with some all-category numbers selected at peak validation. These settings must remain separate. The main talk uses all-category peak validation and separates the plain-LoRA gain from the larger gain with geometry-aware features.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: Supervised overfit reward; REINFORCE training
Supervision boundary
C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Experimental setup
Cat-only versus all-category and peak-validation distinctions

Appendix · feedback learningOngoing work

Training details: stochastic updates and residual feature fusion

Policy update

ℒ_t = −(1/B)(r_t − b_t) log π_θ(q_t)

Temperature τ = 0.1; EMA rate 0.01.
Accumulate B = 10 samples per Adam step.
Clip gradient norm at 1.0.
Skip invalid scores; retain valid zero scores.

Optional geometry-aware features

Cached SD v1.5 features: 640 + 1280 + 1280 channels.
Live DINO ViT-B features: 768 channels.

F̂ = φ(F_DINO) + a · AggNet([F_SD; F_DINO])

φ: DINO projection; a: learnable fusion gate.
Fuse at a shared 60 × 60 feature grid.
Train LoRA and the aggregation network.
The residual branch preserves the DINO signal.

This slide holds the implementation details behind the main mechanism. The batch factor averages gradients over ten sampled matches before an Adam step. Gradient clipping limits the effect of unusually large updates. Invalid judge responses do not create a reward; a valid score of zero remains a negative training signal when the baseline is positive. In the optional geometry-aware branch, three cached Stable Diffusion feature scales are aligned to the same sixty-by-sixty grid as live DINO features. Their 3,968 channels feed an aggregation network. A separate DINO residual path avoids replacing useful pretrained features with a randomly initialized fusion output. Here a denotes the fusion gate; it is different from the LoRA scaling alpha. The residual projection φ maps the DINO features to the output space. The gate a controls the contribution of the aggregated stream. This architectural change is separate from the choice of VLM judge; changing both at once does not isolate the effect of reward quality.

Sources and evidence

Dropbox /Apps/Overleaf/ECCV_2026_keypoint_matching/sec/3_method.tex: Geo-Aware variant; Residual initialisation; REINFORCE Training
Residual fusion and joint adaptation; B=10 and baseline 0.01; output dimension omitted because manuscript configurations differ; geometry-aware result snapshots remain separate
Dropbox /Apps/Overleaf/ECCV_2026_keypoint_matching/sec/appendix.tex: Feature extraction; Aggregation network; Gradient accumulation and clipping
Cached feature dimensions and 60×60 grid; clipping; response handling

Appendix · feedback learningOngoing work

Transfer is measurable, but not uniform

Test category	Frozen DINOv3	Geo-aware, 3 seeds	Δ PCK
Cat, trained category	73.27	77.74 ± 0.64	+4.47
Bird	80.45	83.04 ± 0.27	+2.59
Cow	72.06	73.31 ± 0.45	+1.25
Sheep	61.50	62.64 ± 0.42	+1.14
Dog	61.89	62.82 ± 0.09	+0.93
Horse	65.03	64.32 ± 0.17	−0.71
Person	65.62	64.50 ± 0.56	−1.12

SPair-71k test · cat-trained only · bbox PCK@0.10. Other-category mean gain: +0.68; plain LoRA: +0.02.

These are held-out test results for the cat-trained geometry-aware model. The training category improves, as do bird, cow, sheep and dog, while horse and person decline. The average gain across the six non-cat categories is 0.68 points. For plain LoRA, the corresponding average is approximately zero. This constrains any claim that the adaptation has learned a broadly transferable matching rule. It may instead refine distinctions that are useful in a narrower training distribution, with the added geometry-sensitive feature stack helping some categories more than others. We also should not compare these test values directly with the main all-category peak-validation table, which uses a different training regime and evaluation split. The practical research question is which feedback and training distributions lead to transferable geometric skills, and how to diagnose apparent progress that disappears when the object family changes.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: tab:cross_cat_transfer
Cat-trained held-out category results

Learning from feedbackOngoing work

Alternative updates under the same judge budget

Method	PCK ↑
Frozen DINOv3	76.19
Top-K + LoRA	78.06 ± 0.04
DPO + LoRA	79.09
REINFORCE + LoRA	78.94 ± 0.14

SPair-71k cat validation · bbox PCK@0.10
Kimi reward · 3 seeds where uncertainty is shown.

+2.75 points
over the frozen backbone

The gain demonstrates useful feedback in this regime.

REINFORCE exceeds the listed Top-K setting; it does not establish superiority to DPO.

All rows use the plain DINOv3 setting on SPair cat validation. The reported Top-K baseline uses K=5; the listed DPO setting uses beta 0.1. Their scores show that VLM feedback is useful under several update rules. REINFORCE improves on the reported Top-K setting, while DPO is slightly higher in this table. The draft does not provide enough DPO implementation detail to reconstruct its full objective here, and its row has no seed uncertainty. We therefore use REINFORCE to explain the mechanism, without claiming optimizer superiority. Keep these validation results separate from held-out transfer and from geometry-aware variants.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Table main_results; VLM-reward training
Kimi cat-val comparison: frozen 76.19, Top-K 78.06±0.04, DPO 79.09, REINFORCE 78.94±0.14

Learning from feedbackOngoing work

Semantic feedback still needs a geometry-aware representation

Cat correspondence example comparing plain DINO matches in red, geometry-aware matches in blue and ground truth in green — One SPair cat test pair · red: plain DINO pipeline · blue: geometry-aware variant · green: ground truth

Adding geometry-sensitive features can help resolve symmetry. This is an architectural comparison, not an isolated reward ablation.

The judge is only one part of the system. The feature representation determines which correspondences the policy can distinguish. Here, the source and target cats have different poses, and several landmarks are vulnerable to left-right confusion. In this particular test pair, the geometry-aware variant gives correspondences much closer to the reference landmarks than the plain DINO pipeline. The optional branch combines DINO with cached Stable Diffusion features and a learned residual aggregation network. We should not attribute this entire visual improvement to the reward alone: the architecture and feature stack also change. Likewise, this is one qualitative example rather than a summary of all categories. It supports a narrower design lesson: high-level semantic evaluation is more useful when the representation already retains the geometric distinctions the task requires.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/appendix.tex: fig:symmetry
Plain versus geometry-aware test example, not pure before/after reward isolation
C:/Users/User/Dropbox/Apps/Overleaf/ECCV_2026_keypoint_matching/figures/symmetry_geo_vs_dino.pdf: original figure
Red plain, blue geo-aware, green target landmarks

Learning from feedbackOngoing work

Visual inference after reward training

Training

Visual proposal → VLM feedback
→ adapted visual features

Semantic supervision is expensive
but can be amortized.

Inference

Two images + source point
→ feature matching → target point

No VLM call.
26.3 FPS reported for the plain pipeline.

Reported timing: 512² input, RTX A6000. Geometry-aware timing uses cached SD features and is a different pipeline.

The trained adapter keeps inference visual; the VLM cost is paid during learning.

After adapter training, inference uses only visual feature matching. The manuscript reports 26.3 frames per second for the plain DINO pipeline. Its main.md timing snapshot specifies 512-square inputs, batch size one and an RTX A6000, averaging 100 passes after a ten-image warmup. The geometry-aware timing instead uses cached Stable Diffusion features and excludes their extraction; it should not be presented as the same end-to-end latency measurement. The benefit is that the judge is absent from deployment, rather than a universal speed ratio against every VLM service.

Sources and evidence

/Apps/Overleaf/ECCV_2026_keypoint_matching/sec/4_experiments.tex: Efficiency and comparison with prior work
Plain pipeline reports 26.3 FPS; no inference-time VLM
/Apps/Overleaf/ECCV_2026_keypoint_matching/main.md: Test-time efficiency
RTX A6000, 512-square plain model, batch 1, 100 passes; geometry-aware timing excludes SD extraction

Appendix · executable reconstructionPreliminary update

Uni3D similarity across all reported agents and settings

Agent	Single view	Four views	Active visual	Full 3D
Fable 5	.733	.764	.832	.884
Opus 5	.725	.748	.871	.927
GPT-5.6 Sol	.715	.745	.823	.879
GPT-6 Astra*	.794	.842	.912	.956
Kimi K3	.678	.713	.819	.848
Qwen 3.8 Max	.740	.768	.809	.857
Gemini 3.1 Pro	.728	.753	.691	.783
MiniMax M3	.495	.577	.482	.679

↑ Higher similarity. * Preliminary unpublished aggregate update; Astra run coverage and complete protocol parity remain unverified.

This table retains all eight reported series from the current aggregate update alongside the two-agent comparison used in the main talk. The first seven models belong to the original study. Astra is newly added and must remain visibly preliminary until matched instance coverage, evaluation settings and valid-run counts are checked against the underlying records. No missing supplementary measurements should be inferred from these aggregates. The table is useful for examining how each agent changes with access. For example, both Gemini and MiniMax decline when switching from prescribed multiview images to Active Visual, while other agents improve. A comparison across columns still changes interface and interaction protocol as well as accessible evidence. These similarities are complementary to geometric distance, topology and interaction cost, not a complete ranking of reconstruction quality.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/main_result-update.tex: all8agents; UNI3D column
Completeupdatedaggregate comparison; Astra preliminary

Appendix · executable reconstructionOngoing work

The harness changes the observation and interaction protocol

Setting	Target access	Target geometry	Refinement
Single view	1 fixed image	Hidden	3 rounds
Multi-view	4 fixed images	Hidden	3 rounds
Active Visual	Camera / viewport tools	Blocked by allowlist	Agent-directed
Full 3D	Unconstrained Blender tools	Available to query	Agent-directed

100 sampled objects from 3DCodeBench.
Original procedural programs are withheld.

Active Visual separates target and reconstruction processes.
Full 3D prohibits copying the target.

The four settings share the executable reconstruction objective and underlying target, but not every aspect of compute and interface. Fixed-image runs refine the program through three iterations, including execution debugging. Active Visual places the target and reconstruction in separate Blender processes, with an AST allowlist limiting target-side operations to approved viewing actions. Full 3D instead allows geometric inspection through the broader Blender interface, while instructing the agent not to duplicate the target. This is a study of agent capability under different harnesses rather than a perfectly budget-matched ablation of one additional pixel or measurement. The targets are sampled from a procedural-object benchmark and do not cover every kind of real-world scene. These limits are important when translating findings into a future training environment for articulated objects or larger scenes.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/evaluation_protocol.tex: fourharnessrows
Access/refinementdifferences
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/sec/4_experiment.tex: Benchmark data; Blender environment
100targets andprocessisolation

Appendix · executable reconstructionOngoing work

Evaluation conventions determine what a score can mean

Geometric alignment

Sample 8,192 surface points per mesh.
Center each cloud and scale to the unit sphere.

CD = min_{R ∈ R₂₄} d_sym(RX, Y)

d_sym: mean squared nearest-neighbor distances in both directions.
R₂₄: the 24 right-handed axis-aligned rotations.

Translation, scale and these rotations are discounted.

Complementary checks

Appearance: four 512² renders; match views by the best one-to-one assignment.

e_topo = Σ_i=0..2 |β_i^GT − β_i^rec| / Σ_i=0..2 β_i^GT

Topology: median across objects.
Other quality metrics: mean across objects.

No loadable mesh: zero similarity; Chamfer penalty of 1.5 × the worst valid value in that run.

The Chamfer comparison removes translation and absolute scale independently for each point cloud. It then searches a discrete set of twenty-four right-handed axis-aligned rotations, not arbitrary continuous alignment. The remaining error measures normalized surface geometry; it cannot prove that a program recovered metric dimensions or the original world pose. Appearance uses canonical renders with identical lighting and the best one-to-one assignment between target and reconstruction views, again tolerating a changed canonical orientation. The topology metric compares the first three Betti numbers after merging coincident vertices. Those count connected components, independent loops and enclosed two-dimensional cycles. It is aggregated by the median, while the other quality metrics use means. Failed outputs are penalized rather than silently dropped for similarity and Chamfer. These conventions should stay visible when discussing the leaderboard or comparing an individual qualitative example with dataset averages.

Sources and evidence

Dropbox /Apps/Overleaf/3DV2027/sec/X_suppl.tex: Experimental Settings / Metrics
Alignment, view matching, topology formula, aggregation and failure penalties; read directly from Dropbox on 20 September 2026

Appendix · executable reconstructionOngoing work

Ablations separate observation quality from evidence quantity

Agent / evidence setting	Chamfer ↓	Uni3D ↑
Gemini · four fixed views	.0165	.7609
Gemini · Active Visual	.0234	.6776
Gemini · normalized viewport	.0186	.7321
Gemini · Fable inspection views	.0191	.7605
MiniMax · Active Visual	.0798	.3722
MiniMax · normalized viewport	.0240	.6148

Ten-target ablation: better observations recover much of the lost performance, with metric-dependent outcomes.

The ten-target ablation provides more detail behind the observation-quality argument. Gemini benefits when its active screenshots are normalized and benefits further in Uni3D when supplied with inspection views collected by Fable. MiniMax improves substantially when the viewport is normalized. These comparisons separate aspects of acquiring evidence from the downstream reconstruction, but they do not establish a universal best inspection policy. A change that improves a learned shape similarity need not minimize geometric distance or topological error, and the subset is small. The fixed multiview baseline also remains competitive. For the future programme, this motivates controlled experiments where observation policies, geometric queries and repair actions are evaluated independently before they are trained jointly. Merely increasing the number of allowed tool calls is unlikely to be sufficient.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/2ablation_why_gemini.tex: observation ablation
Normalizedviewport andtransferredinspectionresults

Executable 3D hypothesesOngoing work

The agent chooses both an observation and a program update

a_t ∼ π_θ(· | H_t) H_t: instructions, observations and previous actions

Acquire evidenceNew view or geometric queryo_t: image; plus measurements in Full 3D

→

Revise the hypothesisBlender program P_tParts, dimensions, transforms and materials

→

Execute and inspectĜ_t = E(P_t)E: Blender execution; compare with the target

Active Visual

Separate target and reconstruction processes.
Target tools allow camera and viewport control.
Direct geometry access is blocked.

Full 3D Interaction

Query dimensions, transforms and distances.
The agent decides what to measure and when.
Copying the target is prohibited by instruction.

Evaluate the final executed object Ĝ_T, plus the interaction needed to obtain it.

The program is an explicit, revisable 3D hypothesis. At each step the agent sees the interaction history and chooses a tool action. That action may acquire a target observation, query a measurement when permitted, or revise and execute the reconstruction program. Blender execution maps the current program to an object that can be rendered and compared. Active Visual physically separates the target and reconstruction into two Blender processes. A restricted target-side interface allows approved viewing operations and blocks geometry access. Full 3D uses a shared environment with broader tools, allowing the agent to ask for dimensions and distances. Here the prohibition on copying the target is an instruction, not the same enforced isolation boundary. The target’s original authoring program is withheld in all settings. These are evaluations of existing agents: the policy notation describes their decisions, not a new agent-training algorithm implemented by this benchmark.

Sources and evidence

Dropbox /Apps/Overleaf/3DV2027/sec/3_benchmark.tex: Benchmark Formulation; Active Visual; Full 3D Interaction
History-conditioned actions, program execution, optional measurements and no-copy instruction
Dropbox /Apps/Overleaf/3DV2027/sec/4_experiment.tex: Evaluation Protocol
Separate processes and AST allowlist in Active Visual; shared environment in Full 3D

Executable 3D hypothesesPreliminary update

More access helps only when an agent can use it

Selected agents · original seven-agent results plus preliminary Astra aggregates.

Active inspection can improve or degrade reconstruction. Full 3D access includes explicit geometric evidence.

The aggregate chart follows three illustrative agents across the same four access settings shown in the bird comparison. Opus benefits strongly from agent-directed inspection, whereas Gemini drops relative to its fixed multiview result. More possible observations do not automatically create useful evidence: the agent must choose views, maintain a usable viewport and integrate what it sees. The dashed Astra line is a preliminary, unpublished aggregate update supplied in the current result table. Its values are 0.794, 0.842, 0.912 and 0.956. The underlying run coverage and complete protocol parity have not been verified here, so I present it descriptively and do not use it to establish a final leaderboard claim. The older inspection and repair analyses remain the original seven-agent study. Finally, these are Uni3D similarities, not percentages of accurately reconstructed geometry, and the zero-to-one axis is shown in full.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/main_result-update.tex: UNI3D column across4 harnesses
Selected Astra/Opus/Gemini aggregates; preliminaryAstra
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/sec/4_experiment.tex: Main Results
Originalseven-agentinterpretation

Executable 3D hypothesesOngoing work

Observation quality affects reconstruction

Gemini inspection sequence with two observations — Two Gemini inspections on this target.

Selected views from 22 Opus inspections of the same target.

On 10 ablation targets, viewport normalization raises MiniMax Uni3D: 0.3722 → 0.6148.

The observation process explains some surprising regressions under Active Visual. An agent can issue valid calls but stop early, or take many screenshots that remain badly framed, clipped, unshaded or poorly oriented. These original inspection strips show selected observations for one target; their counts should not be read as global averages. The controlled follow-up uses ten instances. Normalizing the viewport before capture raises MiniMax's Uni3D score from 0.3722 to 0.6148, while its fixed multiview score on that subset is 0.6200. So much of the failure is in obtaining usable evidence, not simply in the number of available views. Giving Gemini inspection views from a stronger policy also improves its result. This suggests that view selection and observation control should be explicit training targets, alongside the program that constructs the object.

Sources and evidence

C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/imgs/agent_inspection_comparison.pdf: Gemini andOpus rows
Selectedscreenshots andper-targetinspectioncounts
C:/Users/User/Dropbox/Apps/Overleaf/3DV2027/tables/2ablation_why_gemini.tex: MiniMax Active/Normalized rows
UNI3D .3722→.6148 on10instances

Appendix: contributions

Research ownership and collaboration

Project	Documented contribution
FourieRF / MILo / Spokes	Coauthorship, mentoring of Diego Gomez, MILo code contributions and LIX/GRAPHDECO collaboration
ZeroKey	Conceived, implemented and evaluated the first-author project
PatchAlign3D	Joint research with Souhail Hadgi and collaborators
Is It a Good Match?	Scientific lead since April 2026, experiments and manuscript
3DHarnessBench	Experiment guidance, project coordination, TaskSolver and restricted Blender interface

These projects involve different forms of contribution, and it is important to distinguish them. ZeroKey is a first-author project that I conceived, implemented and evaluated. For Is It a Good Match, I took scientific leadership in April 2026, ran the experiments and wrote the manuscript, working with Viktoria Ehm, Emery Pierson and Maks Ovsjanikov. On 3DHarnessBench, I guided experiments and coordinated the project with Ling Liu and our collaborators, and developed reusable agent infrastructure and the restricted Blender interface. FourieRF, MILo and Spokes sit within my collaboration and mentoring work with Diego Gomez, including MILo contributions and the connection to GRAPHDECO. PatchAlign3D is joint work with Souhail Hadgi and the author team. The attribution here follows the application dossier rather than inferring roles from author position alone.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: Mentoring and open-source software
Documented roles.
C:/Linux/CNRSApplication/telecom-paris-dossier/Publications_Gong.tex: Three selected publications and software
ZeroKey conception/implementation/evaluation and MILo contributor role.

Appendix: teaching

Teaching and mentoring

Experience

Teaching assistant for eight HKU courses
Certificate in Teaching and Learning in Higher Education
Forthcoming MVA geometry teaching appointment
Research mentoring across vision and graphics

Contribution at Télécom Paris

Graphics, geometric learning and neural 3D representations
Projects linking implementation to measurable evidence
Teaching in English, with a commitment to French within two years

My teaching background includes assistantships in eight HKU courses and completion of the Certificate in Teaching and Learning in Higher Education. The MVA appointment for autumn 2026 is forthcoming at the date of this talk, so I distinguish it from teaching already delivered. At Télécom Paris I could contribute immediately in graphics, geometric learning and neural 3D representations, alongside more general computing topics supported by my earlier teaching. I would use projects that connect implementation choices to observable evidence: reconstruct a scene, inspect its failure modes and evaluate an edit or geometric query. I can teach in English and am learning French, with the commitment stated in my dossier to teach in French within two years. Mentoring and reproducible research practice would be integral to that contribution.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Enseignement_Gong.tex: Experience, pedagogical training and integration
HKU teaching, CTLHE, forthcoming MVA and language commitment.
C:/Linux/CNRSApplication/telecom-paris-dossier/Rapport_Activites_Gong.tex: Teaching and mentoring
Roles and scope; no invented precise teaching hours.

Appendix: research developmentProposed work

A staged research programme

Stage	Scientific focus	Development
First year	Controlled static/articulated tasks, verified trajectories, compact baselines	Initial student project and reproducible releases
Second year	Transfer to new operations and richer dynamics	Collaborative funding and industrial research links
Third year	Integrated reconstruction, animation and delivery evaluation	Broader partnerships and HDR preparation

Resources and extensions follow demonstrated progress on the core tasks.

The staged programme begins with a tractable benchmark and learning question. In the first year, the priority is reproducible static and articulated tasks, verified trajectories and compact-model baselines. The second stage tests transfer to new operations and introduces richer dynamics only where the earlier measurements support it. The third stage evaluates integration with reconstruction, animation and delivery. Funding proposals and industrial links should support these scientific milestones rather than create unrelated projects. The dossier identifies possible French and European funding routes, along with CIFRE opportunities in media, games and simulation, subject to the relevant calls and eligibility. HDR preparation and broader partnerships fit into the longer development of an independent group. The exact expansion should depend on resources and results from the core work.

Sources and evidence

C:/Linux/CNRSApplication/telecom-paris-dossier/Descriptif_Recherche_Gong.tex: Funding, team building and timeline
Proposed staged programme, without implying funding awards or agreed partnerships.

From Reconstructionto ReasoningStructured 3D Representations

What makes a reconstruction geometrically reliable?

How precisely can meaning be grounded in geometry?

When does feedback improve behavior?

MILo · TOG / SIGGRAPH Asia 2025

From Blobs to Spokes · ECCV 2026

Surface fields

Mesh extraction

Feature distance → policy

Adapt only the LoRA weights

Advantage: rt − bt

Gradient through the policy

Right part ≠ exact location

What changes under feedback?

Target

Result

100-target comparison

Controlled viewport ablation

First program versus final program

What the representation enables

Conceptual scene description

What changes

What optimizes the field

Preserve selected Gaussian centers

Suppress unsupported interior surfaces

From Reconstruction
to ReasoningStructured 3D Representations

Advantage: r_t − b_t