Week 2

What we had so far:

IoU 0.743 from the trained voxel decoder
91% classification accuracy on the latent built from a frozen pretrained ResNet18

What we tried this week 2:

whether retrieval already explains the decoder reconstruction problem
an SDF field decoder
point-cloud representation to building the latent z
different multiview aggregators

Question 1: is the decoder recovering shape (recognition) rather than reconstructing?

IoU 0.743 with flat view sweeps (1 to 20 views moved nothing) is also the signature of a decoder that recognises the class and emits its average shape.

baseline	how it is computed	IoU
global mean training grid	average all training grids into one shape, emit it for every test part. Ignores the input entirely, so this is the floor	0.0169
class mean	average training grids within each class into 43 classes, then emit the class of the test part's true label	0.4304
1-NN on frozen ImageNet feats, nothing trained	max-pool cached ResNet18 features over views to 512-d, L2-normalise, cosine-match each test part against all 1,694 train parts, emit the retrieved train part's grid	0.7384
1-NN in the trained latent z	same procedure, trained 256-d z	0.7436
Trained decoder	the actual model	0.7429

Conclusion: decoder does more than class recognition, but nearest-neighbour retrieval matches it, and retrieval on untrained features does too (interestingly the same conclusion as in Tatarchenko et al. CVPR 2019).

Why retrieval works so well: 47% of test parts have a near-twin in training (cosine >= 0.99).

Question 2: if a 32^3 voxel grid is the limit, does an SDF field decoder do better?

VoxelDecoder:  z          ->  [32,32,32]   resolution fixed at build time
FieldDecoder:  (z, xyz)   ->  scalar       resolution chosen at query time

An implicit function: "is this 3D point inside the solid?" z says which object, xyz says where in space. Trained on 8,192 query points per part.