Research · 2026

Have I Scene This Before?

A memory for assistants on smart glasses that connects conversations to 3D objects reconstructed from video.

Role
Co-first author; led the spatial memory approach
Team
D3M Lab, University of TorontoProf. Scott Sanner
When
May – Oct 2026
Outcome
Submitted to ICLR 2027
Built with
PyTorch, Depth Anything 3, SegVGGT, Open3D

The problem

Answering "Which of Sam’s things would fit in this box?" means knowing which objects Sam said were theirs and how big they are. An assistant on smart glasses needs to connect the conversation to the room, without reprocessing the video for every question.

What I built

I led the paper’s spatial memory approach and designed and built the 3D reconstruction behind every result. It turns ordinary video into persistent objects with positions and sizes, without a depth sensor or supplied camera poses.

More detail

My pipeline generated the object maps for all 39 benchmark scenes. I designed the cross-frame matching and merging to keep repeated furniture distinct, including 39 identical chairs in one room.

What improved

52.5% accuracy across 3,100 questions, compared with 43.1% for the strongest baseline.

Removing the 3D geometry I built dropped accuracy from 51.5% to 34.9% in the paper’s ablation study.

More detail

Across the benchmark’s 39 scenes, my reconstruction recovered 91.1% of 864 objects within half a metre of their true positions.