Have I Scene This Before?
A memory for assistants on smart glasses that connects conversations to 3D objects reconstructed from video.
- Role
- Co-first author; led the spatial memory approach
- Team
- D3M Lab, University of TorontoProf. Scott Sanner
- When
- May – Oct 2026
- Outcome
- Submitted to ICLR 2027
- Built with
- PyTorch, Depth Anything 3, SegVGGT, Open3D
The problem
Answering "Which of Sam’s things would fit in this box?" means knowing which objects Sam said were theirs and how big they are. An assistant on smart glasses needs to connect the conversation to the room, without reprocessing the video for every question.
What I built
I led the paper’s spatial memory approach and designed and built the 3D reconstruction behind every result. It turns ordinary video into persistent objects with positions and sizes, without a depth sensor or supplied camera poses.
More detail
My pipeline generated the object maps for all 39 benchmark scenes. I designed the cross-frame matching and merging to keep repeated furniture distinct, including 39 identical chairs in one room.
What improved
52.5% accuracy across 3,100 questions, compared with 43.1% for the strongest baseline.
Removing the 3D geometry I built dropped accuracy from 51.5% to 34.9% in the paper’s ablation study.
More detail
Across the benchmark’s 39 scenes, my reconstruction recovered 91.1% of 864 objects within half a metre of their true positions.