Computational Imaging: The Race Against Time

1 Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Imaging: The Race Against time Computational Imaging: The Race Against time Paul DebevecUSC Institute for Creative TechnologiesUSC Viterbi School of Engineering2005 Symposium on Computational Photography and VideoMIT Stata Center 24 May 2005 Paul DebevecUSC Institute for Creative TechnologiesUSC Viterbi School of Engineering2005 Symposium on Computational Photography and VideoMIT Stata Center 24 May 2005objectPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 ( ui,vi , i, i)Ri( ui,vi , i, i)incident light fieldincident light fieldRr( ur,vr, r, r)Rr( ur,vr, r, r)Ri( ui,vi , i, i)

2 Ri( ui,vi , i, i)incident light fieldincident light fieldradiant light fieldradiant light fieldPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance FieldThe Reflectance FieldR( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)8D reflectance field8D reflectance fieldSince it is linear, we can represent as a matrixSince it is linear, we can represent as a matrixReflectance FieldStorage RequirementsReflectance FieldStorage Requirements360 x 180 x 180 x 180 x 360 x 180 x 180 x 180= measurementsx 6 bytes/pixel (in RGB 16-bit)= 26 exabytes (billion GB)= 82 million 300GB hard drives(41 million if we exploit Helmholz Reciprocity)360 x 180 x 180 x 180 x 360 x 180 x 180 x 180= measurementsx 6 bytes/pixel (in RGB 16-bit)= 26 exabytes (billion GB)= 82 million 300GB hard drives(41 million if we exploit Helmholz Reciprocity)R( ui, vi , i, i;ur, vr, r, r)R( ui, vi , i, i.)

3 Ur, vr, r, r)Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance FieldThe Reflectance FieldR( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)8D reflectance field8D reflectance fieldThe Reflectance FieldThe Reflectance FieldR( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)8D reflectance field8D reflectance fieldPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance FieldThe Reflectance FieldR( xi , yi, zi, i, i;xr, yr, zr, r, r)R( xi , yi, zi, i, i;xr, yr, zr, r, r)10D reflectance / scattering field (plenoptic functions in and out)10D reflectance / scattering field (plenoptic functions in and out)The Reflectance FieldThe Reflectance FieldR( xi , yi, zi, i, i;xr, yr, zr, r, r)R( xi , yi, zi, i, i;xr, yr, zr, r, r)10D reflectance field10D reflectance fieldPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance FieldThe Reflectance FieldR( xi , yi, zi, i, i, t ; xr, yr, zr, r, r)R( xi , yi, zi, i, i, t;xr, yr, zr, r, r)11D reflectance field11D reflectance fieldHawkins, Einarsson, and Debevec.

4 Capturing time -Varying Participating Media , SIGGRAPH 2005 (to appear)Hawkins, Einarsson, and Debevec. Capturing time -Varying Participating Media , SIGGRAPH 2005 (to appear)Smoke Scanning VideoSmoke Scanning VideoPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Scanning AnalysisSmoke Scanning AnalysisWeakly scattering medium allows one laser plane sweep to cover 3 DAngular dependence measured independentlyCamera sees nearly unoccluded view along the z axisWeakly scattering medium allows one laser plane sweep to cover 3 DAngular dependence measured independentlyCamera sees nearly unoccluded view along the z axisR( xi , yi, zi, i, i, t ; xr, yr, zr, r, r)R( xi , yi, zi, i, i, t.)

5 Xr, yr, zr, r, r)DISCODISCOM ichael Goesele, Hendrik Lensch, Jochen Lang, Christian Fuchs and Hans-Peter Seidel. DISCO - Acquisition of Translucent Objects. SIGGRAPH Goesele, Hendrik Lensch, Jochen Lang, Christian Fuchs and Hans-Peter Seidel. DISCO - Acquisition of Translucent Objects. SIGGRAPH ( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance FieldThe Reflectance FieldR( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)8D reflectance field8D reflectance fieldR( ui,vi , i, i;ur,vr, r, r)R( ui,vi , i, i;ur,vr, r, r)4D Slices of the 8D Reflectance Field4D Slices of the 8D Reflectance Fielddistantilluminationsingle camera4D reflectance field4D reflectance fieldPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Light Stage VideoDual Light Stage VideoHawkins, Einarsson, and Debevec.

6 A Dual Light Stage , EGSR 2005 (to appear)Hawkins, Einarsson, and Debevec. A Dual Light Stage , EGSR 2005 (to appear)What was gained?What was gained?Better reflectance resolutionDoes reciprocity help win the race Against time ? amount of information being captured with the same number of measurementsLaser produces bright, concentrated lightBetter reflectance resolutionDoes reciprocity help win the race Against time ? amount of information being captured with the same number of measurementsLaser produces bright, concentrated lightLight Stage 2 Dual Light StagePaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Reflectance Field4D Reflectance Fieldilluminationcamera4D reflectance field4D reflectance fieldR( i, i;ur,vr)R( i, i;ur,vr)4D Reflectance Field4D Reflectance Fieldilluminationcamera4D reflectance field4D reflectance fieldR( i, i;ur,vr)R( i, i.)

7 Ur,vr)Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 4D Reflectance FieldTime-Varying 4D Reflectance Fieldilluminationcamera5D5DR( i, i, t ; ur,vr)R( i, i, t;ur,vr)Light Stage 5 Light Stage 5 Andreas Wenger, Chris Tchou, Andrew Gardner, Tim Hawkins, Jonas Unger, Paul Debevec. Performance Relighting and Reflectance Transformation with time -Multiplexed Illumination , SIGGRAPH 2005 (to appear)Andreas Wenger, Chris Tchou, Andrew Gardner, Tim Hawkins, Jonas Unger, Paul Debevec. Performance Relighting and Reflectance Transformation with time -Multiplexed Illumination , SIGGRAPH 2005 (to appear)Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Stage 5 VideoLight Stage 5 Video4320 lighting conditions per second24 frames per second= 180 lighting conditions per framePaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Motion CompensationMotion blur resynthesizedNo Motion CompensationOptical Flow FieldSurface NormalsAlbedo (diffuse/ambient color)

8 Ambient OcclusionBasis ImagePaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 4D Reflectance FieldTime-Varying 4D Reflectance Fieldilluminationcamera5D5DR( i, i, t ; ur,vr)R( i, i, t;ur,vr)Paul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 8D Reflectance FieldTime-Varying 8D Reflectance FieldR( ui,vi , i, i, t ; ur,vr, r, r)R( ui,vi , i, i, t;ur,vr, r, r)9D9 DIdea: Use an underlying model to extrapolate other dimensionsReal- time 3D ScanningReal- time 3D ScanningS. K. Nayar, M. Watanabe, and M. Noguchi, "Real- time Focus Range Sensor," ICCV Rusinkiewicz, O. Hall-Holt, M. Levoy. Real- time 3D Model Acquisition.

9 SIGGRAPH Zhang, Noah Snavely, Brian Curless, and Steven M. Seitz. Spacetime Faces: High-resolution capture for modeling and animation. SIGGRAPH B. Vieira, L. Velho, A. Sa, P. C. Carvalho. A Camera-Projector System for Real- time 3D Video. Procams K. Nayar, M. Watanabe, and M. Noguchi, "Real- time Focus Range Sensor," ICCV Rusinkiewicz, O. Hall-Holt, M. Levoy. Real- time 3D Model Acquisition. SIGGRAPH Zhang, Noah Snavely, Brian Curless, and Steven M. Seitz. Spacetime Faces: High-resolution capture for modeling and animation. SIGGRAPH B. Vieira, L. Velho, A. Sa, P. C. Carvalho. A Camera-Projector System for Real- time 3D Video. Procams Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 , Gardner, Bolas, MacDowall, and Debevec.

10 Performance Geometry Capture for Spatially Varying Illumination , ICT Technical Report ICT-TR-01-2005 SIGGRAPH 2005 Technical SketchSpatial Relighting VideoSpatial Relighting VideoNa ve Spatial RelightingPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 Indirect Illumination CorrectionTime-Varying Reflectance FieldTime-Varying Reflectance FieldR( xi , yi, zi, i, i, ti;xr, yr, zr, r, r)R( xi , yi, zi, i, i, ti;xr, yr, zr, r, r) time of Flight Laser ScanTime of Flight Laser ScanCan measure depth, separate direct/indirect reflection, surface/subsurfaceTime-ResolvedReflectan ce FieldTime-ResolvedReflectance FieldR( xi , yi, zi, i, i, ti;xr, yr, zr, r, r, tr)R( xi , yi, zi, i, i, ti;xr, yr, zr, r, r, tr)One More DimensionPaul Debevec, MIT Symposium on Computational Photography and VideoMay 25, 2005 could a time -resolved light stage dataset be useful?