Image-Space Modal Bases for Plausible Manipulation of ...

1 Image-Space Modal Bases for Plausible Manipulation of Objects in VideoAbe Davis1 Justin G. Chen2Fr edo Durand11 MIT CSAIL2 MIT Dept. of Civil and Environmental ForcesMaskInput VideoSynthesized HzXYExtracted Mode ForcesObject ResponseImage DeformationModal Basis(from recovered shapes)Figure 1:By extracting the vibration modes of a wire figure from small deformations in a five second video captured with an SLR, we are ableto create an interactive 2D simulation of the figure. Left: an image from the input video showing the object at its rest state, with a rough maskshown in the bottom right corner. Middle: deformation modes extracted from the x and y dimensions of the video at different : synthesized deformations of the object responding to user-defined present algorithms for extracting an Image-Space representa-tion of object structure from video and using it to synthesize physi-cally Plausible animations of objects responding to new, previouslyunseen forces.

2 Our representation of structure is derived from animage-space analysis of Modal object deformation: projections ofan object s resonant modes are recovered from the temporal spectraof optical flow in a video, and used as a basis for the image-spacesimulation of object dynamics. We describe how to extract thisbasis from video, and show that it can be used to create physically- Plausible animations of objects without any knowledge of scene ge-ometry or material [Image Processing and Computer Vision]:Scene Analysis Time-varying Imagery;Keywords:Video, Physically based animation, Video Synthesis,Video Textures, Modal Analysis, Animation, Interactive1 IntroductionComputational photography seeks to capture richer informationabout the world, and provide new visual experiences.

3 One of themost important ways that we experience our environment is by ma-nipulating it: we push, pull, poke, and prod to test hypotheses aboutour surroundings. By observing how objects respond to forces thatwe control, we learn about their dynamics. Unfortunately, videodoes not afford this type of Manipulation - it limits us to observingthe dynamics that were recorded. However, in this paper we showthat many videos contain enough information to locally predict howrecorded objects will respond to new, unseen forces. We use this in-formation to build Image-Space models of object dynamics around arest state, letting us turn short video clips into physically- Plausible ,interactive techniques for physically-based animation derive the proper-ties that govern object dynamics from known virtual models.

4 How-ever, measuring these properties for objects in the real world canbe extremely difficult, and estimating them from video alone isseverely underconstrained. A key observation of our work is thatthere is often enough information in video to create a physicallyplausible model of object dynamics around a rest state in which theobject is filmed, even when fundamental ambiguities make recov-ering a general or fully-accurate model impossible. We show howto extract these physically Plausible models from short video clips,and demonstrate their use in two Animation:Video makes it easy to capture the ap-pearence of our surroundings, but offers no means of physical in-teraction with recorded objects.

5 In the real world, such interactionsare a crucial part of how we understand the physical properties ofobjects. By building a model of dynamics around the state in whichan object is filmed, we turn videos into interactive animations thatusers can explore with virtual forces that they Effects:In film special effects, where objects often needto respond to virtual forces, it is common to avoid modeling thedynamics of real objects by compositing human performances intovirtual environments. Performers act in front of a green screen,and their performance is later composited with computer-generatedobjects that are easy to simulate. This approach can produce com-pelling results, but requires considerable effort: virtual objects mustbe modeled, their lighting and appearence made consistent with anyreal footage being used, and their dynamics synchronized with alive performance.

6 Our work addresses many of these challenges bymaking it possible to apply virtual forces directly to objects as theyappear in OverviewOur approach is based on the same linear Modal analysis behindmany techniques in physically-based animation. However, unlikemost of these techniques, we do not assume any knowledge of ob-ject geometry or material properties, and therefore cannot rely onfinite element model (FEM) methods to derive a Modal basis forsimulation. Instead, we observe non-orthogonal projections of anobject s vibration modes directly in video. For this we derive arelationship between projected modes and the temporal spectra ofoptical flow. We then show that, while non-orthogonal, these pro-jections can still be used as a basis to simulate Image-Space accurate physical models of objects in video is severelyunderconstrained.

7 To deal with this ambiguity, we make a few keyassumptions, which we analyze in Section Related WorkPhysically-based Animation:Many techniques in physically-based animation use Modal analysis to reduce the degrees of free-dom in deformable body simulations [Pentland and Williams 1989;James and Pai 2002; James and Pai 2003; Pai et al. 2001; Huanget al. 2011; Li et al. 2014]. These techniques work by first derivingorthogonal vibration modes from known geometry using FEM ap-proaches. As high frequency modes generally contribute less to anobject s deformation, they can often be discarded to obtain a lower-dimensional basis for faster simulation. We use a similar reducedmodal basis to simulate objects in video, but assume no knowledgeof scene geometry and cannot therefore use FEM approaches tocompute vibration modes.

8 Instead, we observe projections of thesemodes directly in video and show that, while non-orthogonal, theseprojections can still be used as a basis to simulate the dynamics ofobjects in Vibration ModesThe problem of directly observingvibration modes has been explored in several engineering disci-plines, where the structure of objects must be carefully validated inthe real world, even when a virtual model is available. The generalapproach is to relate the spectrum of surface motion, typically mea-sured with accelerometers, to mode shapes. [Helfrick et al. 2011]applied this analysis to motion estimated with a stereo rig, whichthey used to recover mode shapes for shell-like work in graphics and vision has used narrow-band phase-based motion magnification to visualize the Modal vibrations of ob-jects in video [Wadhwa et al.]

9 2013; Wadhwa et al. 2014; Chen et ]. [Davis et al. 2014; Davis et al. 2015] proposed an alterna-tive visualization based on the temporal spectra of weighted opticalflow. However, both approaches focus on providing a visualizationtool, and neither has been used to recover a basis for simulation. Weshow that a similar algorithm, borrowing aspects of each of thesevisualization techniques, can be used to recover mode shapes thatare suitable for Synthesis in Video:Several works in computer graph-ics and vision have focused on synthesizing Plausible animations ofquasi-periodic phenomena based on a video exemplar [Doretto et ; Szummer and Picard 1996; Chuang et al. 2005; Sch odl et ; Pentland and Sclaroff 1991; Tao and Huang 1998].

10 In mostof these applications, video synthesis is formulated as a stochas-tic process with parameters that can be fit to the exemplar. Suchapproaches work especially well for animating phenomena like rip-pling water or smoke, and with skeletal information provided bya user have even been extended to model the motion of structurescaused by stochastic forces like wind [Stam 1996; Sun et al. 2003].The applications we address are similar to many of these works inspirit, but, to our knowledge, we are the first to build image-spacesimulations based on a Modal Bases extracted directly from MagnificationLike recent publications in motion mag-nification [Wadhwa et al. 2013; Wadhwa et al. 2014; Chen et ], our work can be used to magnify and visualize small vibra-tions of an object .