Image Projection - Department of Computer Science ...

1 Image ProjectionGoal:Introduce the basic concepts and mathematics for Image :The mathematics of Image Projection allow us to answer two questions: Given a 3D scene, how does it project to the Image plane? ( Forward model.) Given an Image , what 3D scenes could project to it? ( Inverse model.) Vision is all about guessingthe scene and the story behind it. The latter is a (largely ignored) holy grail of Computer :Szeliski, Chapter : Image Projectionc Allan Jepson, Sept. 2011 Page: 1 The Pinhole CameraImage formation can be approximated with a simple pinhole camera,XYZxyP(x,y,f)(X,Y,Z) Image Plane, Z=fThe Image position for the 3D point(X, Y, Z)is given by the projective transformation xyf =fZ XYZ The distance between the Image plane and the projective pointPis called the focal length, f. Note: for mathematical convenience we put the Image plane in front of the nodal point (since this avoidsthe need to flip the Image coords about the origin); Image coordinatexis taken to the right, andydownwards.

2 This agrees with the standard rasterorder and the convention of a right-handed coordinate frame(X, Y, Z). the primary approximation here is that there is no optical blur, distortion, or defocus (discussedlater).CSC420: Image ProjectionPage: 2 Coordinate FramesConsider the three coordinate frames: World coordinate frame,~Xw. These are 3D coordinates fixed in the world, say with respect toone corner of the room. Camera coordinate frame,~Xc. These are 3D coordinates fixed in the camera. The origin of thecamera coordinates is at the center of Projection of the camera (say at~dwin world coords). Thez-axis is taken to be the optical axis of the camera (with points infront of the camera in the positivezdirection). Image coordinate frame,~p. The Image coordinates are written as a 3-vector,~p= (p1, p2,1)T,withp1andp2the pixel coordinates of the Image point.

3 Here the origin is in the top-left corner ofthe Image (or, in Matlab, the top-left corner has pixel coords (1,1)).The first Image coordinatep1increases to the right, andp2increases we express the transforms from world coordinates to camera coordinates and then to Image : Image ProjectionPage: 3 Extrinsic Calibration MatrixThe extrinsic calibration parameters specify the transformation fromworld to camera coordinates,which is a standard 3D coordinate transformation,~Xc=Mex[~XTw,1]T.(1)Here the extrinsic calibration matrixMexis a3 4matrix of the formMex=(R R~dw),(2)withRis a3 3rotation matrix and~dwis the location, in world coordinates, of the center of projectionof the camera. The inverse of this mapping is simply~Xw=RT~Xc+~dw.(3)The perspective transformation can now be applied to the 3D point~Xc( , in the camera s coordi-nates),~xc=fX3,c~Xc= x1,cx2,cf.

4 (4)Everything here is measured in meters (say), not pixels, andfis the camera s focal : Image ProjectionPage: 4 Intrinsic Calibration MatrixThe intrinsic calibration matrix,Min, transforms the 3D Image position~xc(measured in meters, say)to pixel coordinates,~p=1fMin~xc,(5)whereMinis a3 3matrix. The factor of1/fhere is example, a camera with rectangular pixels of size1/sxby1/sy, with focal lengthf, and piercingpoint(ox, oy)( , the intersection of the optical axis with the Image plane provided in pixel coordi-nates) has the intrinsic calibration matrixMin= f sx0ox0f syoy0 0 1 .(6)Note that, for a 3D point~xcon the Image plane, the third coordinate of the pixel coordinatevector~pisp3= 1. As we see next, this redundancy is (1), (4) and (5) define the transformation from the world coordinates of a 3D point,~Xw, tothe pixel coordinates of the Image of that point,~p.

5 The transformation is nonlinear, due to the scalingbyX3,cin equation (4).CSC420: Image ProjectionPage: 5A Note on UnitsSo far we have written the focal lengthfin meters. But note that only the termsf sxandf syappear inthe intrinsic calibration matrix,Min= f sx0ox0f syoy0 0 1 ,wheresx,yare in the units of horizontal/vertical pixels per meter (andox,yare in pixels).Instead of meters, it is common to measurefin units of pixel width, that is, replacef sxbyf. In whichcase the intrinsic calibration matrix becomesMin= f0ox0f a oy0 0 1 ,(7)wherea=sy/sxis the (unitless) aspect ratio of a pixel (0< a <1if the pixels are rectangular andflat,a= 1if the pixels are square, anda >1rectangular and tall).CSC420: Image ProjectionPage: 6 Homogeneous CoordinatesThe projective transform becomes linear when written in the following homogeneous coordinates,~Xhw=c(~XTw,1)T,~ph=d~p=d(p1 , p2,1) , dare arbitrary nonzero constants.

6 The last coordinate of these homogeneous vectors providethe scale factors. It is therefore easy to convert back and forth between the homogeneous forms andthe standard mapping from world to pixel coordinates can then be written as thelineartransformation,~ph=MinMex~Xhw.( 8)Essentially, the division operation in perspective Projection is now implicit in the homogeneous vector~ph. The division is simply postponed until~phis rescaled by its third coordinate to form the pixelcoordinate vector~ to its linearity, equation (8) is useful in many areas of computational : Image ProjectionPage: 7 Example: Lines Project to LinesAs a first application of the perspective Projection equation (8), con-sider a line in 3D written in homogeneous coordinates, say~Xh(s) =(~X01)+s(~t0).Here~X0is an arbitrary 3D point on the line expressed in world coor-dinates,~tis a 3D vector tangent to the line, andsis the free parameterfor points along the line.

7 To avoid special cases, we assume that theline does not pass through the center of Projection , and the tangent di-rection~thas a positive inner-product with the optical axis (more on thisbelow). By equation (8), the Image the point of~Xh(s)is~ph(s) =M~Xh(s) =~ph(0) +s~pht,whereM=MinMexis a3 4matrix,~ph(0) =M((~X0)T,1)T, and~pht=M(~tT,0)T. Note~phtand~ph(0)are both constant vectors, inde-pendent ofs. Therefore the Image of the 3D line, in pixel coordinates,is~p(s) 1ph3(s)~ph(s) =1 (s)~ph(0) +s (s)~pht,(9)where (s) =ph3(s). Using equations (1) and (7) we find (s) =ph3(0) + s,for =pht,3=~eT3 Mex(~tT,0)T,(10)where~eT3= (0,0,1). The condition that the inner-product of~tand thedirection of the optical axis is positive is equivalent to > that equation (9) shows that~p(s)is in the plane spanned by twoconstant 3D vectors. It is also in the Image plane,p3= 1.

8 Therefore itis in the intersection of these two planes, which is a line in the is, lines in 3D are imaged as lines in 2D. (Although, in practice,some lenses introduce radial distortion , which we discuss later.)One caveat on eqn (9) is that some of these points may be behindtheprincipal plane (and therefore behind the camera). Using equations(1) and (7) it follows thatXc,3(s), theZ-component of the point onthe line written in camera coordinates, is equal to the thirdcomponent~ph(s), which we denoted by (s)above. Thus the point is in front ofthe principal plane if and only if (s)>0(and in front of the lens if (s)> cfor some constantc >0.)Since >0we have from (10) that1/ (s) 0ands/ (s) 1/ ass . Therefore, from (9), the Image points~p(s) (1/ )~phtass . Note that this limit point is a constant Image point dependentonly on the tangent direction~ fact, in homogeneous world coordinates, the 4D vector(~tT,0)Tisthe point at infinity in the direction~t.

9 The perspective Projection of thispoint is simply~pht=M(~tT,0)T, which is homogeneously equivalentto the limit of the Image points we derived above. The next exampleexplores this fact : Image ProjectionNotes: 8 Example: Parallel Lines Project to Intersecting LinesNext consider a set of parallel lines in 3D, say~Xhk(s) =(~X0k1)+s(~t0).Here all these lines have the same tangent direction~t, and hence areparallel in 3D (both in the world and camera coordinates).To eliminate special cases, we again assume that none of these linespasses through the center of Projection , and~thas a positive inner-product with the direction of the optical axis ( , >0, with definedas in equation (10)).Then from the previous example we know that, ass , the perspec-tive projections of the points~Xhk(s)all converge to the same imagepoint, namely~pht=M(~tT,0) the images of the parallel 3D lines~Xhk(s)all intersect at theimage point~pht.

10 Moreover, it can be shown from equations (9) and(10) that, under the natural condition that we only form the Image ofpoints on the 3D line which are in front of the principal plane( ,Xc,3(s) = (s)>0), the projected points on the Image line segmentsconvergemonotonicallyto~pht. That is, in the Image , the projected linesegments all appear to terminate at~pht. (For example, note the sides ofthe road in the left figure above. Although, as the right figureshows,we can always be surprised.)In summary, the common termination point for the images of parallellines in 3D is the perspective Projection of the 3D tangential direction~t. It is referred to as thevanishing : Image ProjectionNotes: 9 Example: The Horizon LineAs another exercise in projective geometry, we consider multiple sets of parallel lines, all of which are coplanar in show that the images ofeach parallel set of lines intersect and terminate at a pointon the horizon line in the multiple families of parallel lines in a plane, where each fam-ily of lines has the tangent direction~tjin 3D.