4 Vector/Matrix Derivatives and Integrals

1 4 Vector/Matrix Derivatives and IntegralsThe operations of differentiation and integration of vectors and matrices arelogical extensions of the corresponding operations on scalars. There are threeobjects involved in this operation: the variable of the operation; the operand (the function being differentiated or integrated); and the result of the the simplest case, all three of these objects are of the same type, andthey are scalars. If either the variable or the operand is a vector or a matrix,however, the structure of the result may be more complicated. This statementwill become clearer as we proceed to consider specific this chapter, we state or show the form that the derivativetakes interms of simpler Derivatives . We state high-level rules forthe nature of thedifferentiation in terms of simple partial differentiation of a scalar with respectto a scalar.

2 We do not consider whether or not the derivativesexist. In gen-eral, if the simpler Derivatives we write that comprise the more complicatedobject exist, then the derivative of that more complicated object exists. Oncea shape of the derivative is determined, definitions or derivations in - termscould be given, but we will refrain from that kind of formal exercise. Thepurpose of this chapter is not to develop a calculus for vectors and matricesbut rather to consider some cases that find wide applicationsin a more careful treatment of differentiation of vectors and matrices, thereader is referred to Rogers (1980) or to Magnus and Neudecker (1999). An-derson (2003), Muirhead (1982), and Nachbin (1965) cover various aspects ofintegration with respect to vector or matrix Basics of DifferentiationIt is useful to recall the heuristic interpretation of a derivative.

3 A derivativeof a function is the infinitesimal rate of change of the function with respectMatrix Algebrac 2007 James E. Gentle146 4 Vector/Matrix Derivatives and Integralsto the variable with which the differentiation is taken. If both the functionand the variable are scalars, this interpretation is unambiguous. If, however,the operand of the differentiation, , is a more complicated function, say avector or a matrix, and/or the variable of the differentiation, , is a morecomplicated object, the changes are more difficult to measure. Change in thevalue both of the function, = new old,and of the variable, = new old,could be measured in various ways; for example, by using various norms, asdiscussed in Sections and (Note that the subtraction is not neces-sarily ordinary scalar subtraction.)Furthermore, we cannot just divide the function values by.

4 We do nothave a definition for division by that kind of object. We need amapping,possibly a norm, that assigns a positive real number to . We can definethe change in the function value as just the simple differenceof the functionevaluated at the two points. This yieldslim 0 ( + ) ( ) .( )So long as we remember the complexity of , however, we can adopt asimpler approach. Since for both vectors and matrices, we have definitions ofmultiplication by a scalar and of addition, we can simplify the limit in theusual definition of a derivative, 0. Instead of using as the elementof change, we will uset , wheretis a scalar and is an element to be addedto . The limit then will be taken in terms oft 0. This leads tolimt 0 ( +t ) ( )t( )as a formula for the derivative of with respect to .The expression ( ) may be a useful formula for evaluating aderivative,but we must remember that it is not the derivative.

5 The type ofobject ofthis formula is the same as the type of object of the function, ; it does notaccommodate the type of object of the argument, , unless is a scalar. Aswe will see below, for example, if is a vector and is a scalar, the derivativemust be a vector , yet in that case the expression ( ) is a expression ( ) is rarely directly useful in evaluating a derivative, butit serves to remind us of both the generality and the complexity of the and its arguments could be functions, for example. (In functionalanalysis, various kinds of functional Derivatives are defined, such as a G ateauxderivative. These Derivatives find applications in developing robust statisticalmethods; see Shao, 2003, for example.) In this chapter, we are interested in thecombinations of three possibilities for , namely scalar, vector , and matrix,and the same three possibilities for and.

6 Matrix Algebrac 2007 James E. Basics of Differentiation 147 ContinuityIt is clear from the definition of continuity that for the derivative of a functionto exist at a point, the function must be continuous at that point. A functionof a vector or a matrix is continuous if it is continuous for each elementof the vector or matrix. Just as scalar sums and products are continuous, Vector/Matrix sums and all of the types of Vector/Matrix products we havediscussed are continuous. A continuous function of a continuous function of the Vector/Matrix functions we have discussed are clearly con-tinuous. For example, the Lpvector norms in equation ( ) are continuousover the nonnegative reals but not over the reals unlesspis an even (posi-tive) integer. The determinant of a matrix is continuous, aswe see from thedefinition of the determinant and the fact that sums and scalar products arecontinuous.

7 The fact that the determinant is a continuous function immedi-ately yields the result that cofactors and hence the adjugate are the relationship between an inverse and the adjugate (equation ( )),we see that the inverse is a continuous and PropertiesWe write the differential operator with respect to the dummy variablexas / xor / xT. We usually denote differentiation using the symbol for par-tial differentiation, , whether the operator is written xifor differentiationwith respect to a specific scalar variable or xfor differentiation with respectto the arrayxthat contains all of the individual elements. Sometimes, how-ever, if the differentiation is being taken with respect to the whole array (thevector or the matrix), we use the notation operand of the differential operator / xis a function ofx. (If itis not a function ofx that is, if it is a constant function with respect tox then the operator evaluates to 0.)

8 The result of the operation, written f/ x, is also a function ofx, with the same domain asf, and we sometimeswrite f(x)/ xto emphasize this fact. The value of this function at the fixedpointx0is written as f(x0)/ x. (The derivative of the constantf(x0) isidentically 0, but it is not necessary to write f(x)/ x|x0because f(x0)/ xis interpreted as the value of the function f(x)/ xat the fixed pointx0.)If / xoperates onf, andf:S T, then / x:S U. The natureofS, or more directly the nature ofx, whether it is a scalar, a vector , ora matrix, and the nature ofTdetermine the structure of the resultU. Forexample, ifxis ann- vector andf(x) =xTx, thenf: IRn IRand f/ x: IRn IRn,Matrix Algebrac 2007 James E. Gentle148 4 Vector/Matrix Derivatives and Integralsas we will see. The outer product,h(x) =xxT, is a mapping to a higher rankarray, but the derivative of the outer product is a mapping toan array of thesame rank; that is,h: IRn IRn nand h/ x: IRn IRn.

9 (Note that rank here means the number of dimensions; see page 5.)As another example, considerg( ) = det( ), sog: IRn n7 this case, g/ X: IRn n7 IRn n;that is, the derivative of the determinant of a square matrixis a square matrix,as we will see differentiation is a composition of the / xoperator withitself or of the / xoperator and the / xToperator. For example, considerthe familiar function in linear least squaresf(b) = (y Xb)T(y Xb).This is a mapping from IRmto IR. The first derivative with respect to them-vectorbis a mapping from IRmto IRm, namely 2 XTXb 2 XTy. The secondderivative with respect tobTis a mapping from IRmto IRm m, namely, 2 XTX.(Many readers will already be familiar with these facts. We will discuss thegeneral case of differentiation with respect to a vector in Section )We see from expression ( ) that differentiation is a linearoperator; thatis, ifD( ) represents the operation defined in expression ( ), is anotherfunction in the class of functions over whichDis defined, andais a scalarthat does not depend on the variable , thenD(a + ) =aD( ) +D( ).

10 This yields the familiar rules of differential calculus for Derivatives of sums orconstant scalar products. Other usual rules of differentialcalculus apply, suchas for differentiation of products and composition (the chain rule). We canuse expression ( ) to work these out. For example, for the derivative of theproduct , after some rewriting of terms, we have the numerator ( )( ( +t ) ( ))+ ( )( ( +t ) ( ))+( ( +t ) ( ))( ( +t ) ( )).Now, dividing bytand taking the limit, assuming that ast 0,( ( +t ) ( )) 0,Matrix Algebrac 2007 James E. Types of Differentiation 149we haveD( ) =D( ) + D( ),( )where againDrepresents the differentiation a differentiable scalar function of a scalar variable,f(x), thedifferentialoffatcwith incrementuisudf/dx|c. This is the linear term in a truncatedTaylor series expansion:f(c+u) =f(c) +uddxf(c) +r(c, u).