The Art of Linear Algebra - raw.githubusercontent.com

1 The Art of Linear Algebra Graphic Notes on Linear Algebra for Everyone Kenji Hiranabe with the kindest help of Gilbert Strang September 1, 2021/updated March 23, 2023 AbstractI tried intuitive visualizations of important concepts introduced in Linear Algebra for Everyone .1 This is aimed at promoting understanding of vector/matrix calculations and algorithms from theperspectives of matrix factorizations. They include Column-Row (CR), Gaussian Elimination (LU),Gram-Schmidt Orthogonalization (QR), Eigenvalues and Diagonalization (Q QT), and Singular ValueDecomposition (U VT). All the artworks including this article are maintained in the GitHub am happy to see Kenji Hiranabe s pictures of matrix operations in Linear Algebra !

2 The pictures are anexcellent way to show the Algebra . We can think of matrix multiplications by row column dot products,but that is not all it is Linear combinations and rank 1 matrices that complete the Algebra and the am very grateful to see the books in Japanese translation and the ideas in Kenji s pictures. Gilbert StrangProfessor of Mathematics at MITC ontents1 Viewing a Matrix 4 Ways22 Vector times Vector 2 Ways23 Matrix times Vector 2 Ways24 Matrix times Matrix 4 Ways45 Practical Patterns46 The Five Factorizations of a .. QT.. VT.. 10 twitter: @hiranabe, Massachusetts Institute of Technology, ~gs/1 Linear Algebra for Everyone : Japanese translation started by Kindai Viewing a Matrix 4 WaysA matrix (m n) can be seen as 1 matrix,mnnumbers,ncolumns andmrows.

3 !!!" !# $% &' (!# $% &' (!# $% &' (!"#$%&'(")*#+$, +0 1"(&'2*,-1",$.")*#+$, +0 !"(&'2*,-3"(&'2*,-4"'5+,/6 Figure 1:Viewing a Matrix in 4 WaysA= a11a12a21a22a31a32 = ||a1a2|| = a 1 a 2 a 3 Here, the column vectors are in bold asa1. Row vectors include as ina 1. Transposed vectors andmatrices are indicated by T as Vector times Vector 2 WaysHereafter I point to specific sections of Linear Algebra for Everyone and present graphics which illustratethe concepts with short names in colored circles. Sec. ( ) Linear combination and dot products Sec. ( ) Matrix of Rank One Sec. ( ) Row way and column way! !!!"#$%&"'()#$$*+(, +2$3$41#&56!"!!"#$ %&$ %"$ "%#$ #%! " #$!$"$#&!"#'$!$"$#& $!( "$"( #$#!"$57$1$,1#&56$#!))))))))

4 "$$ %&8$9:$+.5#;.&$'()1 .$&.7(<#$%57$1$&1+2$3$,1#&568!"#$%&"'()# *!+ "/$57$.6%&.77.'$17$!$"5+$,1#&56$<1+=(1=.$1+'$>5.<'7$1$+(, !#Figure 2:Vector times Vector - (v1), (v2)(v1) is a elementary operation of two vectors, but (v2) multiplies the column to the row and produce arank 1 matrix. Knowing this outer product (v2) is the key for the later Matrix times Vector 2 WaysA matrix times a vector creates a vector of three dot products (Mv1) as well as a Linear combination (Mv2)of the column vectors Sec. ( ) Linear combinations Sec. ( ) Matrices and Column Spaces!!"!"#$%&'$(#)*&%+$&,$ #$/01*2312#4$56$.$(#)*&%$ $5#)&/#$*"#$*"%##$4&*93%&40)*$#1#/#8*+$ !"#$3%&40)*$:7 2+$.$128#.%$)&/528.*2&8$&,$*"#$)&10/8$(# )* !

5 " #$ %& '( )*!*"#+*!,%*"-+&*!, '*"-+(*!, )*"-!" #$ %& '( )*!*"# *!$&(, *"%')!"#!"$Figure 3:Matrix times Vector - (Mv1), (Mv2)At first, you learn (Mv1). But when you get used to viewing it as (Mv2), you can understandAxas alinear combination of the columns ofA. Those products fill the column space ofAdenoted asC(A). Thesolution space ofAx=0is the nullspace ofAdenoted asN(A).Also, (vM1) and (vM2) shows the same patterns for a row vector times a matrix.!" #$!$"$#% &' () *#+$!,'$", )$#- +&$!, ($", *$#-!"#!"$!"!!" #$!$"$#% &' () *# $!% &, $"' (, $#) *!"#$%&'()*+$,- ./$0$ #0&$*' +.'2$'5$+"#$&'6$7#*+'&/$'5$898$&'6$7#*+' &$, ./$3)1+.%1.#($4:$+"#$+6'$*'1)32$7#*+'&/$ '5$802($4#*'3#$+"#$+6'$('+;%&'()*+$#1#3# 2+/$'5$,-9"Figure 4:Vector times Matrix - (vM1), (vM2)The products fill the row space ofAdenoted asC(AT).)

6 The solution space ofyA= 0 is the left-nullspaceofAdenoted asN(AT).The four subspaces consists ofN(A) +C(AT) (which are perpendicular to each other) inRnandN(AT)+C(A) inRm(which are perpendicular to each other). Sec. ( ) Dimensions of the Four Subspaces3!!!"!"#$%&'()*!+!""#"#,"-'()*! +!""#$"&$##()*!+"# % &#+./'&$##()*!+$" % &% '% '% ( )'% *)'!"#!"#!"#!"#+"#+,"-.,"-.,"#-" / !"$!!!!")+,)+&01!$#*,)+,)+&01!$#*,!!% ." 0+"#." 1 +"#!"% +" 0."#2" 1 3"#Figure 5:The Four SubspacesSeeA=CR(Sec ) for the Matrix times Matrix 4 Ways Matrix times Vector naturally extends to Matrix times Matrix . Sec. ( ) Four Ways to MultiplyAB=C Also see the back cover of the book!! "! "# $% &'!(!'"(")*'!+"'", *(!+"(",*#'!+$'", *#(!

7 +$(",*%'!+&'", *%(!+&(",! "# $% &'!(!'"(") -. /)-. -/!!!!! "# $% &0!!0!"0"!0"")1#1$2#%2$%) 1#2#%+1$2$%)!#%0!!0!"+"$&0"!0"")0!!0!"#0 !!#0!"%0!!%0!"+"0"!"0""$0"!$0""&0"!&0"" 37869:48376;<=>:43?:<4@6<A6;<9B=4@6<A6CD! "# $% &'!(!'"(")1#%1$%1&%E )1#%E1$%E1&%EFB9?:G9:;3?:<46-H :@6>7<I8465<J46?<636@B=6<A6734I6!6=3?7:;8@DKL86G7<5B;8567<J@637869:48376;<=>:43?:<4@6<A67<J@DMN87O6898=84?6>8;<=8@6365<?6G7<5B;?6<A67<J6N8;?<73456;< 9B=46N8;?<7D!!"!!#!!$!!%Figure 6:Matrix times Matrix - (MM1), (MM2), (MM3), (MM4)5 Practical PatternsHere, I show some practical patterns which allow you to capture the coming factorizations more ! "!#! "#!"#$%&'()*+,$(-+&.#+$'/.&+%0&+()+&.#+0 (12-)*+(,+&.#+-%&$'34+5.'*+#3"$#**'()+0% )+6#+*##)+%*+&.

8 #+&.$##+1')#%$+0(-6')%&'()*+')+&.#+$'/.& +')+()#+,($-21%4!"!"#!!"#!##!!#"""!!!#"" ""!!#"""$$#$%#$%&'(!"#$%&'()*+,$(-+&.#+1 #,&+%0&+()+&.#+$(7*+(,+&.#+-%&$'34+5.'*+ #3"$#**'()+0%)+6#+*##)+%*+&.#+&.$##+1')# %$+0(-6')%&'()*+')+&.#+$'/.&+')+()#+,($- 21%4!!"!"#""!"!"#"#!"!"#"$$&$%&'(%$#Figu re 7:Pattern 1, 2 - (P1), (P1)Pattern 1 is a combination of (MM2) and (Mv2). Pattern 2 is an extention of (MM3). Note that Pattern1 is a column operation (multiplying a matrix from right), whereas Pattern 2 is a row operation (multiplyinga matrix from left).!!!" #$!$"$#%$%%%&#%$$!%%$"%&$#"& #%$%%%&'$''%''&'#%$'$'%%'%'%&'&'!""#$%&' ()(*%)'+&)#(,) (0.+,(-12(.%'1-34)#23(2)41(4+#5,&6!""#$% &'()(*%)'+&)#(,) (0.+,(-12(#20-34)#23(2)41(.+76!"#!$#Figu re 8:Pattern 1 , 2 - (P1 ), (P2 )(P1 ) multipies the diagonal numbers to the columns of the matrix, whereas (P2 ) multipies the diagonalnumbers to the row of the matrx.)))

9 Both are variants of (P1) and (P2).!"#$%&'(()*+%,'-)$%'+.(")*%/.,0#+'( #.+%.1%%/.23,+$ #22%)+/.3+()*%("#$%#+%7#11)*)+(#'28*)/3* *)+/)%)93'(#.+$4!"# $%!%"%#&$&%&&'$'%'&$ '$&$%!( '%&%%"!'&&&%#!""!"Figure 9:Pattern 3 - (P3)This pattern appears when you solve differential equations and recurrence equations: Sec. 6 ( ) Eigenvalues and Eigenvectors Sec. ( ) Systems of Differential Equations5du(t)dt=Au(t),u(0) =u0un+1=Aun,u0=u0In both cases, the solutions are expressed with eigenvalues ( 1, 2, 3), eigenvectorsX=[x1x2x3]ofA, and the coefficientsc=[c1c2c3]Twhich are the coordinates of the initial conditionu(0) =u0interms of the +c2x2+c3x3c= c1c2c3 =X 1u0and the general solution of the two equations are:u(t) =eAtu0=Xe tX 1u0=Xe tc=c1e 1tx1+c2e 2tx2+c3e 3tx3un=Anu0=X nX 1u0=X nc=c1 n1x1+c2 n2x2+c3 n3x3 See Figure 9: Pattern 3 (P3) above again to getXDc.

10 !""!"#!$%"%#%$&%&&&''%!'&!''!$ &%%%'%!( &&%&'&!!&'%'''!!"#$%&'("')"*&+,-."/+0."% +"$")1#"+2"&$.,"3"#$%&'4-)5$)"'.")'.617$ &"8$71-9-' $71-"/-4+#:+)'%'+.;!"Figure 10:Pattern 4 - (P4)This pattern (P4) works in both eigenvalue decomposition and singular value decomposition. Both de-compositions are expressed as a product of three matrices with a diagonal matrix in the middle, and also asum of rank 1 matrices with the eigenvalue/singular value details are discussed in the next The Five Factorizations of a Matrix Preface , The Plan for the , A=LU, A=QR, A=Q QT, A=U VTare illustrated one by columns inCRow echelon form inRLeads to column rank = row rankA=LULU decomposition fromGaussian elimination(Lower triangular)(Upper triangular)A=QRQR decomposition asGram-Schmidt orthogonalizationOrthogonalQand triangularRS=Q QTEigenvalue decompositionof a symmetric matrixSEigenvectors inQ, eigenvalues in A=U VTSingular value decompositionof all matricesASingular values in Table 1.