Math Camp 1: Functional analysis

1 Math Camp 1: Functional analysisAbout the primerGoalTo briefly review concepts in Functional analysis thatwill be used throughout the course. The followingconcepts will be described1. Function spaces2. Metric spaces3. Dense subsets4. Linear spaces5. Linear functionals The definitions and concepts come primarily from Introductory RealAnalysis by Kolmogorov and Fomin (highly recommended).6. Norms and semi-norms of linear spaces7. Euclidean spaces8. Orthogonality and bases9. Separable spaces10. Complete metric spaces11. Hilbert spaces12. Riesz representation theorem13. Convex functions14. Lagrange multipliersFunction spaceAfunction spaceis a space made of functions. Eachfunction in the space can be thought of as a point.

2 Ex- [a, b], the set of all real-valuedcontinuousfunctionsin the interval [a, b]; [a, b], the set of all real-valued functions whose ab-solute value is integrable in the interval [a, b]; [a, b], the set of all real-valued functions square inte-grable in the interval [a, b]Note that the functions in 2 and 3 are not necessarilycontinuous!Metric spaceBy ametric spaceis meant a pair (X, ) consisting of aspaceXand a distance , a single-valued, nonnegative,real function (x, y) defined for allx, y Xwhich has thefollowing three properties:1. (x, y) = 0iffx=y;2. (x, y) = (y, x);3. Triangle inequality: (x, z) (x, y) + (y, z)Examples1. The set of all real numbers with distance (x, y) =|x y|is the metric space The set of all orderedn-tuplesx= (x1.)

3 , xn)of real numbers with distance (x, y) = n i=1(xi yi)2is the metric space The set of all functions satisfying the criteria f2(x)dx < with distance (f1(x), f2(x)) = (f1(x) f2(x))2dxis the metric spaceL2(IR).4. The set of all probability densities with Kullback-Leiblerdivergence (p1(x), p2(x)) = lnp1(x)p2(x)p1(x)dxis not a metric space. The divergence is not symmetric (p1(x), p2(x))6= (p2(x), p1(x)).DenseA pointx IR is called acontact pointof a setA IR ifevery ball centered atxcontains at least one point set of all contact points of a setAdenoted by Aiscalled subspaces of a metric space saidto bedenseinBifB A. In particularAis said to beeverywhere densein IR if A= The set of all rational points is dense in the real The set of all polynomials with rational coefficients isdense inC[a, b].

4 3. The RKHS induced by the gaussian kernel on [a, b] indense inL2[a, b]Note: A hypothesis space that is dense inL2is a desiredproperty of any approximation spaceA setLof elementsx, y, z, ..is alinear spaceif the fol-lowing three axioms are satisfied:1. Any two elementsx, y Luniquely determine a thirdelement inx+y Lcalled the sum ofxandysuchthat(a)x+y=y+x(commutativity) (b) (x+y) +z=x+ (y+z) (associativity)(c) An element 0 Lexists for whichx+ 0 =xfor allx L(d) For everyx Lthere exists an element x Lwith the propertyx+ ( x) = 02. Any number and any elementx Luniquely deter-mine an element x Lcalled the product such that(a) ( x) = ( x)(b) 1x=x3. Addition and multiplication follow two distributive laws(a)( + )x= x+ x(b) (x+y) = x+ yLinear functionalA Functional ,F, is a function that maps another functionto a real-valueF:f linear Functional defined on a linear spaceL, satisfies thefollowing two properties1.

5 Additive:F(f+g) =F(f) +F(g) for allf, g L2. Homogeneous:F( f) = F(f)Examples1. Let IRnbe a real n-space with elementsx= (x1, .., xn),anda= (a1, .., an) be a fixed element in IRn. ThenF(x) =n i=1aixiis a linear functional2. The integralF[f(x)] = baf(x)p(x)dxis a linear functional3. Evaluation Functional : another linear Functional is theDirac delta function t[f( )] =f(t).Which can be written t[f( )] = baf(x) (x t) Evaluation Functional : a positive definite kernel in aRKHSFt[f( )] = (Kt, f) =f(t).This is simply the reproducing property of the spaceAnormedspace is a linear (vector) spaceNin which anorm is defined. A nonnegative function is a normiff f, g Nand IR1. f 0 and f = 0ifff= 0;2. f+g f + g ;3.

6 F =| | f .Note, if all conditions are satisfied except f = 0ifff= 0then the space has a seminorm instead of a distances in a normed spaceIn a normed spaceN, the distance betweenfandg, orametric, can be defined as (f, g) = g f .Note that f, g, h N1. (f, g) = 0ifff= (f, g) = (g, f).3. (f, h) (f, g) + (g, h).Example: continuous functionsA norm inC[a, b] can be established by defining f = maxa t b|f(t)|.The distance between two functions is then measured as (f, g) = maxa t b|g(t) f(t)|.With this metric,C[a, b] is denoted (cont.)A norm inL1[a, b] can be established by defining f = ba|f(t)| distance between two functions is then measured as (f, g) = ba|g(t) f(t)| this metric,L1[a, b] is denoted (cont.)

7 A norm inC2[a, b] andL2[a, b] can be established by defining f =( baf2(t)dt)1 distance between two functions now becomes (f, g) =( ba(g(t) f(t))2dt)1 this metric,C2[a, b] andL2[a, b] are denoted spaceAEuclideanspace is a linear (vector) spaceEin which adot product is defined. A real valued function ( , ) is a dotproductiff f, g, h Eand IR1. (f, g) = (g, f);2. (f+g, h) = (f, h ) + (g, h) and ( f, g) = (f, g);3. (f, f) 0 and (f, f) = 0ifff= Euclidean space becomes anormed linear spacewhenequipped with the norm f = (f, f).Orthogonal systems and basesA set of nonzero vectors{x }in a Euclidean spaceEissaid to be anorthogonal systemif(x , x ) = 0 for 6= and anorthonormal systemif(x , x ) = 0 for 6= (x , x ) = 1 for =.

8 An orthogonal system{x }is called anorthogonal basisif it is complete (the smallest closed subspace containing{x }is the whole spaceE). A complete orthonormal sys-tem is called anorthonormal IRnis a realn-space, the set ofn-tuplesx= (x1, .., xn),y= (y1, .., yn). If we define the dot product as(x, y) =n i=1xiyiwe get Euclideann-space. The corresponding normsand distances in IRnare x = n i=1x2i (x, y) = x y = n i=1(xi yi) vectorse1=(1,0,0, ..,0)e2=(0,1,0, ..,0) en=(0,0,0, ..,1)form an orthonormal basis in The spacel2with elementsx= (x1, x2, .., xn, ..),y=(y1, y2, .., yn, ..), .., where i=1x2i< , i=1y2i< , .., ..,becomes an infinite-dimensional Euclidean space whenequipped with the dot product(x, y) = i= simplest orthonormal basis inl2consists of vectorse1=(1,0,0,0.)

9 E2=(0,1,0,0, ..)e3=(0,0,1,0, ..)e4=(0,0,0,1, ..) there are an infinite number of these The spaceC2[a, b] consisting of all continuous functionson [a, b] equipped with the dot product(f, g) = baf(t)g(t)dtis another example of Euclidean important example of orthogonal bases in this spaceis the following set of functions1,cos2 ntb a,sin2 ntb a(n= 1,2, ..).Cauchy-Schwartz inequalityLetHbe an Euclidean space. Then f, g H, it holds|(f, g)| f g Sketch of the casef gis trivial, hence letus assume the opposite is true. For allx IR,0<(f+xg, f+xg) =x2 g 2+ 2x(f, g) + f 2,since the quadratic polynomial ofxabove has no zeroes,the discriminant must be negative0> /4 = (f, g)2 f 2 g metric space is said to beseparableif it has a countableeverywhere dense :1.

10 The spaces IR1, IRn,L2[a, b], andC[a, b] are all The set of real numbers is separable since the set ofrational numbers is a countable subset of the reals andthe set of rationals is is everywhere sequence of functionsfnisfundamentalif >0 N such that nandm > N , (fn, fm)< .A metric space iscompleteif all fundamental sequencesconverge to a point in the ,L1, andL2are complete. ThatC2is not complete,instead, can be seen through a ofC2 Consider the sequence of functions (n= 1,2, ..) n(t) = 1 if 1 t < 1/nntif 1/n t <1/n1 if 1/n t 1and assume that nconverges to a continuous function in the metric ofC2. Letf(t) ={ 1 if 1 t <01 if 0 t 1 Incompleteness ofC2(cont.)Clearly,( (f(t) (t))2dt)1/2 ( (f(t) n(t))2dt)1/2+( ( n(t) (t))2dt)1 the term is strictly positive, becausef(t) is notcontinuous, while forn we have (f(t) n(t))2dt , contrary to what assumed, ncannot convergeto in the metric of a metric spaceGiven a metric space IR with closure IR, a complete metricspace IR is called acompletionof IR if IR IR and IR = IR.}