Lecture 3: Solving Equations Using Fixed Point Iterations

1 Cs412: introduction to numerical analysis09/14/10 Lecture 3: Solving Equations Using Fixed Point IterationsInstructor: Professor Amos Ron Scribes: Yunpeng Li, Mark Cowlishaw, Nathanael FillmoreOur problem, to recall, is Solving Equations in one variable. We are given a functionf, andwould like to find at least one solution to the equationf(x) = 0. Note that,a priori, we do notput any restrictions on the functionf; we do need to be able to evaluate the function: otherwise,we cannot even check that a given solutionx=ris true, , thatf(r) = 0. In reality, the mereability to be able to evaluate the function does not suffice. Weneed to assume some kind of goodbehavior . The more we assume, the more potential we have, onthe one hand, to develop fastalgorithms for finding the root. At the same time, the more we assume, the fewer functions aregoing to satisfy our assumptions! This is a fundamental paradigm in Numerical from last week that we wanted to solve the equation:x3= sinxorx3 sinx= 0(1)We know that 0 is a trivial solution to the equation, but we would like to find a non-trivialnumeric solutionr.

2 In a previous Lecture , we introduced an iterative process for finding roots ofquadratic Equations . We will now generalize this process into an algorithm for Solving equationsthat is based on the so-calledfixed Point Iterations , and therefore is referred to asfixed pointalgorithm. In order to use Fixed Point Iterations , we need the followinginformation:1. We need to know that there is a solution to the We need to know approximately where the solution is ( an approximation to the solution).1 Fixed Point IterationsGiven an equation of one variable,f(x) = 0, we use Fixed Point Iterations as follows:1. Convert the equation to the formx=g(x).2. Start with an initial guessx0 r, whereris the actual solution (root) of the Iterate, usingxn+1:=g(xn) forn= 0,1,2, ..How well does this process work? We claim (xn) 0usingxn+1:=g(xn)as described in the process above. If(xn) 0convergesto a limitr, and the functiongis continuous atx=r, then the limitris a root off(x):f(r) = is this true?

3 Assume that (xn) 0converges to some valuer. Sincegis continuous, thedefinition of continuity implies thatlimn xn=r limn g(xn) =g(r)1 Using this fact, we can prove our claim:g(r) = limn g(xn) = limn xn+1= ,g(r) =r, and since the equationg(x) =xis equivalent to the original onef(x) = 0, weconclude thatf(r) = that, for proving this claim, we had to make some assumption ong(viz,gis a continuousfunction). This follows the general pattern: the more restrictions we put on a function, the betterwe can analyze the function real trick of Fixed Point Iterations is in Step 1, finding a transformation of the originalequationf(x) = 0 to the formx=g(x) so that (xn) 0converges. Using our original example,x3= sinx, here are some sin 1(x3) 1x2+x+1+ x3 sinx3x2 cosx (x)Figure 1: Graphical Solution forx3= sinxWe can start withx0= 1, since this is a pretty good approximation to the root, as shown inFigure 1. To choose the best functiong(x), we need to determine how fast (and if)xnconverges toa solution.

4 This is key, because different transformations of a singlef(x) = 0 tox=g(x) can resultin a sequence ofxnthat diverges, converges to the root slowly, or converges tothe root good way to measure the speed of convergence is to use the ratio of the errors betweensuccessive Iterations . The error at iterationncan be defined as:en:=xn r(2)2whereris the actual solution (an alternative definition, isen:=|xn r|.) To measure the rate ofconvergence, we can take the ratio n+1between the error at iterationn+ 1 and the error at theprevious iteration: n+1:=en+1en=xn+1 rxn r(3)However, as you may have noticed, we are Using the actual solutionr, which we do not know,to calculate the errorenand the ratio n. For as long as we do not knowr, we can approximatethe erroren=xn r xn xn 1, and thus we can calculate the error ratio as: n+1=en+1en xn+1 xnxn xn 1(4)Note that the magnitude of the error ratio is what is important, so we can safely ignore Order of ConvergenceClearly, we would like the magnitude of the error ratio to be less than 1.

5 We introduce now twonotions concerning orders of Linear ConvergenceLinear convergence requires that the error is reduced by at least a constant factor at each iteration:|en+1| c |en|(5)for some Fixed constantc <1. We will study algorithms that converge much more quickly thanthis, in fact, we have already seen an algorithm (the square root algorithm) that quadratic ConvergenceQuadratic convergence requires that the error at each iteration is proportional to the square of theerror on the previous iteration:|en+1| c |en|2(6)for some constantc. Note that, in this case,cdoes not have to be less than 1 for the sequence toconverge. For example, if|en| 10 4, then|en+1|< c 10 8, so that a relatively large constant canbe offset by the Superlinear ConvergenceIn general, we can have|en+1| c |en| (7)for constantscand . If = 1, we have linear convergence, while if = 2 we have quadraticconvergence. If >1 (but is not necessarily 2), we say we have superlinear is important to note that Equations 5, 6, and 7 providelower boundson the convergence is possible that an algorithm with quadratic convergencewill converge more quickly than thebound indicates in certain circumstances, but it will neverconverge more slowly than the boundindicates.

6 Also note that these bounds are a better indicator of the performance of an algorithmwhen the errors are small (en 1). Experimental Comparison of Functions for Fixed Point IterationsWe Now return to our test the equationx3= sinx.(8)How do the functions we considered forg(x) compare? Table 1 shows the results of severaliterations Using initial valuex0= 1 and four different functions forg(x). Herexnis the value ofxon thenth iteration and nis the error ratio of thenth iteration, as defined in Equation (x) =3 sinxg(x) =sinxx2g(x) =x+ sinx x3g(x) =x sinx x3cosx 3x2x1 1 2 3 4 26 27 1: Comparison of Functions for Fixed Point IterationsWe can see from the table that when we chooseg(x) =3 sinxorg(x) =x sinx x3cos(x) 3x2(columns1 and 4, respectively), the algorithm does converge to with the error ratio nfar less than 1. However, when we chooseg(x) =sinxx2, the error ratio is greater than 1 and theiteration do not converge.

7 Forg(x) =x+ sinx x3, the error ratio is very close to 1. It appearsthat the algorithm does converge to the correct value, but very is there such a disparity in the rate of convergence?2 Error Analysis of Fixed Point IterationsIn order to carry our analysis of Fixed Point Iterations , we assume thatgis differentiable. Note thecontinued theme: the more we restrict a function, the betterour tools are for analyzing the function4numerically. We previously assumedgto be continuous; now we raised the bar and assume it tobe would like to see howgis behaving in the area around the solution,r. For this, we will usethe Taylor expansion with remainder. Remember that, for anydifferentiable functiong, and forany two valuesx, a, there existscsuch thatg(x) =g(a) +g (c) (x a).(9)We don t know what the precise value ofcis, but we know that it exists and is Testing for DivergenceSubstitutingxnforxandr(the analytic solution) fora, we can use Equation 9 to provide a testfor divergence:en+1=xn+1 r=g(xn) g(r)= (xn r) g (cn)cnbetweenxnandr=en g (cn)(10)Asxnapproachesr,cn(which is betweenxnandr) is getting closer tor, and, thereforeg (cn) g (r).

8 This means that if|g (r)|>1, oncexngets close enough tor, the error will getlarger, and the sequencex0, x1, x2, .. suggests a straightforward test to see if a particular functiongis a poor choice for fixedpoint Iterations . Simply check the numerical values ofg in some interval that is known the containthe solutionr. If|g (x)|>1 on this interval, then the sequence ofx0, x1, x2, ..will not convergeandgis a poor example, if we look atg(x) =sinxx2forx [.9,1], we see that|g (x)|is greater than 1 overthe interval. If we look atg(x) =3 sinxon the same interval, its derivative is explains why the choiceg(x) =sinxx2failed in our experiment andg(x) =3 sinxproduced Testing for ConvergenceWe now have a way to test for divergence, but what about testing for convergence? We can uselocal analysis around the actual solutionrto determine the relevant behavior ofg. Define theintervalI= [x0 , x0+ ] for some small constant with the analytic solutionr I.

9 If we canguarantee that the magnitude of the derivative is less than 1overI: c I g (c) <1for some constant , then we will have convergence, as long as our valuesxistay in the can quantify this with the following two c I1> g (c) 0then the sequencex0, x1, x2, ..stays in intervalI, and willconverge to the thatxis our present approximation ofr, and assume by induction that is lies in theintervalI. Our next approximation isg(x) and once we show that it also lies in the interval, ourinductive proof will go through (our initialx0lies inIby assumption!). Now, our error analysisshows thatg(x) =g(r) + (x r)g (c), for somecbetweenxandr. Now,x I, by induction, andr Iby assumption, hencec I, hence 0< g (c)<1. Also,g(r) =r. Altogether,g(x) =r+ (x r)g (c) =r(1 g (c)) +xg (c),so,g(x) is an average ofrandx, with positive weights, hence lies betweenxandrhence lies above induction proves that all the iterative valuesx0, x1, ..lie inI.

10 Our error analysisthen can be applied to show that we have convergence, since everywhere inI g < < (x )0x0x a0x + a0x1x a0x + a00x 2a0x + 2ax0g(x )0x1rrfixed Point for gg(x)y = x1 > g (x) > 0 Fixed Point for g 1 < g (x) < 0 y = xrrg(x)slope = 1 Figure 2: Examples of Convergence for Functionsgwith Positive and Negative SlopeIf the derivative ofgis not positive onI, it is possible that the Iterations do not stay insideI,and hence the fact thatg is small enough onImay not suffice. However, once we assume thatr I(as before), whileg is small enough on a slightly larger interval, convergence can be Ias before, and, withI := [x0 2 , x0+ 2 ], c I |g (c)| <1, thenx0, x1, x2, ..converges need to prove that the iterationsx0, x1, ..stay inI : sinceg is small enough onI , ourprevious error analysis will show that the error decreases by at least a factor of per ,x0surely lies inI . Assume by induction thatx0, .. , xn 1all lie inI.