Dynamic optimization - michaelcarteronline.com

1 Chapter 7 Dynamic optimizationChapter 5 deals essentially with static optimization , that is optimal choice at a singlepoint of time. Many economic models involve optimization over time. While the sameprinciples of optimization apply to Dynamic models, new considerations arise. On theone hand, the repetitive nature of Dynamic models adds additional structure to themodel which can be exploited in analyzing the solution. On the other hand, manydynamic models have nofinite time horizon or are couched in continuous time, sothat the underlying space is infinite-dimensional. This requires a more sophisticatedtheory and additional solution techniques. Dynamic models are increasingly employedin economic theory and practice, and the student of economics needs to be familiarwith their analysis. This need not be seen as an unrewarding chore - the additionalcomplexity of Dynamic models adds to their interest, and many interesting examplescan be factor complicating the study of Dynamic optimization is the existence ofthree distinct approaches, all of which areusedinpractice.

2 Theclassicapproachisbased on the calculus of variations, a centuries-old extension of calculus to infinite-dimensional space. This was generalized under the stimulus of the space race in thelate 1950s to develop optimal control theory, the most common technique for dealingwith models in continuous time. The second principle approach, Dynamic program-ming, was developed at the same time, primarily to deal with optimization in discretetime. Dynamic programming has already been explored in some detail to illustratethe material of Chapter 2 (Example ). The third approach to Dynamic optimiza-tion extends the Lagrangean technique of static optimization to Dynamic , we call this the Lagrangean rigorous treatment of Dynamic optimization (especially optimal control theory)is quite difficult. Although many of the necessary prerequisites are contained in earlierchapters, some essential elements (such as integration) are missing.

3 The goals of thissupplementary chapter are therefore more modest. We aim to take advantage of thefoundation already developed, utilizing as much as possible the optimization theoryof Chapter 5. To this end, we start with the Lagrangean approach in Section 2,deriving the maximum principle for discrete time problems. We then (Section 3) extendthis by analogy to continuous time problems, stating the continous-time maximumprinciple and illustrating its use with several examples. Section 4 takes up the dynamicprogramming 7. Dynamic IntroductionThe basic intuition of Dynamic optimization can be illustrated with a simple exampleof intertemporal allocation. Suppose you embark on a two-day hiking trip withwunitsof food. Your problem is to decide how much food to consume on thefirst day, andhow much to save for the second day.

4 It is conventional to label thefirst period , letc0denote consumption on thefirst day andc1denote consumption onthe second day. The optimization problem ismaxc0,c1U(c0,c1)subject toc0+c1=wClearly, optimality requires that daily consumption be arranged so as to equalize themarginal utility of consumption on the two days, that isDc0U(c0,c1)=Dc1U(c0,c1)Otherwise, the intertemporal allocation of food could be rearranged so as to increase to-tal utility. Put differently, optimality requires that consumption in each period be suchthat marginal benefit equals marginal cost, where the marginal cost of consumption inperiod 0 is the consumption foregone in period 1. This is the fundamental intuitionof Dynamic optimization - optimality requires that resources be allocated over time insuch a way that there are no favorable opportunities for intertemporal , the utility function is assumed to beSeparableU(c0,c1)=u0(c0)+u1(c1)andStat ionaryU(c0,c1)=u(c0)+ u(c1)where represents the discount rate of future consumption (Example ).

5 Thenthe optimality condition isuI(c0)= uI(c1)Assuming thatuis concave, we can deduce thatc0>c1 <1 Consumption is higher in thefirst period if future consumption is is straightforward to extend this model in various ways. For example, if it ispossible to borrow and lend at interest rater, the two-period optimization problem ismaxc0,c1u(c0)+ u(c1)subject toc1=(1+r)(w c0)assuming separability and stationarity. Forming the LagrangeanL=u(c0)+ u(c1)+ Dc1 (1 +r)(w c0)ithefirst-order conditions for optimality areDc0L=uI(c0) (1 +r)=0Dc1L= uI(c1) =0 Eliminating , we conclude that optimality requires thatuI(c0)= uI(c1)(1 +r)( )CHAPTER 7. Dynamic OPTIMIZATION3 The left-hand side is the marginal benefit of consumption today. For an optimal alloca-tion, this must be equal to the marginal cost of consumption today, which is the interestforegone (1 +r) times the marginal benefit of consumption tomorrow, discounted atthe rate.

6 Alternatively, the optimality condition can be expressed asuI(c0) uI(c1)=1+r( )The quantity on the left hand side is the intertemporal marginal rate of quantity on the right can be thought of as the marginal rate of transformation,the rate at which savings in thefirst period can be transformed into consumption inthe second period. Assuminguis concave, we can deduce from ( ) thatc0>c1 (1 +r) <1 The balance of consumption between the two periods depends upon the interaction oftherateoftimepreference( ) and the interest log utilityu(c)=logc, show that the optimal allocation ofconsumption isc0=w1+ ,c1=(1+r) w1+ Note that optimal consumption in period 0 is independent of the interest rate (Drc0=0).Exercise (c)= c. Show thatDrc0< extend the model toTperiods, letctdenote consumption in periodtandwtthe remaining wealth at the beginning of (1+r)(w0 c0)w2=(1+r)(w1 c1)andsoondowntowT=(1+r)(wT 1 cT 1)wherew0denotes the initial wealth.

7 The optimal pattern of consumption throughtimessolvestheproblemmaxct,wt+1T 13t=0 tu(ct)subject towt=(1+r)(wt 1 ct 1),t=1,2,..,Twhich is a standard equality constrained optimization problem. Assigning multipliers( 1, 2,.. , T)totheTconstraints, the Lagrangean isL=T 13t=0 tu(ct) T3t=1 tpwt (1 +r)(wt 1 ct 1)QCHAPTER 7. Dynamic OPTIMIZATION4which can be rewritten asL=T 13t=0 tu(ct) T3t=1 twt+T3t=1 t(1 +r)(wt 1 ct 1)=T 13t=0 tu(ct) T3t=1 twt+T 13t=0 t+1(1 +r)(wt ct)=u(c0) 1(1 +r)(w0 c0)+T 13t=1 tu(ct) T 13t=1 twt+T 13t=1 t+1(1 +r)(wt ct)+ TwT=u(c0) 1(1 +r)(w0 c0)+T 13t=1p tu(ct) twt+ t+1(1 +r)(wt ct)Q+ TwT=u(c0) 1(1 +r)(w0 c0)+T 13t=1p tu(ct) t+1(1 +r)ct+D t+1(1 +r) tiwtQ+ TwTThefirst-order necessary conditions for optimality are (Corollary )Dc0L=uI(c0) 1(1 +r)=0 DctL= tuI(ct) t+1(1 +r)=0,t=1,2.

8 ,T 1 DwtL=(1+r) t+1 t=0,t=1,2,..,T 1 DwTL= T=0 Together, these equations imply tuI(ct)= t+1(1 +r)= t( )in every periodt=0,1,..,T 1andtherefore t+1uI(ct+1)= t+1( )Substituting ( ) in ( ) , we get tuI(ct)= t+1uI(ct+1)(1 +r)oruI(ct)= uI(ct+1)(1 +r),t=0,1,..,T 1( )which is identical to ( ), the optimality condition for the two-period problem. Anoptimal consumption plan requires that consumption be allocated through time so thatmarginal benefit of consumption in periodt(uI(ct)) is equal to its marginal cost, whichis the interest foregone (1 +r) times the marginal benefit of consumption tomorrowdiscounted at the rate . Again, we observe that whether consumption increases ordecreases through time depends upon the interaction of the rate of time preference and the concave, ( ) implies thatct>ct+1 (1 +r) <1 CHAPTER 7.

9 Dynamic OPTIMIZATION5 The choice of the level of consumption in each periodcthas two effects. First, itprovides contemporaneous utility in periodt. In addition, it determines the level ofwealth remaining,wt+1, to provide for consumption in future periods. The Lagrangemultiplier tassociated with the constraintwt=(1+r)(wt 1 ct 1)measures the shadow price or value of this wealthwtat the beginning of periodt.( ) implies that this shadow price is equal to tuI(ct). Additional wealth in periodtcan either be consumed or saved, and its value in these two uses must be , its value must be equal to the discounted marginal utility of consumptionin periodt. Not that thefinalfirst-order condition is T= 0. Any wealth left over isassumed to be alternative approach to the multi-period optimal savings problem utilizesa single intertemporal budget constraint(1 +r)Tc0+ +(1+r)2cT 2+(1+r)cT 1=(1+r)Tw0( )Derive( )and solve the problem of maximizingdiscounted total utility subjectmaxctT 13t=0 tu(ct)subject to this generalfinite horizon Dynamic optimization problem can be depicted ass0a0 f0(a0,s0)s1a1 f1(a1,s1)s2a2 f2(a2,s2).

10 AT 1 fT 1(aT 1,sT 1)st v(st)Starting from an initial states0, the decision maker chooses some actiona0 A0in thefirst period. This generates a contemporaneous return or benefitf0(a0,s0)andleadstoanewstates1, the transition to which is determined by some functiongs1=g0(a0,s0)In the second period, the decision maker choses another actiona1 A1, generatinga contemporaneous returnf(a1,s1)andleadingtoanewstates2to begin the thirdperiod, and so on forTperiods. In each periodt, the transition to the new state isdetermined by thetransition equationst+1=gt(at,st)The resulting sequence of choicesa0,a1,..,aT 1and implied transitions leaves a ter-minal statesT, the value of which isv(sT).Assuming separability, the objective of the decision maker is to choose that sequenceof actionsa0,a1,..,aT 1which maximizes the discounted sum of the contemporane-ous returnsft(at,st) plus the value of the terminal statev(sT).