Lecture 2: Markov Decision Processes - David Silver

← Back to document page

Lecture 2: Markov Decision ProcessesLecture 2: Markov Decision ProcessesDavid SilverLecture 2: Markov Decision Processes1Markov Processes2Markov Reward Processes3Markov Decision Processes4Extensions to MDPsLecture 2: Markov Decision ProcessesMarkov ProcessesIntroductionIntroduction to MDPsMarkov Decision processesformally describe an environmentfor reinforcement learningWhere the environment isfully The currentstatecompletely characterises the processAlmost all RL problems can be formalised as MDPs, control primarily deals with continuous MDPsPartially observable problems can be converted into MDPsBandits are MDPs with one stateLecture 2: Markov Decision ProcessesMarkov ProcessesMarkov PropertyMarkov Property The future is independent of the past given the present DefinitionA stateStisMarkovif and only ifP[St+1|St] =P[St+1|S1.]

State Transition Matrix For a Markov state s and successor state s0, the state transition probability is de ned by P ss0= P S t+1 = s 0jS t = s State transition matrix Pde nes transition probabilities from all states s to all successor states s0, to P = from 2 6 4 P 11::: P 1n... P n1::: P nn 3 7 5 where each row of the matrix sums to 1.

  Transition, Probabilities, Transition probabilities