Deep Reinforcement Learning for Flappy Bird

1 deep Reinforcement Learning forFlappy BirdKevin ChenAbstract Reinforcement Learning is essential for appli-cations where there is no single correct way to solve aproblem. In this project, we show that deep reinforcementlearning is very effective at Learning how to play the gameFlappy bird , despite the high-dimensional sensory agent is not given information about what the birdor pipes look like - it must learn these representationsand directly use the input and score to develop an optimalstrategy. Our agent uses a convolutional neural network toevaluate the Q-function for a variant of Q- Learning , andwe show that it is able to achieve super-human perfor-mance.

2 Furthermore, we discuss difficulties and potentialimprovements with deep Reinforcement INTRODUCTIONR einforcement Learning is useful when we need anagent to perform a task, but there is no single correct way of completing it. For example, how would oneprogram a robot to travel from one place to anotherand bring back food? It would be unrealistic to programevery move and step that it must take. Instead, it shouldlearn to make decisions under uncertainty and with veryhigh dimensional input (such as a camera) in order toreach the end goal.

3 This project focuses on a first stepin realizing goal of the project is to learn a policy to have anagent successfully play the game Flappy bird . FlappyBird is a game in which the player tries to keep the birdalive for as long as possible. The bird automatically fallstowards the ground by due to gravity, and if it hits theground, it dies and the game ends. The bird must alsonavigate through pipes. The pipes restrict the height ofthe bird to be within a certain specific range as the birdpasses through them. If the bird is too high or too low,it will crash into the pipe and die.

4 Therefore, the playermust time flaps/jumps properly to keep the bird aliveas it passes through these obstacles. The game score ismeasured by how many obstacles the bird successfullypasses through. Therefore, to get a high score, the playermust keep the bird alive for as long as possible as itencounters the an agent to successfully play the game isespecially challenging because our goal is to providethe agent with only pixel information and the agent is not provided with information regardingwhat the bird looks like, what the pipes look like, orwhere the bird and pipes are.

5 Instead, it must learn theserepresentations and interactions and be able to generalizedue to the very large state 1: Three screenshots of the game Flappy bird atthree different difficulties (easy, medium, hard) RELATEDWORKThe related work in this area is primarily by GoogleDeepmind. Mnih et al. are able to successfully trainagents to play the Atari 2600 games using deep rein-forcement Learning , surpassing human expert-level onmultiple games [1],[2]. These works inspired thisproject, which is heavily modeled after their use a deep Q-network (DQN) to evaluate the Q-function for Q- Learning and also use experience replayto de-correlate experiences.

6 Their approach is essentiallystate-of-the-art and was the main catalyst for deep re-inforcement Learning , after which many papers tried tomake improvements. The main strength is that they wereable to train an agent despite extremely high dimensionalinput (pixels) and no specification about intrinsic gameparameters. In fact, they are able to outperform a humanexpert on three out of seven Atari 2600 games . However,further improvements involve prioritizing experience re-play, more efficient training, and better stability whentraining. [2] tried to address the stability issues byclipping the loss to+1or 1, and by updating the targetnetwork once in every C updates to the DQN rather thanupdating the target network every METHODIn this section, we describe how the model is param-eterized and the general MDP FormulationTheactionsthat the agent can take are to flap (a= 1)or to do nothing and let the bird drop (a= 0).

7 Thestateis represented by a sequence of frames from the FlappyBird game as well as the recent actions that the playertook. Specifically, the state is the sequence shown in1st= (xt histLen+1,at histLen+1,..,xt 1,at 1,xt)(1) iLi( i) =Es,a ( );s [(r+ maxa Q(s ,a ; i 1) Q(s,a; i)) iQ(s,a; i)](2)Equation 1 wherestis the state at timet,xtis the pixelinput (or the frame or screen capture) at timet, andatisthe action taken at (orhistLen)is a hyperparameter that specifies how many of the mostrecent frames to keep track of. This is to reduce thestorage and state space compared to saving all framesand actions starting fromt= 1.

8 The reason for storingmultiplex s anda s rather than storing a single framexis because the agent needs temporal information to example, the agent cannot deduce the velocity ofthe bird from a single frame, but velocity is essential formaking a factorwas set to = Thetransition probabilitiesand therewardsare unknownto the agent. Since Q- Learning is model-free, we donot explicitly estimate the transition probabilities andrewards, but instead directly try to estimate the optimalQ-function. This is described further in the , we still must define the rewards intrinsic tothe game.

9 Ideally, the reward should essentially be thescore of the game. It starts out as 0 and every time thebird passes a pipe, the score increases by 1. However,this is potentially problematic in that the rewards will bevery sparse. Specifically, if the bird dies instantly at thestart of the game, the reward would be similar to if thebird died right before reaching the pipe. The performanceis clearly better if the bird survives up until the pipecompared to dying instantly. Therefore, adding a rewardfor staying alive encourages the agent to think simi-larly.

10 Without this additional reward, the agent shouldeventually realize this, but adding the reward, calledrewardAlive, speeds up the training process. In total,we have three rewards:rewardAlive,rewardPipe,andreward Dead. The agent getsrewardAliveforevery frame it stays alive,rewardPipefor successfullypassing a pipe, andrewardDeadfor Q-learningThe goal in Reinforcement Learning is always to maxi-mize the expected value of the total payoff (or expectedreturn). In Q- Learning , which is off-policy, we use theBellman equation as an iterative updateQi+1(s,a) =Es [r+ maxa Qi(s ,a )|s,a](3)wheres is the next state,ris the reward, is the envi-ronment, andQi(s,a)is the Q-function at theith itera-tion.