Soft Actor-Critic: Off-Policy Maximum Entropy Deep ...

1 Soft Actor-Critic: Off-Policy Maximum Entropy Deep ReinforcementLearning with a Stochastic ActorTuomas Haarnoja1 Aurick Zhou1 Pieter Abbeel1 Sergey Levine1 AbstractModel-free deep reinforcement learning (RL) al-gorithms have been demonstrated on a range ofchallenging decision making and control , these methods typically suffer from twomajor challenges: very high sample complexityand brittle convergence properties, which necessi-tate meticulous hyperparameter tuning. Both ofthese challenges severely limit the applicabilityof such methods to complex, real-world this paper, we propose soft actor-critic, an Off-Policy actor-critic deep RL algorithm based on themaximum Entropy reinforcement learning frame-work. In this framework, the actor aims to maxi-mize expected reward while also maximizing en-tropy.

2 That is, to succeed at the task while actingas randomly as possible. Prior deep RL methodsbased on this framework have been formulatedas Q-learning methods. By combining off-policyupdates with a stable stochastic actor-critic formu-lation, our method achieves state-of-the-art per-formance on a range of continuous control bench-mark tasks, outperforming prior on-policy andoff-policy methods. Furthermore, we demonstratethat, in contrast to other Off-Policy algorithms, ourapproach is very stable, achieving very similarperformance across different random IntroductionModel-free deep reinforcement learning (RL) algorithmshave been applied in a range of challenging domains, fromgames (Mnih et al., 2013; Silver et al., 2016) to roboticcontrol (Schulman et al., 2015). The combination of RLand high-capacity function approximators such as neuralnetworks holds the promise of automating a wide range ofdecision making and control tasks, but widespread adoption1 Berkeley Artificial Intelligence Research, University of Cal-ifornia, Berkeley, USA.

3 Correspondence to: Tuomas these methods in real-world domains has been hamperedby two major challenges. First, model-free deep RL meth-ods are notoriously expensive in terms of their sample com-plexity. Even relatively simple tasks can require millions ofsteps of data collection, and complex behaviors with high-dimensional observations might need substantially , these methods are often brittle with respect to theirhyperparameters: learning rates, exploration constants, andother settings must be set carefully for different problemsettings to achieve good results. Both of these challengesseverely limit the applicability of model-free deep RL toreal-world cause for the poor sample efficiency of deep RL meth-ods is on-policy learning: some of the most commonly useddeep RL algorithms, such as TRPO (Schulman et al.)

4 , 2015),PPO (Schulman et al., 2017b) or A3C (Mnih et al., 2016),require new samples to be collected for each gradient quickly becomes extravagantly expensive, as the num-ber of gradient steps and samples per step needed to learnan effective policy increases with task complexity. Off-Policy algorithms aim to reuse past experience. This is notdirectly feasible with conventional policy gradient formula-tions, but is relatively straightforward for Q-learning basedmethods (Mnih et al., 2015). Unfortunately, the combina-tion of Off-Policy learning and high-dimensional, nonlinearfunction approximation with neural networks presents a ma-jor challenge for stability and convergence (Bhatnagar et al.,2009). This challenge is further exacerbated in continuousstate and action spaces, where a separate actor network isoften used to perform the maximization in Q-learning.

5 Acommonly used algorithm in such settings, deep determinis-tic policy gradient (DDPG) (Lillicrap et al., 2015), providesfor sample-efficient learning but is notoriously challengingto use due to its extreme brittleness and hyperparametersensitivity (Duan et al., 2016; henderson et al., 2017 ).We explore how to design an efficient and stable model-free deep RL algorithm for continuous state and actionspaces. To that end, we draw on the Maximum entropyframework, which augments the standard Maximum rewardreinforcement learning objective with an Entropy maximiza-tion term (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., [ ] 8 Aug 2018 Soft Actor-Critic2012; Fox et al., 2016; Haarnoja et al., 2017 ). Maximum en-tropy reinforcement learning alters the RL objective, thoughthe original objective can be recovered using a tempera-ture parameter (Haarnoja et al.)

6 , 2017 ). More importantly,the Maximum Entropy formulation provides a substantialimprovement in exploration and robustness: as discussedby Ziebart (2010), Maximum Entropy policies are robustin the face of model and estimation errors, and as demon-strated by (Haarnoja et al., 2017 ), they improve explorationby acquiring diverse behaviors. Prior work has proposedmodel-free deep RL algorithms that perform on-policy learn-ing with Entropy maximization (O Donoghue et al., 2016),as well as Off-Policy methods based on soft Q-learning andits variants (Schulman et al., 2017a; Nachum et al., 2017a;Haarnoja et al., 2017 ). However, the on-policy variants suf-fer from poor sample complexity for the reasons discussedabove, while the Off-Policy variants require complex approx-imate inference procedures in continuous action this paper, we demonstrate that we can devise an Off-Policy Maximum Entropy actor-critic algorithm, which wecall soft actor-critic (SAC), which provides for both sample-efficient learning and stability.

7 This algorithm extends read-ily to very complex, high-dimensional tasks, such as theHumanoid benchmark (Duan et al., 2016) with 21 actiondimensions, where Off-Policy methods such as DDPG typi-cally struggle to obtain good results (Gu et al., 2016). SACalso avoids the complexity and potential instability associ-ated with approximate inference in prior Off-Policy maxi-mum Entropy algorithms based on soft Q-learning (Haarnojaet al., 2017 ). We present a convergence proof for policyiteration in the Maximum Entropy framework, and then in-troduce a new algorithm based on an approximation to thisprocedure that can be practically implemented with deepneural networks, which we call soft actor-critic. We presentempirical results that show that soft actor-critic attains asubstantial improvement in both performance and sampleefficiency over both Off-Policy and on-policy prior also compare to twin delayed deep deterministic (TD3)policy gradient algorithm (Fujimoto et al.)

8 , 2018), which isa concurrent work that proposes a deterministic algorithmthat substantially improves on Related WorkOur soft actor-critic algorithm incorporates three key in-gredients: an actor-critic architecture with separate policyand value function networks, an Off-Policy formulation thatenables reuse of previously collected data for efficiency, andentropy maximization to enable stability and review prior works that draw on some of these ideas inthis section. Actor-critic algorithms are typically derivedstarting from policy iteration, which alternates betweenpol-icy evaluation computing the value function for a policy andpolicy improvement using the value function to obtaina better policy (Barto et al., 1983; Sutton & Barto, 1998). Inlarge-scale reinforcement learning problems, it is typicallyimpractical to run either of these steps to convergence, andinstead the value function and policy are optimized this case, the policy is referred to as the actor, and thevalue function as the critic.

9 Many actor-critic algorithmsbuild on the standard, on-policy policy gradient formulationto update the actor (Peters & Schaal, 2008), and many ofthem also consider the Entropy of the policy, but instead ofmaximizing the Entropy , they use it as an regularizer (Schul-man et al., 2017b; 2015; Mnih et al., 2016; Gruslys et al., 2017 ). On-policy training tends to improve stability butresults in poor sample have been efforts to increase the sample efficiencywhile retaining robustness by incorporating Off-Policy sam-ples and by using higher order variance reduction tech-niques (O Donoghue et al., 2016; Gu et al., 2016). How-ever, fully Off-Policy algorithms still attain better effi-ciency. A particularly popular Off-Policy actor-critic method,DDPG (Lillicrap et al., 2015), which is a deep variant of thedeterministic policy gradient (Silver et al.)

10 , 2014) algorithm,uses a Q-function estimator to enable Off-Policy learning,and a deterministic actor that maximizes this such, this method can be viewed both as a determinis-tic actor-critic algorithm and an approximate Q-learningalgorithm. Unfortunately, the interplay between the deter-ministic actor network and the Q-function typically makesDDPG extremely difficult to stabilize and brittle to hyperpa-rameter settings (Duan et al., 2016; henderson et al., 2017 ).As a consequence, it is difficult to extend DDPG to complex,high-dimensional tasks, and on-policy policy gradient meth-ods still tend to produce the best results in such settings (Guet al., 2016). Our method instead combines Off-Policy actor-critic training with astochasticactor, and further aims tomaximize the Entropy of this actor with an Entropy maxi-mization objective.