GANs Trained by a Two Time-Scale Update Rule ... - NeurIPS

1 GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Sepp Hochreiter LIT AI Lab & Institute of Bioinformatics, Johannes Kepler University Linz A-4040 Linz, Austria Abstract generative adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the con- vergence of GAN training has still not been proved. We propose a two Time-Scale Update rule (TTUR) for training GANs with stochastic gradient descent on ar- bitrary GAN loss functions.

2 TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the perfor- mance of GANs at image generation, we introduce the Fr chet Inception Distance . (FID) which captures the similarity of generated images to real ones better than the Inception Score.

3 In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN train- ing on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark. 1 Introduction generative adversarial networks (GANs) [16] have achieved outstanding results in generating realistic images [42, 31, 25, 1, 4] and producing text [21]. GANs can learn complex generative models for which maximum likelihood or a variational approximations are infeasible. Instead of the likelihood, a discriminator network serves as objective for the generative model, that is, the generator.

4 GAN. learning is a game between the generator, which constructs synthetic data from random variables, and the discriminator , which separates synthetic data from real world data. The generator's goal is to construct data in such a way that the discriminator cannot tell them apart from real world data. Thus, the discriminator tries to minimize the synthetic-real discrimination error while the generator tries to maximize this error. Since training GANs is a game and its solution is a Nash equilibrium, gradient descent may fail to converge [44, 16, 18].

5 Only local Nash equilibria are found, because gradient descent is a local optimization method. If there exists a local neighborhood around a point in parameter space where neither the generator nor the discriminator can unilaterally decrease their respective losses, then we call this point a local Nash equilibrium. To characterize the convergence properties of training general GANs is still an open challenge [17, 18]. For special GAN variants, convergence can be proved under certain assumptions [34, 20, 46]. A. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

6 5000 10000 15000. Iteration 0. 0 1 2 3 4 5. 10 10 10 10 10 10. Iteration Convergence of deterministic algorithm under different step sizes. Fig. 4. Convergence under noisy feedback (the biased case). 500 orig 1e-5 = = [ , , , ]. orig 1e-4. Flow 1 ( n = ). = = [ , , , ]. TTUR 1e-5 1e-4. Flow 2 ( n = ). = = [1, 1, 1, 1]. Flow 3 ( n = ). = = [5, 5, 5, 5]. 400 Flow 4 ( n = ). Flow 1 ( n = 1/n). Constant step size Diminishing step size Flow 2 ( = 1/n). n 300 Flow 3 ( n = 1/n). ||x(n) x*||. FID Flow 4 ( = 1/n). n 200 100. 0. 5000. 0 10000 15000. Iteration 0 50 100 150 200 250 10.

7 1 2. 10 10. 3. 10. 4. 10. 5. mini-batch x 1k Iteration Figure 1: Left: Original vs. TTUR GAN. Convergence under noisy feedback (the unbiased case). Fig. training 5. on CelebA. Zoomed-in Right: Figure convergence from behavior of Zhang 2007in[50]. the iterates Figure 4. which shows the distance of the parameter from the optimum for a one Time-Scale Update of a 4. node network flow problem. When the upper bounds on the errors ( , ) are small, the iterates V. S TOCHASTIC S TABILITY OF T WO T IME -S CALE. oscillate and repeatedly he convergence to a neighborhood is the bestreturn we to a neighborhood ofAthe can optimal solution LGORITHM (cf.)

8 Supplement U NDER Section ). N OISY F EEDBACK. However, whereas by using diminishing when step the upper sizes, bounds on the errors convergence In theare large, thesections, previous iterates typically we have diverge. applied the dual decom- obability one to the optimal points is made possible. position method to Problem (1) and devised the primal-dual ability of The Stochastic Algorithm: The Biased Case: algorithm, which is a single Time-Scale algorithm. As noted hat when the gradient estimation error is convergence biased, we proofs in Section I, there are many other decomposition methods.

9 Prerequisit for many is local stability In particular, [30] which the primal was shown for decomposition GANs is method by a useful hope to obtain almost sure convergence to the optimal s. Instead, we have shown Nagarajan and Kolter that provided that[39]. the for a min-max biased GAN setting. machinery However,with for problem Nagarajan coupled andvariables Kolter require [31];forand when asymptotically uniformly their proof either bounded, therather strong iterates and unrealistic return some of assumptions the variablesor a restriction are fixed,to a linear the restdiscriminator.

10 Of the problem may decouple into several subproblems. This naturally yields Recent ontraction region infinitely convergence often. proofs for GANs In this example, we hold for expectations over training samples or for the number multiple Time-Scale algorithms. It is also of great interest to that s (n) = (i,j) (n) and are uniformly bounded by a of examples going to infinity d positive value. We also assume that s (n) N (0, 1). [32, 38, 35, 2], thusthe examine do not consider stability ofmini-batch the multiplelearning which leads Time-Scale to algorithms in a stochastic gradient [47, 23, 36, 33].