Auxiliary Deep Generative Models - arXiv

1 Auxiliary deep Generative ModelsLars Maal Kaae S ren Kaae S of Applied Mathematics and Computer Science, Technical University of Denmark2 Bioinformatics Centre, Department of Biology, University of CopenhagenAbstractDeep Generative Models parameterized by neu-ral networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep genera-tive Models with Auxiliary variables which im-proves the variational approximation. The aux-iliary variables leave the Generative model un-changed but make the variational distributionmore expressive. Inspired by the structure of theauxiliary variable we also propose a model withtwo stochastic layers and skip connections. Ourfindings suggest that more expressive and prop-erly specified deep Generative Models convergefaster with better results.

2 We show state-of-the-art performance within semi-supervised learningon MNIST, SVHN and NORB IntroductionStochastic backpropagation, deep neural networks and ap-proximate Bayesian inference have made deep generativemodels practical for large scale problems (Kingma, 2013;Rezende et al., 2014), but typically they assume a meanfield latent distribution where all latent variables are in-dependent. This assumption might result in Models thatare incapable of capturing all dependencies in the data. Inthis paper we show that deep Generative Models with moreexpressive variational distributions are easier to optimizeand have better performance. We increase the flexibility ofthe model by introducing Auxiliary variables (Agakov andBarber, 2004) allowing for more complex latent distribu-tions.

3 We demonstrate the benefits of the increased flexibil-ity by achieving state-of-the-art performance in the semi-supervised setting for the MNIST (LeCun et al., 1998),Proceedings of the33rdInternational Conference on MachineLearning, New York, NY, USA, 2016. JMLR: W&CP volume48. Copyright 2016 by the author(s).SVHN (Netzer et al., 2011) and NORB (LeCun et al., 2004) there have been significant improvements withinthe semi-supervised classification tasks. Kingma et al.(2014) introduced a deep Generative model for semi-supervised learning by modeling the joint distribution overdata and labels. This model is difficult to train end-to-endwith more than one layer of stochastic latent variables, butcoupled with a pretrained feature extractor it performs the Ladder network (Rasmus et al.)

4 , 2015; Valpola,2014) and virtual adversarial training (VAT) (Miyato et al.,2015) have improved the performance further with end-to-end this paper we train deep Generative Models with mul-tiple stochastic layers. TheAuxiliary deep GenerativeModels (ADGM)utilize an extra set of Auxiliary latent vari-ables to increase the flexibility of the variational distribu-tion (cf. Sec. ). We also introduce a slight change tothe ADGM, a 2-layered stochastic model with skip con-nections, theSkip deep Generative model (SDGM)( ). Both Models are trainable end-to-end and offerstate-of-the-art performance when compared to other semi-supervised methods. In the paper we first introduce toy datato demonstrate that:(i) The Auxiliary variable Models can fit complex la-tent distributions and thereby improve the variationallower bound (cf.

5 Sec. and ).(ii) By fitting a complex half-moon dataset using only sixlabeled data points the ADGM utilizes the data mani-fold for semi-supervised classification (cf. Sec. ).For the benchmark datasets we show (cf. Sec. ):(iii) State-of-the-art results on several semi-supervisedclassification tasks.(iv) That multi-layered deep Generative Models for semi-supervised learning are trainable end-to-end withoutpre-training or feature [ ] 16 Jun 2016 Auxiliary deep Generative Models2. Auxiliary deep Generative modelsRecently Kingma (2013); Rezende et al. (2014) have cou-pled the approach of variational inference with deep learn-ing giving rise to powerful probabilistic Models constructedby an inference neural networkq(z|x)and a generativeneural networkp(x|z). This approach can be perceived asa variational equivalent to the deep auto-encoder, in whichq(z|x)acts as the encoder andp(x|z)the decoder.

6 How-ever, the difference is that these Models ensures efficientinference over continuous distributions in the latent spacezwith automatic relevance determination and regulariza-tion due to the KL-divergence. Furthermore, the gradientsof the variational upper bound are easily defined by back-propagation through the network(s). To keep the computa-tional requirements low the variational distributionq(z|x)is usually chosen to be a diagonal Gaussian, limiting theexpressive power of the inference this paper we propose a variational Auxiliary vari-able approach (Agakov and Barber, 2004) to improvethe variational distribution: The Generative model is ex-tended with variablesatop(x,z,a)such that the originalmodel is invariant to marginalization overa:p(x,z,a) =p(a|x,z)p(x,z). In the variational distribution, on theother hand,ais used such that marginalq(z|x) = q(z|a,x)p(a|x)dais a general non-Gaussian hierarchical specification allows the latent variables tobe correlated througha, while maintaining the computa-tional efficiency of fully factorized Models (cf.)

7 Fig. 1). InSec. we demonstrate the expressive power of the infer-ence model by fitting a complex multimodal Variational auto-encoderThe variational auto-encoder (VAE) has recently been in-troduced as a powerful method for unsupervised a latent variable Generative modelp (x|z)for dataxis parameterized by a deep neural network with parameters . The parameters are inferred by maximizing the varia-tional lower bound of the likelihood iUVAE(xi)withlogp(x) = log zp(x,z)dz Eq (z|x)[logp (x|z)p (z)q (z|x)](1) UVAE(x).The inference modelq (z|x)is parameterized by a seconddeep neural network. The inference and Generative param-eters, and , are jointly trained by optimizing Eq. 1 withstochastic gradient ascent, where we use the reparameter-ization trick for backpropagation through the Gaussian la-tent variables (Kingma, 2013; Rezende et al.

8 , 2014).yzax(a) Generative (b) Inference graphical model of the ADGM for semi-supervised learning. The incoming joint connections to each vari-able are deep neural networks with parameters and . Auxiliary variablesWe propose to extend the variational distribution with aux-iliary variablesa:q(a,z|x) =q(z|a,x)q(a|x)such that themarginal distributionq(z|x)can fit more complicated pos-teriorsp(z|x). In order to have an unchanged generativemodel,p(x|z), it is required that the joint modep(x,z,a)gives back the originalp(x,z)under marginalization overa, thusp(x,z,a) =p(a|x,z)p(x,z). Auxiliary variablesare used in the EM algorithm and Gibbs sampling andhave previously been considered for variational learningby Agakov and Barber (2004). Concurrent with this workRanganath et al. (2015) have proposed to make the param-eters of the variational distribution stochastic, which leadsto a similar model .

9 It is important to note that in ordernot to fall back to the original VAE model one has to re-quirep(a|x,z)6=p(a), see Agakov and Barber (2004) andApp. A. The Auxiliary VAE lower bound becomeslogp(x) = log a zp(x,a,z)dadz Eq (a,z|x)[logp (a|z,x)p (x|z)p(z)q (a|x)q (z|a,x)](2) UAVAE(x).withp (a|z,x)andq (a|x)diagonal Gaussian distribu-tions parameterized by deep neural Semi-supervised learningThe main focus of this paper is to use the Auxiliary ap-proach to build semi-supervised Models that learn clas-sifiers from labeled and unlabeled encom-pass the class information we introduce an extra la-tent variabley. The Generative modelPis defined asp(y)p(z)p (a|z,y,x)p (x|y,z)(cf. Fig. 1a):p(z) =N(z|0,I),(3)p(y) =Cat(y| ),(4)p (a|z,y,x) =f(a;z,y,x, ),(5)p (x|z,y) =f(x;z,y, ),(6) Auxiliary deep Generative Modelswherea,y,zare the Auxiliary variable, class label, and la-tent features, respectively.

10 Cat( )is a multinomial distribu-tion, whereyis treated as a latent variable for the unlabeleddata points. In this study we only experimented with cate-gorical labels, however the method applies to other distri-butions for the latent (x;z,y, )is iid categori-cal or Gaussian for discrete and continuous ( )are deep neural networks with parameters . The in-ference model is defined asq (a|x)q (z|a,y,x)q (y|a,x)(cf. Fig. 1b):q (a|x) =N(a| (x),diag( 2 (x))),(7)q (y|a,x) =Cat(y| (a,x)),(8)q (z|a,y,x) =N(z| (a,y,x),diag( 2 (a,y,x))).(9)In order to model Gaussian distributionsp (a|z,y,x),p (x|z,y),q (a|x)andq (z|a,y,x)we define two sepa-rate outputs from the top deterministic layer in each deepneural network, ( )andlog 2 ( ). From these out-puts we are able to approximate the expectationsEby ap-plying the reparameterization key point of the ADGM is that the Auxiliary unitaintroduce a latent feature extractor to the inference modelgiving a richer mapping betweenxandy.