FiLM: Visual Reasoning with a General …

1 FiLM: Visual Reasoning with a General conditioning LayerEthan Perez1,2, Florian Strub4, Harm de Vries1, Vincent Dumoulin1, Aaron Courville1,31 MILA, Universit e de Montr eal,2 Rice University,3 CIFAR Fellow,4 Univ. Lille, CNRS, Centrale Lille, Inria, UMR 9189 CRIStAL introduce a General -purpose conditioning method for neu-ral networks layers influence neural network computation via a sim-ple, feature-wise affine transformation based on conditioninginformation. We show that FiLM layers are highly effectivefor Visual Reasoning answering image-related questionswhich require a multi-step, high-level process a task whichhas proven difficult for standard deep learning methods thatdo not explicitly model Reasoning . Specifically, we show onvisual Reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features ina coherent manner, 3) are robust to ablations and architecturalmodifications, and 4) generalize well to challenging, new datafrom few examples or even IntroductionThe ability to reason about everyday Visual input is a fun-damental building block of human intelligence.

2 Some haveargued that for artificial agents to learn this complex, struc-tured process, it is necessary to build in aspects of reason-ing, such as compositionality (Hu et al. 2017; Johnson etal. 2017b) or relational computation (Santoro et al. 2017).However, if a model made from General -purpose compo-nents could learn to visually reason, such an architecturewould likely be more widely applicable across understand if such a General -purpose architecture ex-ists, we take advantage of the recently proposed CLEVR dataset (Johnson et al. 2017a) that tests Visual Reasoning viaquestion answering. Examples from CLEVR are shown inFigure 1. Visual question answering, the General task of ask-ing questions about images, has its own line of datasets (Ma-linowski and Fritz 2014; Geman et al. 2015; Antol et ) which generally focus on asking a diverse set ofsimpler questions on images, often answerable in a singleglance. From these datasets, a number of effective, General -purpose deep learning models have emerged for Visual ques-tion answering (Malinowski, Rohrbach, and Fritz 2015;Yang et al.)

3 2016; Lu et al. 2016; Anderson et al. 2017). How-ever, tests on CLEVR show that these General deep learningapproaches struggle to learn structured, multi-step reason-ing (Johnson et al. 2017a). In particular, these methods tendCopyrightc 2018, Association for the Advancement of ArtificialIntelligence ( ). All rights reserved.(a)Q:What number ofcylinders are small pur-ple things or yellow rubberthings?A:2(b)Q:What color is theother object that is the sameshape as the large brownmatte thing?A:BrownFigure 1: CLEVR examples and FiLM model exploit biases in the data rather than capture complex un-derlying structure behind Reasoning (Goyal et al. 2017).In this work, we show that a General model architecturecan achieve strong Visual Reasoning with a method we intro-duce asFiLM:Feature-wiseLinearModulation. A FiLMlayer carries out a simple, feature-wise affine transformationon a neural network s intermediate features, conditioned onan arbitrary input. In the case of Visual Reasoning , FiLM lay-ers enable a Recurrent Neural Network (RNN) over an inputquestion to influence Convolutional Neural Network (CNN)computation over an image.

4 This process adaptively and rad-ically alters the CNN s behavior as a function of the inputquestion, allowing the overall model to carry out a varietyof Reasoning tasks, ranging from counting to comparing, forexample. FiLM can be thought of as a generalization of Con-ditional Normalization, which has proven highly successfulfor image stylization (Dumoulin, Shlens, and Kudlur 2017;Ghiasi et al. 2017; Huang and Belongie 2017), speech recog-nition (Kim, Song, and Bengio 2017), and Visual questionanswering (de Vries et al. 2017), demonstrating FiLM sbroad this paper, which expands upon a shorter report (Perezet al. 2017), our key contribution is that we show FiLM isa strong conditioning method by showing the following onvisual Reasoning tasks:1. FiLM models achieve state-of-the-art across a variety ofvisual Reasoning tasks, often by significant FiLM operates in a coherent manner. It learns a complex,underlying structure and manipulates the conditioned net-work s features in a selective manner.

5 It also enables [ ] 18 Dec 2017 CNN to properly localize question-referenced FiLM is robust; many FiLM model ablations still outper-form prior state-of-the-art. Notably, we find there is noclose link between normalization and the success of a con-ditioned affine transformation, a previously untouched as-sumption. Thus, we relax the conditions under which thismethod can be FiLM models learn from little data to generalize to morecomplex and/or substantially different data than seen dur-ing training. We also introduce a novel FiLM-based zero-shot generalization method that further improves and val-idates FiLM s generalization MethodOur model processes the question-image input using FiLM,illustrated in Figure 2. We start by explaining FiLM and thendescribe our particular model for Visual Feature-wise Linear ModulationFiLM learns to adaptively influence the output of a neuralnetwork by applying an affine transformation, or FiLM, tothe network s intermediate features, based on some formally, FiLM learns functionsfandhwhich output i,cand i,cas a function of inputxi: i,c=fc(xi) i,c=hc(xi),(1)where i,cand i,cmodulate a neural network s activationsFi,c, whose subscripts refer to theithinput scthfeature orfeature map, via a feature-wise affine transformation:FiLM(Fi,c| i,c, i,c) = i,cFi,c+ i,c.

6 (2)fandhcan be arbitrary functions such as neural of a target neural network s processing can bebased on the same input to that neural network or some otherinput, as in the case of multi-modal or conditional tasks. ForCNNs,fandhthus modulate the per-feature-map distribu-tion of activations based onxi, agnostic to spatial practice, it is easier to refer tofandhas a single func-tion that outputs one( , )vector, since, for example, itis often beneficial to share parameters acrossfandhformore efficient learning. We refer to this single function asthe FiLM generator. We also refer to the network to whichFiLM layers are applied as the Feature-wise Linearly Mod-ulated network, the FiLM-ed layers empower the FiLM generator to manipulatefeature maps of a target, FiLM-ed network by scaling themup or down, negating them, shutting them off, selectivelythresholding them (when followed by a ReLU), and feature map is conditioned independently, giving theFiLM generator moderately fine-grained control over acti-vations at each FiLM FiLM only requires two parameters per modulated fea-ture map, it is a scalable and computationally efficient con-ditioning method.

7 In particular, FiLM has a computationalcost that does not scale with the image 2: A single FiLM layer for a CNN. The dot signifiesa Hadamard product. Various combinations of and canmodulate individual feature maps in a variety of ModelOur FiLM model consists of a FiLM-generating linguis-tic pipeline and a FiLM-ed Visual pipeline as depicted inFigure 3. The FiLM generator processes a questionxius-ing a Gated Recurrent Unit (GRU) network (Chung et ) with 4096 hidden units that takes in learned, 200-dimensional word embeddings. The final GRU hidden stateis a question embedding, from which the model predicts( ni, , ni, )for eachnthresidual block via affine Visual pipeline extracts 12814 14image featuremaps from a resized,224 224image input using eithera CNN trained from scratch or a fixed, pre-trained featureextractor with a learned layer of3 3convolutions. TheCNN trained from scratch consists of 4 layers with 1284 4kernels each, ReLU activations, and batch normalization,similar to prior work on CLEVR (Santoro et al.)

8 2017). Thefixed feature extractor outputs theconv4layer of a ResNet-101 (He et al. 2016) pre-trained on ImageNet (Russakovskyet al. 2015) to match prior work on CLEVR (Johnson et ; 2017b). Image features are processed by several 4 for our model FiLM-ed residual blocks (ResBlocks) with 128 feature maps and a final classifier. The classifierconsists of a1 1convolution to 512 feature maps, globalmax-pooling, and a two-layer MLP with 1024 hidden unitsthat outputs a softmax distribution over final FiLM-ed ResBlock starts with a1 1convolu-tion followed by one3 3convolution with an architec-ture as depicted in Figure 3. We turn the parameters of batchnormalization layers that immediately precede FiLM layersoff. Drawing from prior work on CLEVR (Hu et al. 2017;Santoro et al. 2017) and Visual Reasoning (Watters et ), we concatenate two coordinate feature maps indi-cating relative x and y spatial position (scaled from 1to1) with the image features, each ResBlock s input, and theclassifier s input to facilitate spatial train our model end-to-end from scratch withFigure 3: The FiLM generator (left), FiLM-ed network (mid-dle), and residual block architecture (right) of our (Kingma and Ba 2015) (learning rate3e 4), weightdecay (1e 5), batch size 64, and batch normalization andReLU throughout FiLM-ed network.

9 Our model uses onlyimage-question-answer triplets from the training set with -out data augmentation. We employ early stopping based onvalidation accuracy, training for 80 epochs maximum. Fur-ther model details are in the appendix. Empirically, we foundFiLM had a large capacity, so many architectural and hyper-parameter choices were for added stress that our model reliessolelyon feature-wiseaffine conditioning to use question information influence thevisual pipeline behavior to answer questions. This approachdiffers from classical Visual question answering pipelineswhich fuse image and language information into a singleembedding via element-wise product, concatenation, atten-tion, and/or more advanced methods (Yang et al. 2016;Lu et al. 2016; Anderson et al. 2017).3 Related WorkFiLM can be viewed as a generalization of Conditional Nor-malization (CN) methods. CN replaces the parameters of thefeature-wise affine transformation typical in normalizationlayers, as introduced originally (Ioffe and Szegedy 2015), with a learned function of some conditioning forms of CN have proven highly effective across anumber of domains: Conditional Instance Norm (Dumoulin,Shlens, and Kudlur 2017; Ghiasi et al.)

10 2017) and AdaptiveInstance Norm (Huang and Belongie 2017) for image styl-ization, Dynamic Layer Norm for speech recognition (Kim,Song, and Bengio 2017), and Conditional Batch Norm forgeneral Visual question answering on complex scenes suchas VQA and GuessWhat?! (de Vries et al. 2017). This workcomplements our own, as we seek to show that feature-wiseaffine conditioning is effective for multi-step Reasoning andunderstand the underlying mechanism behind its , prior work in CN has not examined whetherthe affine transformation must be placed directly after nor-malization. Rather, prior work includes normalization in themethod name for instructive purposes or due to implemen-tation details. We investigate the connection between FiLMand normalization, finding it not strictly necessary for theaffine transformation to occur directly after , we provide a unified framework for all of these meth-ods through FiLM, as well as a normalization-free relaxationof this approach which can be more broadly CN, there are many connections between FiLMand other conditioning methods.