FiLM: Visual Reasoning with a General Conditioning Layer

1 FiLM: Visual Reasoning with a General Conditioning LayerEthan Perez1,2, Florian Strub4, Harm de Vries1, Vincent Dumoulin1, Aaron Courville1,31 MILA, Universit e de Montr eal,2 Rice University,3 CIFAR Fellow,4 Univ. Lille, CNRS, Centrale Lille, Inria, UMR 9189 CRIStAL introduce a General -purpose Conditioning method for neu-ral networks layers influence neural network computation via a sim-ple, feature-wise affine transformation based on conditioninginformation. We show that FiLM layers are highly effectivefor Visual Reasoning answering image-related questionswhich require a multi-step, high-level process a task whichhas proven difficult for standard deep learning methods thatdo not explicitly model Reasoning .

2 Specifically, we show onvisual Reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features ina coherent manner, 3) are robust to ablations and architecturalmodifications, and 4) generalize well to challenging, new datafrom few examples or even IntroductionThe ability to reason about everyday Visual input is a fun-damental building block of human intelligence. Some haveargued that for artificial agents to learn this complex, struc-tured process, it is necessary to build in aspects of reason-ing, such as compositionality (Hu et al.

3 2017; Johnson etal. 2017b) or relational computation (Santoro et al. 2017).However, if a model made from General -purpose compo-nents could learn to visually reason, such an architecturewould likely be more widely applicable across understand if such a General -purpose architecture ex-ists, we take advantage of the recently proposed CLEVR dataset (Johnson et al. 2017a) that tests Visual Reasoning viaquestion answering. Examples from CLEVR are shown inFigure 1. Visual question answering, the General task of ask-ing questions about images, has its own line of datasets (Ma-linowski and Fritz 2014; Geman et al.

4 2015; Antol et ) which generally focus on asking a diverse set ofsimpler questions on images, often answerable in a singleglance. From these datasets, a number of effective, General -purpose deep learning models have emerged for Visual ques-tion answering (Malinowski, Rohrbach, and Fritz 2015;Yang et al. 2016; Lu et al. 2016; Anderson et al. 2017). How-ever, tests on CLEVR show that these General deep learningapproaches struggle to learn structured, multi-step reason-ing (Johnson et al. 2017a). In particular, these methods tendCopyrightc 2018, Association for the Advancement of ArtificialIntelligence ( ).

5 All rights reserved.(a)Q:What number ofcylinders are small pur-ple things or yellow rubberthings?A:2(b)Q:What color is theother object that is the sameshape as the large brownmatte thing?A:BrownFigure 1: CLEVR examples and FiLM model exploit biases in the data rather than capture complex un-derlying structure behind Reasoning (Goyal et al. 2017).In this work, we show that a General model architecturecan achieve strong Visual Reasoning with a method we intro-duce asFiLM:Feature-wiseLinearModulation. A FiLMlayer carries out a simple, feature-wise affine transformationon a neural network s intermediate features, conditioned onan arbitrary input.

6 In the case of Visual Reasoning , FiLM lay-ers enable a Recurrent Neural Network (RNN) over an inputquestion to influence Convolutional Neural Network (CNN)computation over an image. This process adaptively and rad-ically alters the CNN s behavior as a function of the inputquestion, allowing the overall model to carry out a varietyof Reasoning tasks, ranging from counting to comparing, forexample. FiLM can be thought of as a generalization of Con-ditional Normalization, which has proven highly successfulfor image stylization (Dumoulin, Shlens, and Kudlur 2017;Ghiasi et al.)

7 2017; Huang and Belongie 2017), speech recog-nition (Kim, Song, and Bengio 2017), and Visual questionanswering (de Vries et al. 2017), demonstrating FiLM sbroad this paper, which expands upon a shorter report (Perezet al. 2017), our key contribution is that we show FiLM isa strong Conditioning method by showing the following onvisual Reasoning tasks:1. FiLM models achieve state-of-the-art across a variety ofvisual Reasoning tasks, often by significant FiLM operates in a coherent manner. It learns a complex,underlying structure and manipulates the conditioned net-work s features in a selective manner.

8 It also enables [ ] 18 Dec 2017 CNN to properly localize question-referenced FiLM is robust; many FiLM model ablations still outper-form prior state-of-the-art. Notably, we find there is noclose link between normalization and the success of a con-ditioned affine transformation, a previously untouched as-sumption. Thus, we relax the conditions under which thismethod can be FiLM models learn from little data to generalize to morecomplex and/or substantially different data than seen dur-ing training. We also introduce a novel FiLM-based zero-shot generalization method that further improves and val-idates FiLM s generalization MethodOur model processes the question-image input using FiLM,illustrated in Figure 2.

9 We start by explaining FiLM and thendescribe our particular model for Visual Feature-wise Linear ModulationFiLM learns to adaptively influence the output of a neuralnetwork by applying an affine transformation, or FiLM, tothe network s intermediate features, based on some formally, FiLM learns functionsfandhwhich output i,cand i,cas a function of inputxi: i,c=fc(xi) i,c=hc(xi),(1)where i,cand i,cmodulate a neural network s activationsFi,c, whose subscripts refer to theithinput scthfeature orfeature map, via a feature-wise affine transformation:FiLM(Fi,c| i,c, i,c) = i,cFi,c+ i,c.

10 (2)fandhcan be arbitrary functions such as neural of a target neural network s processing can bebased on the same input to that neural network or some otherinput, as in the case of multi-modal or conditional tasks. ForCNNs,fandhthus modulate the per-feature-map distribu-tion of activations based onxi, agnostic to spatial practice, it is easier to refer tofandhas a single func-tion that outputs one( , )vector, since, for example, itis often beneficial to share parameters acrossfandhformore efficient learning. We refer to this single function asthe FiLM generator.