SlowFast Networks for Video Recognition

1 SlowFast Networks for Video Recognition Christoph Feichtenhofer Haoqi Fan Jitendra Malik Kaiming He Facebook AI Research (FAIR). Abstract We present SlowFast Networks for Video Recognition . Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast path- C T C T. C T. way, operating at high frame rate, to capture motion at prediction Low frame rate H,W. T. fine temporal resolution. The Fast pathway can be made C. very lightweight by reducing its channel capacity, yet can learn useful temporal information for Video Recognition .

2 Our models achieve strong performance for both action T. T. C. classification and detection in Video , and large improve- T C. High frame rate C. ments are pin-pointed as contributions by our SlowFast con- cept. We report state-of-the-art accuracy on major Video Figure 1. A SlowFast network has a low frame rate, low temporal Recognition benchmarks, Kinetics, Charades and AVA. Code resolution Slow pathway and a high frame rate, higher temporal resolution Fast pathway. The Fast pathway is lightweight by using has been made available at: a fraction ( , , 1/8) of channels.

3 Lateral connections fuse them. facebookresearch/ SlowFast . For example, waving hands do not change their identity as 1. Introduction hands over the span of the waving action, and a person It is customary in the Recognition of images I(x, y) to is always in the person category even though he/she can treat the two spatial dimensions x and y symmetrically. This transit from walking to running. So the Recognition of the cat- is justified by the statistics of natural images, which are to egorical semantics (as well as their colors, textures, lighting a first approximation isotropic all orientations are equally etc.)

4 Can be refreshed relatively slowly. On the other hand, likely and shift-invariant [41, 26]. But what about Video the motion being performed can evolve much faster than signals I(x, y, t)? Motion is the spatiotemporal counterpart their subject identities, such as clapping, waving, shaking, of orientation [2], but all spatiotemporal orientations are walking, or jumping. It can be desired to use fast refreshing not equally likely. Slow motions are more likely than fast frames (high temporal resolution) to effectively model the motions (indeed most of the world we see is at rest at a given potentially fast changing motion.)

5 Moment) and this has been exploited in Bayesian accounts of Based on this intuition, we present a two-pathway how humans perceive motion stimuli [58]. For example, if SlowFast model for Video Recognition (Fig. 1). One path- we see a moving edge in isolation, we perceive it as moving way is designed to capture semantic information that can be perpendicular to itself, even though in principle it could given by images or a few sparse frames, and it operates at also have an arbitrary component of movement tangential to low frame rates and slow refreshing speed. In contrast, the itself (the aperture problem in optical flow).

6 This percept is other pathway is responsible for capturing rapidly changing rational if the prior favors slow movements. motion, by operating at fast refreshing speed and high tem- If all spatiotemporal orientations are not equally likely, poral resolution. Despite its high temporal rate, this pathway then there is no reason for us to treat space and time sym- is made very lightweight, , 20% of total computation. metrically, as is implicit in approaches to Video Recognition This is because this pathway is designed to have fewer chan- based on spatiotemporal convolutions [49, 5].

7 We might nels and weaker ability to process spatial information, while instead factor the architecture to treat spatial structures such information can be provided by the first pathway in a and temporal events separately. For concreteness, let us less redundant manner. We call the first a Slow pathway and study this in the context of Recognition . The categorical the second a Fast pathway, driven by their different temporal spatial semantics of the visual content often evolve slowly. speeds. The two pathways are fused by lateral connections. 6202. Our conceptual idea leads to flexible and effective designs Beyond spatiotemporal filtering or their separable ver- for Video models.

8 The Fast pathway, due to its lightweight sions, our work pursuits a more thorough separation of mod- nature, does not need to perform any temporal pooling it eling expertise by using two different temporal speeds. can operate on high frame rates for all intermediate layers and maintain temporal fidelity. Meanwhile, thanks to the Optical flow for Video Recognition . There is a classical lower temporal rate, the Slow pathway can be more focused branch of research focusing on hand-crafted spatiotemporal on the spatial domain and semantics. By treating the raw features based on optical flow.

9 These methods, including Video at different temporal rates, our method allows the two histograms of flow [33], motion boundary histograms [6], pathways to have their own expertise on Video modeling. and trajectories [53], had shown competitive performance There is another well known architecture for Video recog- for action Recognition before the prevalence of deep learning. nition which has a two-stream design [44], but provides con- In the context of deep neural Networks , the two-stream ceptually different perspectives. The Two-Stream method method [44] exploits optical flow by viewing it as another [44] has not explored the potential of different temporal input modality.

10 This method has been a foundation of many speeds, a key concept in our method. The two-stream method competitive results in the literature [12, 13, 55]. However, it adopts the same backbone structure to both streams, whereas is methodologically unsatisfactory given that optical flow is our Fast pathway is more lightweight. Our method does not a hand-designed representation, and two-stream methods are compute optical flow, and therefore, our models are learned often not learned end-to-end jointly with the flow. end-to-end from the raw data. In our experiments we observe that the SlowFast network is empirically more effective.