SuperPoint: Self-Supervised Interest Point Detection and ...

1 SuperPoint: Self-Supervised Interest Point Detection and Description Daniel DeTone Tomasz Malisiewicz Andrew Rabinovich Magic Leap Magic Leap Magic Leap Sunnyvale, CA Sunnyvale, CA Sunnyvale, CA. Abstract Image Pair SuperPoint Network Point Correspondence This paper presents a Self-Supervised framework for Interest Points training Interest Point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural net- works, our fully-convolutional model operates on full-sized images and jointly computes pixel-level Interest Point loca- Descriptors tions and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi- homography approach for boosting Interest Point detec- tion repeatability and performing cross-domain adapta- tion ( , synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of Interest points than the initial pre-adapted deep model and any other traditional corner detector.

2 The final system gives rise to state-of-the-art homography estimation results Figure 1. SuperPoint for Geometric Correspondences. We on HPatches when compared to LIFT, SIFT and ORB. present a fully-convolutional neural network that computes SIFT- like 2D Interest Point locations and descriptors in a single forward 1. Introduction pass and runs at 70 FPS on 480 640 images with a Titan X GPU. The first step in geometric computer vision tasks such as Simultaneous localization and mapping ( slam ), cations labeled by human annotators. Structure-from-Motion (SfM), camera calibration, and im- It seems natural to similarly formulate Interest Point de- age matching is to extract Interest points from images. In- tection as a large-scale supervised machine learning prob- terest points are 2D locations in an image which are stable lem and train the latest convolutional neural network ar- and repeatable from different lighting conditions and view- chitecture to detect them.

3 Unfortunately, when compared points. The subfield of mathematics and computer vision to semantic tasks such as human-body keypoint estimation, known as Multiple View Geometry [9] consists of theorems where a network is trained to detect body parts such as the and algorithms built on the assumption that Interest points corner of the mouth or left ankle, the notion of Interest Point can be reliably extracted and matched across images. How- Detection is semantically ill-defined. Thus training convo- ever, the inputs to most real-world computer vision systems lution neural networks with strong supervision of Interest are raw images, not idealized Point locations. points is non-trivial. Convolutional neural networks have been shown to be Instead of using human supervision to define Interest superior to hand-engineered representations on almost all points in real images, we present a Self-Supervised solu- tasks requiring images as input. In particular, fully- tion using self-training.

4 In our approach, we create a large convolutional neural networks which predict 2D key- dataset of pseudo-ground truth Interest Point locations in points or landmarks are well-studied for a variety of real images, supervised by the Interest Point detector itself, tasks such as human pose estimation [31], object detec- rather than a large-scale human annotation effort. tion [14], and room layout estimation [12]. At the heart To generate the pseudo-ground truth Interest points, we of these techniques is a large dataset of 2D ground truth lo- first train a fully-convolutional neural network on millions 1337. (a) Interest Point Pre-Training (b) Interest Point Self-Labeling (c) Joint Training SuperPoint Labeled Interest Interest Unlabeled Image Pseudo-Ground Point Images Point Loss Truth Interest Base Detector Points Homographic Descriptor Train Warp Base Detector Adaptation Loss Interest Point Loss [see Section 4] [see Section 5] [see Section 3]. Figure 2. Self-Supervised Training Overview.

5 In our Self-Supervised approach, we (a) pre-train an initial Interest Point detector on synthetic data and (b) apply a novel Homographic Adaptation procedure to automatically label images from a target, unlabeled domain. The generated labels are used to (c) train a fully-convolutional network that jointly extracts Interest points and descriptors from an image. of examples from a synthetic dataset we created called Syn- vances in applying deep learning to Interest Point Detection thetic Shapes (see Figure 2a). The synthetic dataset con- and descriptor learning. At the ability to match image sub- sists of simple geometric shapes with no ambiguity in the structures, we are similar to UCN [3] and to a lesser extent Interest Point locations. We call the resulting trained de- DeepDesc [6]; however, both do not perform any Interest tector MagicPoint it significantly outperforms traditional Point Detection . On the other end, LIFT [32], a recently in- Interest Point detectors on the synthetic dataset (see Sec- troduced convolutional replacement for SIFT stays close to tion 4).

6 MagicPoint performs surprising well on real im- the traditional patch-based detect then describe recipe. The ages despite domain adaptation difficulties [7]. However, LIFT pipeline contains Interest Point Detection , orientation when compared to classical Interest Point detectors on a di- estimation and descriptor computation, but additionally re- verse set of image textures and patterns, MagicPoint misses quires supervision from a classical SfM system. These dif- many potential Interest Point locations. To bridge this gap ferences are summarized in Table 1. in performance on real images, we developed a multi-scale, Interest Descriptors? Full Image Single Real multi-transform technique Homographic Adaptation. Points? Input? Network? Time? Homographic Adaptation is designed to enable self- SuperPoint (ours) . LIFT [32] . supervised training of Interest Point detectors. It warps the UCN [3] . input image multiple times to help an Interest Point detec- TILDE [29].

7 DeepDesc [6] . tor see the scene from many different viewpoints and scales SIFT . (see Section 5). We use Homographic Adaptation in con- ORB . junction with the MagicPoint detector to boost the perfor- Table 1. Qualitative Comparison to Relevant Methods. Our Su- mance of the detector and generate the pseudo-ground truth perPoint method is the only one to compute both Interest points Interest points (see Figure 2b). The resulting detections are and descriptors in a single network in real-time. more repeatable and fire on a larger set of stimuli; thus we named the resulting detector SuperPoint. On the other extreme of the supervision spectrum, Quad- The most common step after detecting robust and repeat- Networks [23] tackles the Interest Point Detection problem able Interest points is to attach a fixed dimensional descrip- from an unsupervised approach; however, their system is tor vector to each Point for higher level semantic tasks, , patch-based (inputs are small image patches) and relatively image matching.

8 Thus we lastly combine SuperPoint with shallow 2-layer network. The TILDE [29] Interest Point a descriptor subnetwork (see Figure 2c). Since the Super- Detection system used a principle similar to Homographic Point architecture consists of a deep stack of convolutional Adaptation; however, their approach does not benefit from layers which extract multi-scale features, it is straightfor- the power of large fully-convolutional neural networks. ward to then combine the Interest Point network with an ad- Our approach can also be compared to other self- ditional subnetwork that computes Interest Point descriptors supervised methods, synthetic-to-real domain-adaptation (see Section 3). The resulting system is shown in Figure 1. methods. A similar approach to Homographic Adaptation is by Honari et al. [10] under the name equivariant land- 2. Related Work mark transform. Also, Geometric Matching Networks [20]. Traditional Interest Point detectors have been thoroughly and Deep Image Homography Estimation [4] use a similar evaluated [24, 16].

9 The FAST corner detector [21] was the self-supervision strategy to create training data for estimat- first system to cast high-speed corner Detection as a machine ing global transformations. However, these methods lack learning problem, and the Scale-Invariant Feature Trans- Interest points and Point correspondences, which are typi- form, or SIFT [15], is still probably the most well-known cally required for doing higher level computer vision tasks traditional local feature descriptor in computer vision. such as slam and SfM. Joint pose and depth estimation Our SuperPoint architecture is inspired by recent ad- models also exist [33, 30, 28], but do not use Interest points. 2338. Interest Point Decoder W. 3. SuperPoint Architecture Conv W/8. Input Softmax Reshape Encoder We designed a fully-convolutional neural network archi- W H/8 H. 65. tecture called SuperPoint which operates on a full-sized im- 1. Descriptor Decoder age and produces Interest Point detections accompanied by Conv W.

10 H. fixed length descriptors in a single forward pass (see Fig- W/8. Bi-Cubic L2. ure 3). The model has a single, shared encoder to pro- 1 Interpolate Norm H/8 H. cess and reduce the input image dimensionality. After the D. D. encoder, the architecture splits into two decoder heads , which learn task specific weights one for Interest Point de- Figure 3. SuperPoint Decoders. Both decoders operate on a tection and the other for Interest Point description. Most of shared and spatially reduced representation of the input. To keep the network's parameters are shared between the two tasks, the model fast and easy to train, both decoders use non-learned upsampling to bring the representation back to RH W . which is a departure from traditional systems which first detect Interest points, then compute descriptors and lack the ability to share computation and representation across the Descriptor Decoder two tasks. The descriptor head computes D RHc Wc D and out- puts a tensor sized RH W D.