A Unified Architecture for Instance and Semantic Segmentation

1 A Unified Architecture forInstance and Semantic SegmentationAlexander KirillovKaimingHeRoss GirshickPiotr Doll rFACEBOOK AI RESEARCHUNIVERSIT THEIDELBERGO bject Detection vs Semantic SegmentationSemantic Segmentation Object DetectionObject Detection vs Semantic SegmentationSemantic Segmentation Object Detection/SegSemantic Segmentation Object Detection/SegDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Seg Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Segpredictdilated net Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Segpredict DeepLab PSPN etdilated net Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Seg SegNet U-Net RefineNetpredictdecoder-encoder net DeepLab PSPN etpredictdilated net Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Seg SegNet U-Net RefineNetpredictdecoder-encoder net DeepLab PSPN etpredictpredictpredictpredict Mask R-CNN RetinaNetpredictFPN netdilated net Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionSemantic Segmentation Object Detection/Seg SegNet U-Net RefineNetpredictdecoder-encoder net DeepLab PSPN etpredictpredictpredictpredict Mask R-CNN RetinaNetpredictFPN netdilated net Fast/erR-CNNclassification netpredictDeep Networks in Object RecognitionFPN Architecture1418116132image1 ResNet152 [1] /ResNeXt152 [2]high resolutionlow resolutionweak featuresstrong features[1] He, K.

2 , Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR Architecture1418116132image1high resolutionlow resolutionstrong featuresstrong features[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR 2017.[3] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object pyramid Network (FPN) [3]325651210242048256256256256 FPN Architecture1418116132image12x up1x1 conv+high resolutionlow resolutionstrong featuresstrong features[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition.

3 CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR 2017.[3] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object pyramid Network (FPN) [3]325651210242048256256256256 FPN Architecture1418116132image1[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR 2017.[3] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object pyramid Network (FPN) [3]325651210242048256256256256network headnetwork headnetwork headnetwork headFPN Architecture1418116132image1 Mask R-CNN[4] feature pyramid Network (FPN) [3]network head[1] He, K.

4 , Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR 2017.[3] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[4] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. network headnetwork headnetwork headnetwork head325651210242048256256256256 FPN Architecture1418116132image1 Mask R-CNN[4] feature pyramid Network (FPN) [3]network head[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR 2017.[3] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S.

5 feature pyramid networks for object 2017.[4] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. network headnetwork headnetwork headnetwork headseghead 1/32seghead 1/16seghead 1/8seghead 1/4our work325651210242048256256256256 FPN for Semantic Segmentation1418116132image1seghead 1/32seghead 1/16seghead 1/8seghead 1/4[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. assembling325651210242048256256256256 FPN for Semantic Segmentation1418116132image3256512102420 482562562562561128128128128[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P.

6 , Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. FPN for Semantic Segmentation1418116132image3256512102420 4825625625625611281281281283x3 conv3x3 conv[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. FPN for Semantic Segmentation1418116132image3256512102420 482562562562561128128128128[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K.

7 , Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. FPN for Semantic Segmentation1418116132image3256512102420 482562562562561128128128128[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. 8x upFPN for Semantic Segmentation1418116132image3256512102420 4825625625625611281281281284x up[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. FPN for Semantic Segmentation1418116132image3256512102420 4825625625625611281281281282x up[1] He, K.

8 , Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. FPN for Semantic Segmentation1418116132image3256512102420 482562562562561128128128128[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. 512 FPN for Semantic Segmentation1418116132image3256512102420 482562562562561128128128128[1] He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. CVPR 2016.[2] Lin, T. Y., Doll r, P., Girshick, R.

9 , He, K., Hariharan, B., & Belongie, S. feature pyramid networks for object 2017.[3] He, K., Gkioxari, G., Doll r, P., & Girshick, R. Mask 2017. 512K14prediction3x3convResNeXt-FPN EfficiencyResNeXt1521418116132 ResNeXt-FPN EfficiencyResNeXt1523 blocks8blocks36blocks3blocks1418116132 ResNeXt-FPN Efficiency1x ResNeXt152 ResNeXt1523 blocks8blocks36blocks3blocks14181161321x 1x0x1x2x3x4xMemoryFLOPsResNeXt152 ResNeXt-FPN EfficiencyResNeXt152-dilation (stride 8)3 blocks8blocks3blocks141813236blocks1161x 1x0x1x2x3x4xMemoryFLOPsResNeXt152 ResNeXt-FPN EfficiencyResNeXt152-dilation (stride 8)3 blocks8blocks3blocks14181321836blocks4x FLOPs1x1x0x1x2x3x4xMemoryFLOPsResNeXt152 ResNeXt-FPN EfficiencyResNeXt152-dilation (stride 8)3 blocks8blocks3blocks16x FLOPs1418181836blocks4x FLOPs1x1x0x1x2x3x4xMemoryFLOPsResNeXt152 ResNeXt-FPN EfficiencyResNeXt152-dilation (stride 8)3 blocks8blocks14181818 ResNeXt1523blocks16x FLOPs36blocks4x (stride 8)ResNeXt-FPN EfficiencyResNeXt152-FPN14181321163 (stride 8)

10 ResNeXt-FPN EfficiencyResNeXt152-FPN14181321163 blocks8blocks3blocks36blocks 1x (stride 8)ResNeXt-FPN EfficiencyResNeXt152-FPN14181321163 blocks8blocks3blocks36blocks 1x ResNeXt152 (stride 8)ResNeXt-FPN EfficiencyResNeXt152-FPN14181321163 blocks8blocks3blocks36blocks ResNeXt152 1x (stride 8)ResNeXt152-FPNResNeXt-FPN Training DetailsData augmentation:[1] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. SSD: Single shot multiboxdetector. ECCV 2016.[2] Xie, S., Girshick, R., Doll r, P., Tu, Z., & He, K. Aggregated residual transformations for deep neural networks. CVPR : : (max 800x800)..Color augmentation [1]ResNeXt-FPN Training DetailsData augmentation:[1] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. SSD: Single shot multiboxdetector. ECCV 2016.[2] Xie, S., Girshick, R.