Florence: A New Foundation Model for Computer Vision

1 Florence: A New Foundation Model for Computer VisionLu Yuan1 Dongdong Chen* 1Yi-Ling Chen* 1 Noel Codella* 1 Xiyang Dai* 1 Jianfeng Gao* 2 Houdong Hu* 1 Xuedong Huang* 1 Boxin Li* 1 Chunyuan Li* 2Ce Liu* 1 Mengchen Liu* 1 Zicheng Liu* 1 Yumao Lu* 1Yu Shi* 1 Lijuan Wang* 1 Jianfeng Wang* 1 Bin Xiao* 1 Zhen Xiao* 1 Jianwei Yang* 2 Michael Zeng* 1 Luowei Zhou* 1 Pengchuan Zhang* 2 AbstractAutomated visual understanding of our diverseand open world demands Computer Vision modelsto generalize well with minimal customization forspecific tasks, similar to human Vision . Computervision Foundation models, which are trained ondiverse, large-scale dataset and can be adaptedto a wide range of downstream tasks, are criti-cal for this mission to solve real-world computervision applications. While existing Vision founda-tion models such as CLIP (Radford et al., 2021),ALIGN (Jia et al., 2021), and Wu Dao (Wud)focus mainly on mapping images and textual rep-resentations to a cross-modal shared representa-tion, we introduce a new Computer Vision foun-dation Model ,Florence, to expand the represen-tations from coarse (scene) to fine ( object ), fromstatic (images) to dynamic (videos), and fromRGB to multiple modalities (caption, depth).

2 Byincorporating universal visual-language represen-tations from Web-scale image-text data, ourFlo-rencemodel can be easily adapted for variouscomputer Vision tasks, such as classification, re-trieval, object detection , VQA, image caption,video retrieval and action recognition. Moreover,Florencedemonstrates outstanding performancein many types of transfer learning: fully sampledfine-tuning, linear probing, few-shot transfer andzero-shot transfer for novel images and of these properties are critical for our visionfoundation Model to serve general purpose new state-of-the-art re-sults in majority of44representative benchmarks, zero-shot classification withtop-1 accuracy the top-5 accuracy , on COCO fine tuning, , Kinetics-600.*Florence Team member in alphabetic order1 Microsoft Cloudand AI2 Microsoft Research Redmond. Correspondence to: (only)ClassificationFlowerAction RecognitionPlaying SoccerObject DetectionEagleEagleSegmentationDepthA group of women sitting around a ,,How many red buttons?

3 Visual Question AnsweringObject TrackingVideo ReasoningWhat are they talking about?Figure Computer Vision tasks are mapped to aSpace- time -Modalityspace. A Computer Vision Foundation Model shouldserve as general purpose Vision system for all of these IntroductionHuman-like AI is not achieved by designing specific modelsto solve specific problems, but by holistic, joint modelsthat can simultaneously solve diverse, real-world problemswithout too much human involvement. It is thus desirableto have new AI architectures that learn joint, fundamentalrepresentations to support a broad range of downstream AItasks with limited additional domain knowledge, similarto what humans would do. One such proposal is XYZ-code (Huang), where monolingual text (X), audio and visualsensory signals (Y), and multilingual (Z) are organicallyintegrated to create AI models that can speak, hear, see, andunderstand.

4 Another approach is Pathways (Dean), a singlemodel that can generalize across millions of concrete step towards this direction is the developmentoffoundationmodels. The term offoundationmodel wasfirst introduced in (Bommasani et al., 2021) to refer to anymodel that is trained from broad data at scale that is capableof being adapted ( ) to a wide range of down-stream tasks. Foundation models become promising due totheir impressive performance and generalization [ ] 22 Nov 2021 Florence: A New Foundation Model for Computer VisionTwo kangaroos on a beachRetrievalImage-Text Dataset by Data Curationfrom InternetImageTextDogRowers carrying boat over heads on a dockLanguage Encoder Unified Contrastive LearningImage Encoder (CoSwin)Florence Pretrained ModelsObject-level Representation (Dynamic Head Adaptor)Fine-grained V+L Representation (METER Adaptor)Video Representation (Video CoSwin)Classification/Retrieval AdaptationFlorence Adaptation ModelsFlorence ( Vision Foundation Model )ClassificationFlowerObject DetectionEagleEagleHow many red buttons?

5 VQAA ction RecognitionPlaying SoccerUnified Vision StackScalable Training InfrastructureDeploymentTasksFigure of building Florence. Our workflow consists of data curation, unified learning, Transformer architectures and shows the Foundation Model can be adapted to various downstream tasks and finally integrated into modern Computer Vision systemto power real-world Vision and multimedia applications. Compared with existing image-text pretraining models (Radford et al., 2021;Jia et al., 2021; Wud), mainly limited on cross-modal shared representation for classification and retrieval (illustrated bylight-greenadaptation module), Florence expands the representation to support object level, multiple modality, and videos are quickly integrated and deployed into real-worldAI systems by many researchers and Foundation models have already demonstratedhuge impact in NLP, , BERT (Devlin et al.)

6 , 2019), GPT-3 (Brown et al., 2020), in Computer Vision it is still standardpractice to pre-train models on labeled data sets such asImageNet (Deng et al., 2009). More recently, large-scalepre-training methods such as CLIP (Radford et al., 2021),ALIGN (Jia et al., 2021), and Wu Dao (Wud), whichlearn directly from Web-scale image-text pairs, show veryencouraging progress for efficient transfer learning, andzero-shot capability. However, such models are restrictedto image to text mapping only tasks such as classification,retrieval, and raise the question: What is the Foundation Model forcomputer Vision ? . But first, in order to better define what Foundation means in Computer Vision , we capture the spec-trum of tasks in a problem space (Figure 1) with three orthog-onal axes: 1)Space: from coarse ( classifi-cation) to fine-grained ( detection ), 2) time : fromstatic ( ) to dynamic ( ), and 3)Modal-ity: from RGB only to multiple senses ( anddepth).

7 Due to the diversity nature of visual understanding,we redefinefoundation models for Computer visionto bea pre-trained Model and its adaptersfor solving all visiontasks in this Space- time -Modality space, with transferabil-ity such as zero-/few-shot learning and fully fine tuning, adaptation for transferability is restricted to minimumcustomization for the pre-trained Foundation models, suchas continuing training, few epochs or few layers for finetuning without significantly increasing or changing this paper, we present an emerging paradigm for buildingavision Foundation Model , calledFlorence. We use the nameofFlorenceas the origin of the trail for exploringvisionfoundationmodels, as well as the birthplace of trained on noisy Web-scale data end-to-end witha unifying objective, allowing the Model to achieve best-in-class performance across a wide range of ecosystem of constructingFlorenceconsists ofdatacuration, Model pretraining,task adaptationsandtraininginfrascturue, as shown in Figure 2.

8 Data , large-scale data is thelifeblood of Foundation models. Enabled by largeamounts of publicly available images on the Internetwith natural language weak supervision, we curate anew dataset of900million image-text pairs for train-ing. As Web-crawled data is usually noisy free-formtexts ( , word, phrase or sentence), to attain moreeffective learning, we considerUniCL, a unified image-text contrastive learning objective recently proposedin (Yang et al., 2022), which has demonstrated im-provements over contrastive and supervised learningapproaches. Model pretraining(representation learning). To learna good representation from image-text pairs, we used aFlorence: A New Foundation Model for Computer Visiontwo-towerarchitectureincluding an image encoder anda language encoder, as commonly used in CLIP (Rad-ford et al., 2021) and ALIGN (Jia et al., 2021). Forthe image encoder, we chose hierarchical Vision Trans-formers ( , Swin (Liu et al.))

9 , 2021a), CvT (Wu et al.,2021), Vision Longformer (Zhang et al., 2021a), FocalTransformer (Yang et al., 2021), and CSwin (Donget al., 2021)). While inheriting performance benefits ofthe transformer self-attention operations (Dosovitskiyet al., 2021b), these hierarchical architectures modelthe scale invariance nature of images and have linearcomputational complexity with respect to image size, aproperty that is essential to dense prediction tasks suchas object detection and segmentation. Task adaptations. As we have defined Computer vi-sion Foundation models to adapt to various downstreamtasks, it is vital forFlorenceto beextensibleandtrans-ferablefor this purpose. We extended the learned fea-ture representation along space (from scene to objects)using the dynamic head adapter (Dai et al., 2021a), time (from static image to videos) via proposed videoCoSwin adapter, and modality (from images to lan-guage) via METER adapter (Dou et al.

10 , 2021).Flo-renceis designed to effectively adapted in the openworld via few-shot and zero-shot transfer learning, withthe ability of efficient deployment by extra trainingwith few epochs ( retrieval). Our Model canbe customized for various domains that application-developers can use. Training infrastructure. For both energy and costconcerns, it is critical to build Foundation models withas low cost as possible. We developed scalable traininginfrastructure to improve training efficiency. It consistsof several key techniques such as ZeRO (Rajbhandariet al., 2019), activation checkpointing, mixed-precisiontraining, gradient cache (Gao et al., 2021) to greatlyreduce the memory consumption and thus improvesthe training outperforms previous large-scale pre-training methods and achieves new state-of-the-art resultson a wide range of Vision and Vision -language showed strength in zero-shot transfer in12classificationdownstream tasks (win9/12, SOTA in ImageNet-1K zero-shot with top-1 accuracy the top-5 accuracy ), linear probe in11classification downstream tasks(win9/11), image retrieval zero-shot ( @1onFlickr30K image-to-text / text-to-image, @1on MSCOCO image-to-text / text-to-image) and fine-tuning( @1on Flickr30K image-to-text / text-to-image, @1on MSCOCO image-to-text/ text-to-image), object detection ( on COCO, on Ob-ject365, on Visual Genome), VQA ( ),text-to-video retrieval zero-shot ( @1on MSR-VTT),and video action recognition (top-1 / Kinetics-600).