L LM 2: MULTI MODAL PRE TRAINING FOR V -R DOCUMENT …

1 LayoutLMv2: MULTI - MODAL Pre- TRAINING for Visually-richDocument UnderstandingYang Xu1 , Yiheng Xu2 , Tengchao Lv2 , Lei Cui2, furu Wei2, Guoxin Wang3,Yijuan Lu3, Dinei Florencio3, Cha Zhang3, Wanxiang Che1, Min Zhang4, Lidong Zhou21 Research Center for Social Computing and Information Retrieval,Harbin Institute of Technology2 Microsoft Research Asia3 Microsoft Azure AI4 Soochow of text and layout has provedeffective in a variety of visually-rich docu-ment understanding tasks due to its effec-tive model architecture and the advantageof large-scale unlabeled scanned/digital-borndocuments.

2 We proposeLayoutLMv2archi-tecture with new pre- TRAINING tasks to modelthe interaction among text, layout, and imagein a single MULTI - MODAL framework. Specif-ically, with a two-stream MULTI - MODAL Trans-former encoder, LayoutLMv2 uses not onlythe existing masked visual-language model-ing task but also the new text-image align-ment and text-image matching tasks, whichmake it better capture the cross-modality in-teraction in the pre- TRAINING stage. Meanwhile,it also integrates a spatial-aware self-attentionmechanism into the Transformer architectureso that the model can fully understand therelative positional relationship among differ-ent text results showthat LayoutLMv2 outperforms LayoutLM bya large margin and achieves new state-of-the-art results on a wide variety of down-stream visually-rich DOCUMENT understandingtasks, including FUNSD ( ),CORD ( ), SROIE ( ), Kleister-NDA ( ),RVL-CDIP ( ), and DocVQA( ).

3 We made our model andcode publicly available IntroductionVisually-rich DOCUMENT Understanding (VrDU)aims to analyze scanned/digital-born business doc-uments (images of invoices, forms in PDF format,etc.) where structured information can be automat-ically extracted and organized for many business Equal contributions during internship at MSRA applications. Distinct from conventional informa-tion extraction tasks, the VrDU task relies on notonly textual information but also visual and lay-out information that is vital for visually-rich docu-ments. Different types of documents indicate thatthe text fields of interest located at different posi-tions within the DOCUMENT , which is often deter-mined by the style and format of each type as wellas the DOCUMENT content.

4 Therefore, to accuratelyrecognize the text fields of interest, it is inevitableto take advantage of the cross-modality nature ofvisually-rich documents, where the textual, visual,and layout information should be jointly modeledand learned end-to-end in a single recent progress of VrDU lies primarilyin two directions. The first direction is usuallybuilt on the shallow fusion between textual andvisual/layout/style information (Yang et al., 2017;Liu et al., 2019; Sarkhel and Nandi, 2019; Yu et al.,2020; Majumder et al., 2020; Wei et al., 2020;Zhang et al., 2020). These approaches leveragethe pre-trained NLP and CV models individuallyand combine the information from multiple modali-ties for supervised learning.

5 Although good perfor-mance has been achieved, the domain knowledgeof one DOCUMENT type cannot be easily transferredinto another, so that these models often need tobe re-trained once the DOCUMENT type is the local invariance in general documentlayout (key-value pairs in a left-right layout, tablesin a grid layout, etc.) cannot be fully exploited. Tothis end, the second direction relies on the deep fu-sion among textual, visual, and layout informationfrom a great number of unlabeled documents in dif-ferent domains, where pre- TRAINING techniques playan important role in learning the cross-modalityinteraction in an end-to-end fashion (Lockard et al.)

6 ,2020; Xu et al., 2020). In this way, the pre- [ ] 10 Jan 2022models absorb cross- MODAL knowledge from dif-ferent DOCUMENT types, where the local invarianceamong these layouts and styles is preserved. Fur-thermore, when the model needs to be transferredinto another domain with different DOCUMENT for-mats, only a few labeled samples would be suf-ficient to fine-tune the generic model in order toachieve state-of-the-art accuracy. Therefore, theproposed model in this paper follows the seconddirection, and we explore how to further improvethe pre- TRAINING strategies for the VrDU this paper, we present an improved versionof LayoutLM (Xu et al.

7 , 2020), from the vanilla LayoutLM model wherevisual embeddings are combined in the fine-tuningstage, we integrate the visual information in thepre- TRAINING stage in LayoutLMv2 by taking ad-vantage of the Transformer architecture to learnthe cross-modality interaction between visual andtextual information. In addition, inspired by the1-D relative position representations (Shaw et al.,2018; Raffel et al., 2020; Bao et al., 2020), we pro-pose the spatial-aware self-attention mechanism forLayoutLMv2, which involves a 2-D relative posi-tion representation for token pairs. Different fromthe absolute 2-D position embeddings that Lay-outLM uses to model the page layout, the relativeposition embeddings explicitly provide a broaderview for the contextual spatial modeling.

8 For thepre- TRAINING strategies, we use two new TRAINING ob-jectives for LayoutLMv2 in addition to the maskedvisual-language modeling. The first is the proposedtext-image alignment strategy, which aligns the textlines and the corresponding image regions. The sec-ond is the text-image matching strategy popular inprevious vision-language pre- TRAINING models (Tanand Bansal, 2019; Lu et al., 2019; Su et al., 2020;Chen et al., 2020; Sun et al., 2019), where themodel learns whether the DOCUMENT image and tex-tual content are select six publicly available benchmarkdatasets as the downstream tasks to evaluate the per-formance of the pre-trained LayoutLMv2 model,which are the FUNSD dataset (Jaume et al.)

9 , 2019)for form understanding, the CORD dataset (Parket al., 2019) and the SROIE dataset (Huang et al.,2019) for receipt understanding, the Kleister-NDAdataset (Grali nski et al., 2020) for long docu-ment understanding with a complex layout, theRVL-CDIP dataset (Harley et al., 2015) for doc-ument image classification, and the DocVQAdataset (Mathew et al., 2021) for visual question an-swering on DOCUMENT images. Experiment resultsshow that the LayoutLMv2 model significantly out-performs strong baselines, including the vanillaLayoutLM, and achieves new state-of-the-art re-sults in all of these contributions of this paper are summarizedas follows: We propose a MULTI - MODAL Transformer modelto integrate the DOCUMENT text, layout, andvisual information in the pre- TRAINING stage,which learns the cross- MODAL interaction end-to-end in a single framework.

10 Meanwhile,a spatial-aware self-attention mechanism isintegrated into the Transformer architecture. In addition to the masked visual-languagemodel, we add text-image alignment and text-image matching as the new pre- TRAINING strate-gies to enforce the alignment among differentmodalities. LayoutLMv2 significantly outperforms andachieves new SOTA results not only on theconventional VrDU tasks but also on the VQAtask for DOCUMENT images, which demon-strates the great potential for the MULTI -modalpre- TRAINING for ApproachIn this section, we will introduce the model archi-tecture and the MULTI - MODAL pre- TRAINING tasks ofLayoutLMv2, which is illustrated in Figure Model ArchitectureWe build a MULTI - MODAL Transformer architectureas the backbone of LayoutLMv2, which takes text,visual, and layout information as input to estab-lish deep cross- MODAL interactions.