A Two-Level Learning Hierarchy of Nonnegative …

1 A Two-Level Learning Hierarchy of NonnegativeMatrix factorization based Topic Modeling forMain Topic ExtractionHendri MurfiDepartment of Mathematics, Universitas IndonesiaDepok 16424, modeling is a type of statistical model that has beenproven successful for tasks including discovering topics and their trendsover time. In many applications, documents may be accompanied bymetadata that is manually created by their authors to describe the se-mantic content of documents, titles and tags. A proper way of in-corporating this metadata to topic modeling should improve its perfor-mance. In this paper, we adapt a Two-Level Learning Hierarchy method forincorporating the metadata into Nonnegative matrix factorization basedtopic modeling. Our experiments on extracting main topics show that themethod improves the interpretability scores and also produces more in-terpretable topics than the baseline one-level Learning Hierarchy :topic modeling, Nonnegative matrix factorization , incorpo-rating metadata, Nonnegative least squares, main topic extraction1 IntroductionAs our collection of digital documents continues to be stored and gets huge, wesimply do not have the human power to read all of the documents to providethematic information.

2 Therefore, we need automatic tools for extracting thethematic information from the collection. Topic modeling is a type of statisticalmodel that has been proven successful for this task including discovering topicsand their trends over time. Topic modeling is an unsupervised Learning in thesense that it does not need labels of the documents. The topics are mined fromtextual contents of the documents. In other words, the general problem for topicmodeling is to use the observed documents to infer the hidden topic , with the discovered topics we can organize the collection for manypurposes, indexing, summarization, dimensionality reduction, etc [5]Latent Dirichlet allocation (LDA) [6] is a popular probabilistic topic model. Itwas developed to fix some issues with a previously developed topic model prob-abilistic latent semantic analysis (pLSA) [9].

3 LDA assumes that a documenttypically represents multiple topics which are modeled as distributions over avocabulary. Each word in the document is generated by randomly choosing a2H. Murfitopic from a distribution over topics , and then randomly choosing a word froma distribution over the vocabulary. The common methods to compute posteriorof the model are approximate inference techniques. Unfortunately, the maxi-mum likelihood approximations are NP-hard [2]. As a result, several researcherscontinue to design algorithms with provable guarantees for the problem of learn-ing the topic models. These algorithms include Nonnegative matrix factorization (NMF) [2, 4, 1].In many applications, the documents may contain metadata that we mightwant to incorporate into topic modeling. Titles and tags are examples of themetadata that usually accompany the documents in many applications.

4 Thismetadata is manually created by human to describe the thematic informationof documents. It becomes important because not only reflects the main topicsof documents but it also has a compact form. Therefore, a proper way to incor-porate this metadata to topic modeling is expected to improve the performanceof topic modeling. As far as we know, the methods that address the issue ofincorporating these metadata into NMF- based topic models are still rare. Thesimple approach to incorporate the metadata into NMF- based topic modeling isby unifying the metadata and the textual contents of documents, and then ex-tracting topics from this union set. The union of both textual data sets may use afusion parameter reflecting the importance of each set. We call this method as anone-level Learning Hierarchy (OLLH) method. Another approach is a two-levellearning Hierarchy (TLLH) method that is originally proposed for tag recom-mendations [15].

5 This Learning method extracts topics from the textual sourcesseparately. At the lower level, topics and topic-entity structures are discoveredby a NMF algorithm from tags. Having these topic-entity structures, the ex-tracted topics are enriched by words existing in textual contents related to theentity using a NLS algorithm at higher level. Recently, a method called non-negative multiple matrix factorization (NMMF) is proposed [17]. This methodincorporates the metadata as an auxiliary matrix that shares column with thecontent matrix and then decomposes both matrices simultaneously. From tech-nical point of view, this method is similar to OLLH which extracts topics fromthe contents and the metadata together. Moreover, this method is applicableonly for a specific NMF algorithm, multiplicative update this paper, we adapt the TLLH method for main topic extraction.

6 First themethod is extended to be applicable for general NMF algorithms. At the lowerlevel, topics is discovered by a NMF algorithm from the contents (the meta-data). Given the topics and the contents (the metadata), topic-content (topic-metadata) structures are approximated using a NLS algorithm. Having thesetopic-content (topic-metadata) structures, the extracted topics are enhanced bywords existing in the metadata (the contents) using a NLS algorithm at higherlevel. In contrast with OLLH, TLLH combines the vocabularies from the contentsand the metadata after the Learning process. Therefore, TLLH is more efficientin adapting to the characteristic of both textual sources. For example, someonline news portals share complete titles and only small part of contents, butother applications may share both titles and contents in a complete form.

7 OurA Two-Level Learning Hierarchy of NMF- based Topic Modeling3experiments on extracting main topics from online news show that incorporatingthe metadata into topic modeling improves interpretability or coherence scoresof the extracted topics . Moreover, the experiments show that TLLH is not onlymore efficient but it also gives higher interpretability scores than OLLH. Thetrends of extracted main topics over a time period may be used as backgroundinformation for other applications, sentiment analysis [8, 7].The rest of the paper is organized as follows: Section 2 discusses learningthe topic model parameters using Nonnegative matrix factorization . Section 3describes our proposed Two-Level Learning Hierarchy method. In Section 4, weshow our case study and results. We conclude and give a summary in Section Learning Model ParametersTopic modeling has been used to various text analyzes, where the most commontopic model currently in use is latent Dirichlet allocation (LDA) [6].

8 The intu-ition behind LDA is that all documents in the collection represent the same set oftopics in different proportion, where each topic is a combination of words. Thus,each combination of topics is itself a distribution on words. LDA hypothesizes aDirichlet distribution to generate the topic combinations. LDA addresses someissues with probabilistic latent semantic analysis (pLSA) [9] relating to the num-ber of parameters to be estimated and how to deal with documents outside thetraining set. Both models decompose the collection of documents into groupsof words representing the main topics and the new document representationsindicate which topics each document has. Because of some limitation to learn-ing the model parameters, NP-hard and getting stuck in a local minimal,several researchers continue the work to design algorithms with provable guar-antees [2].

9 The problem for Learning the topic model parameters is described inthe following formulation: There is an unknown topics matrixA Rn kwhereaij 0, and astochastically generated unknown matrixW Rk m. Each column ofAWis viewed as a probability distribution on rows, and for each columnwe are givenN << niid samples from the associated distribution. Thegoal of this meta problem in topic modeling is to reconstructAandparameters of the generating distribution forW [2].In literature, the problem of finding Nonnegative matrixA,Wwhen giventhe matrixAWis called Nonnegative matrix factorization (NMF). Actually,research related to the NMF was initiated by Paatero and Tapper in 1994 underthe name positive matrix factorization [16]. It became more widely known asnonnegative matrix factorization after Lee and Seung published some simpleand useful algorithms called a multiplicative update algorithm [12].

10 The NMFproblem is formulated as a constrained optimization problem. The algorithmhad shown that the objective function value is non-increasing and claimed thatthe limit points of the sequenceA,Wis a stationary point which is a necessarycondition for the local minimum [13]. However, this claim was later shown to4H. Murfibe incorrect. A summary about the convergence of the multiplicative updatealgorithm is: When the algorithm has converged to a limit point in the interior ofthe feasible region, this point is a stationary point. This stationary pointmay or may not be a local minimum. When the limit point lies on theboundary of the feasible region, its stationary can not be determined [3].Due to the shortcoming related to convergence properties, another methodcalled alternating Nonnegative least squares (ANLS) algorithm is considered tobe an alternative one.