Approaches, methods, metrics, measures, and subjectivity ...

1 Undefined 1 (2014) 1 51 IOS PressApproaches, methods , metrics , measures, andsubjectivity in ontology evaluation : A surveyEditor(s):Name Surname, University, CountrySolicited review(s):Name Surname, University, CountryOpen review(s):Name Surname, University, CountryHlomani Hlomani , and Deborah StaceySchool of Computer Science, University of Guelph, 50 Stone Road East, Guelph, Ontario, N1G 2W1,CanadaE-mail: evaluation is concerned with ascertain two important aspects of ontologies: quality and correctness. Thedistinction between the two is attempted in this survey as a way to better approach ontology evaluation . The role that ontologiesplay on the semantic web at large has been has been argued to have catalyzed the proliferation of ontologies in existence. Thishas also presented the challenge of deciding one the suitability of an given ontology to one s purposes as compared with anotherontology in a similar domain.

2 This survey intends to analyze the state of the art in ontology evaluate spanning such topics as theapproaches to ontology evaluation , the metrics and measures used. Particular interest is given to Data-driven ontology evaluationwith special emphasis on the notion of bias and it s relevance to evaluation results. Chief among the outputs of this survey is thegap analysis on the topic of ontology : Ontology, ontology evaluation , ontology evaluation methods , ontology evaluation metrics , ontology evaluationapproach1. IntroductionOntology evaluation is a emerging field that has anumber of frameworks and methodologies in existence[40]. The relevance and importance of ontology evalu-ation is evident in the role the play in the semantic weband ontology-enabled applications. They are the cen-trepiece of knowledge description in the semantic weballowing for the definition of a shared knowledge basethat can be acted upon by agents acting on the behalfof humans.

3 Given this role, they have since attracteda lot of interest from both academic and industrial do-main leading to the proliferation of ontologies in exis-tence. While this is attractive, it presents a challenge indeciding which ontology to reuse (as they are reusable*Corresponding author. E-mail: artefacts) and hence the topic of paper discusses some of the frameworks andmethodologies for ontology evaluation with the viewof performing a gap analysis. It first gives a contextto ontology evaluation by defining the notion of on-tology evaluation (Section ) and discussing ontol-ogy evaluation in the context of ontology reuse as anexample scenario for the role of ontology evaluation (Section ). This is then followed by an overviewof related work that has be done on ontology evalua-tion by breaking it down into several categories (Sec-tion 4).

4 Prior to the discussion on the different cat-egories of ontology evaluation , this survey discussesmeasures/ metrics that have been used in the differentmethods to decide the quality of ontologies (Section3). Finally the paper concludes by discussing what still0000-0000/14/$ 2014 IOS Press and the authors. All rights reserved2H. Hlomani, and Deborah Stacey / approaches , methods , metrics , measures, and subjectivity in ontology evaluation : A surveyneeds to be done with respect to ontology evaluation (Section 6).2. A context for ontology evaluationAn ontology has been previously defined as an ex-plicit formal specification of a conceptualization wherethe conceptualization in this context refers to the ab-straction of a domain of interest [18]. These abstrac-tions are increasingly used in various fields such as in-formation extraction, data integration, and the biggestof which is the semantic web.

5 This apparent increasein the use of ontologies has lead to an increase in thenumber of ontologies in existence which in turn hasheightened the need for evaluating the A definitionGenerally ontology evaluation can be defined as theprocess of deciding on the quality of an ontology inrespect to a particular criterion with the view of deter-mining which in a collection of ontologies would bestsuit a particular purpose [2]. An interesting definitionof ontology evaluation has be given by G mez-P rezet al. [17] and later echoed by Vrandecic et al. [40]. Inthese works, ontology evaluation is defined in the con-text of two interesting concepts; verification and vali-dation. The definition is interesting because it also of-fers a way to categorize current ontology evaluationendeavours. Ontology verification is concerned withbuilding an ontology correctly, while ontology valida-tion on the other hand is concerned with building thecorrect Ontology integration and mergingA considerable number of ontologies have been cre-ated.

6 It follows, therefore, that the time has come forthese ontologies to be reused. Ontology integration andmerging offers the two most obvious scenarios for theuses of ontology evaluation under the frame of ontol-ogy reuse [31]. Their primary aim is to build an on-tology from existing ontologies. There are subtle dif-ferences between the two concepts but both are a partof the ontology building process. Ontology integrationis focused on building an ontology through specializ-ing, extending and/or adapting existing ontologies thatform a part of the resultant ontology. For example,given an algorithms ontology, one may introduce an-other class or type of algorithm ( greedy, simple re-cursive) within the same ontology thereby specializingthe algorithms class. Ontology merging, on the otherhand, is concerned with merging different ontologiesinto a single one.

7 For example, if two ontologies modeltime: temporal ontology (O1) and time ontology (O2),thenO1andO2 s concepts and relations are strate-gically merged together resulting in a new ontology(O1+O2=O )3. metrics for ontology Ontology evaluation metrics : State of the artThe relevance of ontologies as engineering artifactshas been justified in literature. Hence, like every en-gineering artifact, evaluation is a precursor to their(re)use. The challenge, however, has been on how toactually go about evaluating these engineering arti-facts. In this regard, most questions have been directedat the applicability of software evaluation metrics toontology evaluation . Arguments about this applicabil-ity stem from distinguishing ontologies from softwareprocesses, and rather seeing them as data models [20].

8 It has been said that an ontology is a conceptualiza-tion of a domain" [18]. Guarino [20], however, reducesthis definition to reflect that ontologies are ratherap-proximatespecifications of a domain and argues thatontology evaluation should reflect the degree of suchapproximation. In the same work, propositions of met-rics relevant to information retrieval were suggested,more specifically the notions of precision and recall (equated with coverage). These propositions are echoedin Brewster et al. [4] and more recently in Ouyang etal. [25], albeit with a warning on naive interpretationsof the meaning of precision, and recall or et al. [6] s approach to ontology qual-ity takes from the measurement tradition in softwareengineering. From this viewpoint, measures of a pro-gram s internal attributes (such as coupling and co-hesion) are believed to influence external quality at-tributes (such as maintainability and performance), andhence their measurement is indicative of the program squality at least in the context of the aforementionedquality attributes.

9 In this regard, they proposed a theo-retical framework based on a metric suite consisting offour metrics , and ten attributes, where the metrics are:(i) Syntactic quality, (ii) Semantic quality, (iii) Prag-matic quality, and (iv) Social quality [6].Ontology quality metrics were described and parti-tioned into three main types in Gangemi et al. [14]:H. Hlomani, and Deborah Stacey / approaches , methods , metrics , measures, and subjectivity in ontology evaluation : A survey3(i) Structural metrics - those that are concerned withthe syntax and semantics aspects, (ii) Functional met-rics - those that are focused on the intended use of theontology and its components (basically, their functionin a given context), and (iii) Usability-profiling - thosethat are focused on the communication aspect of anontology ( are also reliant of the ontology annotationsand are reminiscent of the pragmatic quality of Burton-Jones [6]).

10 Ontology evaluation measures:Perspective,criteria, metricsAfter an analysis of the state of the art in ontologyevaluation, this work perceives ontology evaluation tobe done in the view of two complementary perspective:(i) Ontology Quality, and (ii) Ontology quality perspectiveMost research on on-tology evaluation considers ontology evaluation fromthe ontology quality perspective. Burton-Jones etal. [6] s approach discussed earlier epitomizes thisquality-oriented view to ontology evaluation . They dis-cuss a software engineering-influenced view to ontol-ogy quality where internal attributes (which are usu-ally directly measurable) influence the external qualityattributes (which are not usually directly measurable).Ontology correctness perspectiveAccording to Guar-ino and G mez-P rez [20,17], ontology evaluation isdefined in the view depicted in Figure 1.