PhD thesis quality: The views of examiners Keynote …

1 PhD thesis quality: The views of examiners Keynote address: Sid Bourke, university of newcastle . australia 1 stellenbosch university , South Africa Abstract The PhD is clearly the pinnacle of formal education qualifications internationally, representing excellence and attracting both resources and prestige to universities. The expectation is that each PhD project will provide an original and significant research outcome, presented in the form of a thesis . In australia , normally the only assessment for the award of a PhD is through the medium of written reports on the thesis by either two or three external examiners . The examiners effectively set PhD standards. The major focus of our work has been on the content of the examiner reports in providing indicators of thesis quality, and the bases for their recommendations on the fate of the thesis .

2 These reports were also linked with candidature and examiner information to provide background information thought to be relevant to the thesis examination process. The study discussed here adds considerable empirical information to the literature in the important and neglected area of doctoral assessment. On the basis of 2121 examiner reports for 804 theses across all discipline areas at eight Australian universities, discipline differences, examiner consistency, and thesis quality are discussed. The incidence and proportion of examiner comments devoted to (1) themselves and the context of their examination of the thesis , (2) to assessable areas covered, (3) dialogic elements of the reports, and (4) to the evaluative elements of their reports are described.

3 Information about what examiners comment on in their reports, and therefore presumably value, should be of considerable use to candidates and their supervisors in assisting them to meet examiner expectations of theses more effectively. In general, the nature of examiner comments do not differ by discipline, although some candidature characteristics do differ in this way, eg, nature of enrolment and time taken to completion. examiners of the same thesis are usually quite consistent in the nature of the comments they make, although they more often diverge in their final recommendations. It would seem that the same issues in a thesis are identified by examiners , although the extent of change they recommend or demand may vary rather more.

4 Because examiner recommendations are intended for another specific purpose, there are problems in identifying thesis quality from the recommendations, except in distinguishing between the best and the marginal theses. However, more fine-grained distinctions between theses would seem to be possible from a detailed examination of the written comments. Our work is proceeding along these lines. 1 This paper is based on results of an Australian Research Council Discovery Grant awarded to Allyson Holbrook, Sid Bourke and Terence Lovat, and subsequent work undertaken by Sid Bourke. INTRODUCTORY REMARKS Despite differences in the nature of PhD programs and examination internationally, the doctorate is clearly recognised as the pinnacle of our formal educational systems and has been referred to as the gold standard (Scott et al, 2004, ).

5 And the quality of the PhD remained largely unquestioned until recently when there has been a large increase in the numbers undertaking doctorates in what has been termed the massification of doctoral qualifications. As there is no PhD curriculum as such, it is the thesis examiners who set the standard of what is acceptable as a thesis (or dissertation), and consequently set the standard for the award of a PhD. To a large extent the examiners also influence what style of reporting may be used and what should be included. What examiners do in examining is crucial. A few years back it could be said, with authority, that little research had been undertaken into the written assessments of PhD theses (Morley, Leonard & David, 2002; Mullins & Kiley, 2002; Tinkler & Jackson, 2000).

6 In the UK, the major focus in doctoral assessment has been on the oral examination or viva (see, for example, the work by Trafford, 2003 and by Tinkler & Jackson, 2004). With respect to assessment of the written thesis itself, there is a recent and developing literature on examination criteria and process based on interviews with either experienced examiners (Mullins & Kiley, 2002), inexperienced examiners (Kiley & Mullins, 2004) or on questionnaires completed by examiners (Winter et al, 2000). Interest in the USA concerning dissertation quality (Lovitts, 2007) and doctoral assessment (Maki & Borkowski, 2006) is even more recent. But there were a few prior studies of thesis assessment based on examiner reports (Ballard, 1996; Hansford & Maxwell, 1993; Johnston, 1997; Nightingale, 1984; Pitkethly & Prosser, 1995).

7 In the main these were single-institution studies. Only Hansford & Jackson (1993), which examined reports on Masters level theses, identified the literature review and conceptual framework as substantial themes. In this study 34 per cent of examiners made critical comment on the literature review. The two most common criticisms were the failure to use recent literature and the inability to critically assess the existing literature ( ). The study drew on 125 students in Education in one institution over a four year period. The authors noted that if their study had endeavoured to identify the nature of examiner comment relating to high quality theses, their belief was that a number of such comments would refer to aspects of theoretical or conceptual frameworks ( ).

8 Despite these few studies of various aspects of thesis examination, there remains a dearth of direct information about what examiners value as indicators of thesis quality. The most direct source of such information is the content of the written examiner report itself these reports can be analysed in detail to determine how important each component is in the assessment regime. The study reported below involved direct and detailed examination of examiners reports on PhD theses. OUR STUDY A total of 2121 examiner reports on a total of 804 PhD theses across all discipline areas (Broad Fields of Study) at eight Australian universities (with varying research intensiveness) were collected. The associated examiner recommendation on the thesis and information about the candidates, their candidature and their supervision were also collected.

9 The content of the reports were coded across specific categories (described below), and the report length was also recorded. I will return to the examiner reports shortly, but first I will describe the samples of candidates and examiners and relate these characteristics to the examiner recommendations made. The distributions of candidates and examiners across disciplines are shown in Table 1. INSERT TABLE 1 ABOUT HERE There were initially 10 Broad Fields of Study (BFOS) recorded. However, as the numbers of theses in some areas, notably Architecture, Law and Veterinary Science, were small, theses in these areas were included with the fields with which they had the most affinity, Built Environment, Business and Science respectively.

10 It will be noted that the broad field of Science had the largest number of theses included, followed by the Arts, Humanities and Social Sciences grouping. The majority of Australian universities use three external examiners for PhD theses, others use two. In the main, reports are provided to the university independently by each examiner, without prior consultation between examiners . This is a strength of the thesis examination regime the university has multiple and independent assessments available to it when making a final decision on the fate of the thesis . Each examiner recommendation on the thesis and the university decision were recorded for this study. In most cases the decision is made by a university -wide committee, in other cases where there is little difference between examiner recommendations, the decision may be made by the Dean of Graduate Studies (or equivalent).