Relative difficulty of examinations in different subjects

1 Report for SCORE (Science Community Supporting Education), July 2008 Relative difficulty of examinations in different subjects Robert Coe, Jeff Searle, Patrick Barmby, Karen Jones, Steve Higgins CEM Centre, Durham University 2 Executive Summary 1. This report reviews the evidence on whether examinations in some subjects can legitimately be described as harder than those in other subjects , and, if so, whether STEM subjects (those that form a foundation for further study in science, technology, engineering and mathematics) are generally more difficult than others.

2 2. A number of different statistical methods have been used to try to address these questions in the past, including Subject Pairs Analysis (SPA), Kelly s (1976) method, the Rasch model, Reference Tests and value-added (including multilevel) models. 3. Evidence from the existing statistical analyses conducted in the UK appears to show a high level of consistency in their estimates of subject difficulties across methods and over time, though the publicly available evidence is somewhat limited. Nevertheless, there is a clear indication that the STEM subjects are generally harder than other subjects .

3 4. A separate body of literature has examined the subjective perceptions of difficulty of candidates in different subjects . There is evidence that sciences are often perceived as more difficult. 5. In new analyses conducted for this report, we applied five different statistical methods to two national datasets for England from 2006 examinations , comparing difficulties of 33 A-level subjects and of 34 subjects at GCSE. Agreement across methods was generally high. 6. Analysis conducted by the ALIS project on A-level Relative difficulties every year since 1994 shows that these are highly stable over time, and especially so since 2002.

4 7. When GCSE examination results are analysed separately for different subgroups, such as males and females, there are some differences in the Relative difficulties of different subjects . Other subgroup splits, by Free School Meals status and Independent/Maintained school sector, show smaller differences and are more problematic to interpret. At A-level, subgroup differences appear to be much smaller. 8. At A-level, most methods put the range between the easiest and hardest subjects at around two grades, though the use of the Rasch model suggests that this range is a lot larger at the lower grades than at the top, and that grade intervals are far from equal.

5 The STEM subjects are all at the top end of this difficulty range. 9. At GCSE, most methods put the range between the easiest and hardest subjects at around one-and-a-half grades, though one subject (short course IT) is an outlier, so the range is about one grade for the majority of subjects . Again the Rasch model shows that grade intervals are far from equal and the difficulty of 3 a subject depends very much on which grade is compared. There is a tendency for STEM subjects to be more difficult on average, though this is less marked than at A-level. 10. A number of objections have been made in the past to the simplistic interpretation of statistical differences as indicating subject difficulties, and we discuss these.

6 Although it is possible to argue that statistical differences are meaningless, there are at least three possible interpretations of such differences that we believe are defensible. 11. Given the evidence about the Relative difficulties of different subjects , we believe there are three possible options for policy: to leave things as they are; to make grades statistically comparable, or to adjust them for specific uses. These three options are presented and discussed. 4 CONTENTS Part I: Introduction ..11 1. Introduction ..12 A brief historical outline of the controversy over subject difficulties.

7 12 Overview of the report ..15 2. Methods for comparing difficulties ..17 Statistical Subject Pairs Analysis (SPA)..17 Common examinee linear models ..18 Latent trait Reference tests ..22 Value-added Judgement methods ..25 Judgement against an explicit standard ..26 Judgement against other scripts ..26 How examination grades are awarded ..27 The grade awarding process in England, Wales and Northern The grade awarding process in Scotland ..29 Raw Scores and Grade Boundaries ..30 Interpreting statistical differences in Problems with statistical Interpretations of statistical differences.

8 32 Part II Existing Evidence ..34 3. Evidence from existing comparability studies ..35 Early work (up to the 1980s)..35 Osborn, L. G. (1939) - Relative difficulty of High School Nuttall, , Backhouse, and Willmott, (1974) - Comparability of standards between subjects ..35 WJEC reporting on 1971 O-level WJEC reporting on 1972 O-level Kelly, A. (1976a) - A study of the comparability of external examinations in different subjects : Scottish Higher examinations 1969-1972 ..38 5 Newbould, (1982) - Subject Preferences, Sex Differences and comparability of Standards.

9 39 Newbould, C. A. and Schmidt, C. C. (1983) - Comparison of grades in physics with grades in other subjects : Oxford & Cambridge Forrest G. M. and Vickerman C.( 1982) - Standards in GCE subject pairs comparisons 1972-80: JMB exam More recent studies (from the 1990s onwards) ..41 Fitz-Gibbon, C. and Vincent, L. (1994) - Candidates performance in mathematics and science ..41 Alton, A. and Pearson, S. (1996) - Statistical Approaches to Inter subject Comparability: 1994 A-level data for all boards, part of the 16+/18+ project funded by Dearing, R. (1996) - Review of qualifications for 16-19 year olds.

10 46 Pollitt, A. (1996) - The difficulty of A-level Patrick, H. (1996) - Comparing Public examinations Standards over Wiliam, D. (1996a) - Meanings and Consequences in Standard Setting ..50 Wiliam, D. (1996b) - Standards in examinations : a matter of trust? ..50 Goldstein, H. and Cresswell, M. (1996) - The comparability of different subjects in public examinations : a theoretical and practical critique ..51 Fitz-Gibbon, C. and Vincent, L. (1997) Difficulties regarding subject difficulties ..51 Newton, P. (1997) - Measuring Comparability of Standards between subjects : why our statistical techniques do not make the grade.