Learning in practice - bmj.com

1 Learning in practice BMJ: first published as on 12 April 2003. Downloaded from on 26 November 2018 by guest. Protected by copyright. Effect of ethnicity on performance in a final objective structured clinical examination : qualitative and quantitative study Val Wass, Celia Roberts, Ron Hoogenboom, Roger Jones, Cees Van der Vleuten Department of General practice Abstract there is little further published work on the impact of and Primary Care, differences in ethnicity on performance in examina- Objective To assess the effect of ethnicity on student This is becoming an increasingly important Guy's, King's, and St Thomas's School of performance in stations assessing communication issue in undergraduate assessment. Fairness and Medicine, London skills within an objective structured clinical consistency of assessment across UK medical schools is SE11 6SP. examination . We need to understand any source of potential Val Wass senior lecturer in Design Quantitative and qualitative study.

2 Bias that may lead to racial disadvantage when general practice Setting A final UK clinical examination consisting of developing tests for these skills. Roger Jones a two day objective structured clinical examination When looking for potential discrimination within Wolfson professor of general practice with 22 stations. examinations, standardisation is a key issue; the more Participants 82 students from ethnic minorities and standardised the content, the less the potential for bias. Department of Education and 97 white students. Objective structured clinical examinations are cur- Professional Main outcome measures Mean scores for stations rently most often used to assess undergraduate skills Studies, King's (quantitative) and observations made using discourse College, London and include standardised simulated scenarios to test SE11 6SP analysis on selected communication stations communication 10 Yet it is still difficult to achieve Celia Roberts (qualitative).

3 True 12 However carefully designed, the senior research fellow Results Mean performance of students from ethnic scenario presented to students will vary since neither Department of minorities was significantly lower than that of white simulated patient nor student is speaking from scripts. Educational students for stations assessing communication skills Development and Examiners and simulated patients make judgments Research, University on days 1 ( (SD ) and ( ); based on an impression of how well the student man- of Maastricht, P= ) and 2 ( ( ) and ( ); aged the consultation. This judgment, in turn, will be Maastricht, Netherlands P= ). No examples of overt discrimination were informed by their assumptions about what makes an Ron Hoogenboom found in 309 video recordings. Transcriptions showed effective research assistant subtle differences in communication styles in some We aimed to investigate whether students from Cees Van der students from ethnic minorities who performed ethnic minorities are disadvantaged by a bias in mark- Vleuten professor of education poorly.

4 Examiners' assumptions about what is good ing in a final year objective structured clinical communication may have contributed to differences examination , with a particular focus on stations assess- Correspondence to: V Wass in grading. ing communication skills. @ Conclusions There was no evidence of explicit discrimination between students from ethnic minorities and white students in the objective Methods BMJ 2003;326:800 3. structured clinical examination . A small group of male students from ethnic minorities used particularly Our study took place in June 1999 during the final poorly rated communicative styles, and some subtle MBBS examination of the then Guy's and St Thomas's problems in assessing communication skills may have medical school. This comprised a three and a half hour introduced bias. Tests need to reflect issues of diversity objective structured clinical examination conducted to ensure that students from ethnic minorities are not over two days, consisting of two stations for history tak- disadvantaged.

5 Ing of long cases (21 minutes each) and 20 stations (seven minutes each) for clinical examination (nine sta- tions), communication skills (six), and practical skills Introduction (five). The stations were similar but not identical on the Students from ethnic minorities seem to perform less two days. Simulated patients were professionally well overall than white students in both undergraduate trained to standardise the scenarios used on communi- and postgraduate medical 4 Any form cation stations. of potential racial discrimination within our examina- A different examiner marked each station against a Extracts from tion systems is a cause for 6 Problems with checklist and gave a final five point global rating for transcripts appear complex discourse may disadvantage students in oral overall clinical competency. Simulated patients on examinations who have been trained overseas, but awarded a five point global rating for overall 800 BMJ VOLUME 326 12 APRIL 2003 Learning in practice Table 1 Communication scenarios BMJ: first published as on 12 April 2003.

6 Downloaded from on 26 November 2018 by guest. Protected by copyright. Day 1 Station 8 Station 18. Day 1 Explain to a rather obstinate, elderly, middle class white woman that her chest x Take a sexual history from a young Muslim student who had had unprotected ray film showed a possible lung metastasis. Bronchoscopy had been casual sex at a party. She was concerned she might have caught something. recommended. She denied that the cancer may have returned and just wanted She felt very upset about this accidental break with her cultural tradition and antibiotics for her cough loss of her virginity Day 2 Assess a Chinese businessman who has come for the results of liver function Negotiate with a young Afro-Caribbean man who wants a methadone prescription tests. These indicate he may be drinking too much (this role player was asked to because he says he has lost the one given to him at the drug rehabilitation present himself as not entirely fluent in English) centre communication skills, independent of the examiner.

7 The examiners and simulated patients had been Table 2 Analysis of ethnicity by sex and age for 175 students of known ethnicity briefed on the procedure. A minimum competence Mean age score for each station was set in advance, using the Ethnicity Male Female (years) Age range Total (%). White 50 47 23-32 97 (55). Angoff standard setting South Asian* 31 27 22-30 58 (33). Each day we selected two communication sce- Afro-Caribbean 1 4 23-32 5 (3). narios, using role players from different ethnic Other 6 9 23-29 15 (9). backgrounds (table 1). The students gave informed Total 88 87 22-32 175 (100). verbal consent for video recording the scenarios. The *Indian, Pakistani, Bangladeshi, Chinese, and Asian other. local research ethics committee approved our study. Details of the students' ethnicity were made available after the examination . of the remaining 175 students. Seventy eight (45%). The students were grouped as white, south Asian were from ethnic minorities.

8 All but two students had (Indian, Pakistani, Bangladeshi, Chinese, and Asian received secondary school education in the United other), Afro-Caribbean, and other. For the purpose of Kingdom. our study, all students from ethnic minorities were cat- Table 3 shows the mean performance of the egorised as one group, ethnic minorities (82 students in the overall examination and in the specific students) and all other candidates as white majority study stations. Mean performance on communication (97 students). stations was significantly higher for white majority stu- dents than for students from ethnic minorities on both Quantitative analysis day 1 ( (SD ) ( ); P= ) and day We analysed the mean performance for stations for 2 ( ( ) and ( ); P= ). The each day on all 22 stations, stations grouped by Cronbach reliability of the objective structured clini- communication, practical, clinical skills, and long cases, cal examination was and on days 1 and 2, and the specific study stations.

9 We used an independ- respectively. ent two sample t test to examine relations between stu- We were unable to assess 49 (14%) of the video dent performance and ethnicity. We regarded P values recorded interactions due to technical faults. We found greater than as non-significant. The reliability of no explicit examples of breakdown in communication each objective structured clinical examination was cal- or of discriminatory behaviour in the remaining 309. culated with Cronbach's . Qualitative analysis Table 3 Mean scores, standard deviations, and T and P values (independent two All video recorded encounters were viewed as well as sample t test) for students from white majority compared with students from ethnic recorded comments made by simulated patients and minorities for components of objective structured clinical examination on days 1 and 2. examiners after students had left the station. The dura- White majority group Ethnic minority group Comparison of means*.

10 Tion of the interaction, student's ethnicity, and observa- Exam component No of Mean (SD) No of Mean (SD) T P. tions made during the viewing were recorded on a (No of stations) students score (%) students score (%) value value standard form. An assessment was made of the extent Total examination (22): to which simulated patients and students established a Day 1 49 ( ) 35 ( ) relatively patient centred encounter. Any potential Day 2 48 ( ) 43 ( ) misunderstandings, false assumptions, or explicit Communication (6): discriminatory behaviour were noted. The analyst (CR) Day 1 49 ( ) 35 ( ) viewed the encounters blind, allocating grades and Day 2 48 ( ) 43 ( ) comparing these after the viewing with the ratings clinical (9): from the simulated patients and examiners. Discrepan- Day 1 49 ( ) 35 ( ) Day 2 48 ( ) 43 ( ) cies between analyst, simulated patient, and examiner Practical (5): were recorded. Day 1 49 ( ) 35 ( ) We used these records to select specific interactions Day 2 48 ( ) 43 ( ) for detailed transcription.