the Key Features - DME

1 Assessing Clinical Decision Making:Focusing only on the critical, challenging decisions, the Key FeaturesGeorges BORDAGE, MD, PhDDepartment of Medical EducationCollege of MedicineUniversity of Illinois at ChicagoSchool of Medicine Chiba University, March 2007 G. Bordage, UIC, Chicago, change? s Key Features approach?Basic issues : -Adequate&representativesample problems-Critical decisions-Format (how)serves purpose(what)Slides FeaturesProblemsPMPs: Patient Management ProblemsChristine McGuireCED-UIC 1960s Paper & pencil test Clinical scenario(CC)+ Sections(H&P, Lab. invest., Dx, Managt., F-up)[L]Options || [R]Latent images answersSection C 1. Alanine Aminotransferase (ALT) 2. Alcohol level 3. Aldolase, serum 4. Alkaline phosphatase, serum 5. Amylase, serum 6. Arterial blood gases (ABG) 7. Aspartate Aminotransferase (AST) 8. Brain CT-scan 9. Brain MRI 10. Brain PET-scan 11. Calcium, serum 12. Carotid US-doppler 13.

2 Cerebral angiography 14. Cerebro-spinal fluid examination 15. Complete Blood Count (CBC) 16. C-Reactive Protein 17. Creatine Phophokinase, serum 18. Creatinine, serum 19. Drug screening, serum 20. Drug screening, 1. 2. 3. 4. 5. 6. 7. 8. No abnormalities noted 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. ( mg/dL) 19. Salicylate: 32mg/dL (20-25 mg/dL) image7 PMPs: Patient management ProblemsQ3-hour, -day: 10-12 casesQThe more good things,the more THOROUGH,the higher the score81984 : Cambridge ConferencePMPsDx: 5 ailments95 ailments re: PMPs/Low content validity: 3 to 10 reliability (consistency): .3/Problem solving = General skill/Unique format (latent image): cueing/Over-rewarding thoroughnessThoroughnesspredictor of poor performanceElstein, Shulman & Sprafka, 1978 When in doubt, collectingmore data(EKG Features )- not improve Dx accuracy- indicator of uncertainty,Dx errorHatala et al, 1998 PMPs: more THOROUGH, higher wrong behaviorProblem solving in medicine is:Not a general skillSpecific toeach case13 Case specificityElstein, Shulman & Sprafka, 1978 Inter-case correlation =.

3 1 - .3 Arthritis Anemia Crohn Eclampsia Each case presents unique challengesKey Features (KFs)14 Csqs for assessment:QFocus on the KFsfor wide rangeof problemsQMany short, KF problemsContent validityReliability15 Object (purpose)assessmentKnowledge recall (describe DT)Clinical reasoning, problem solvingClinical decisions, actions(recognize & manage DT)H&P, Dx, Rx, Investigation, F-up 1986-91 Medical Council of CanadaQualifying (MD) exam graduating clerks5-yr R & D : Q4 Project Page & Bordage 1992: Replaced PMPs with KF cases 2002: 12-yr review17 A clinical scenario, with age & clinical situation specified:Severe (life-threatening) respiratory distress in an infant .. Typically followed by 2 or 3 questions Assessing onlyunique challenges("key Features ")or critical decisions and actions in the resolution of the problem (not underlying knowledge or reasoning) Paper & pencil or OSCEWhat is a KF case?183 test material(cases & questions)& set scoring key19I- Select problemsAdequate&representativenumberof clinical problemsfrom the domainfor graduating students20Ia- Domain of clin.

4 ProblemsMedical Council of Canada Objectives for the Qualifying Exam(3 rded., 2004, Dauphinee & Mandin)120 primary clinical presentations140 related clinical presentations260 clinical presentations21By alphabetical order Dysphagia/Difficulty swallowing 26-E Dyspnea 27-EAcute dyspnea (minutes to hours) 27-1-E Chronic dyspnea (weeks to months) 27-2-E Pediatric dyspnea Resp. distress 27-3-E Scrotal Mass Scrotal Pain Seizures (Epilepsy)SD22By disciplines Primary Care Medicine Ob-Gyne Population, ethics, legal, org. Peds Psychiatry Surgery23 Each clinical presentation Rationale Causal conditions Key objectives Objectives Ethics Applied sc. concepts24Ib- How many problems, pts? Inter-case corr. = .1 - .3 Desired reliability = Spearman-Brown Formula40problems25Ic- In what proportion? Blueprint Preg., neonat., infant5%*3 Children (Peds)16%6 Adolescents16%6 Adults47%19 Elderly (geriatrics)16%6* Health Services Data40 Age groups26 Test Committee ProcessI- Chair randomly selects problemsII- Assigns a problem to a memberto defineKFsDiscussion with committeeIII- Member writes test case & with (epilepsy) Rationale Causal Conditions Key treatment of ObjectivesStatus Epilepticus Ethics Applied Sc.

5 Concepts28II- Define KFsFor what clinical situation?Seizures: - Undifferentiated complaint- Life-threatening Clinical situation Undifferentiated complaint Simple, typical/ atypical Multiple, multi-system Urgent, life-threatening Prevention, health promotion30II-e Define KFs+ Unique challenges, criticaldecisions or steps inthe resolution of the problem- Most difficult aspect in practiceSteps, actions most likelyto lead to errorGiven man w/suspected alcohol dependencebrought to ER w/multiple seizures w/oregainingconsciousness, graduating medical studentshould:KF-1 Generate provisional Dx status epilepticusKF-2 Secure & maintain cardio-resp. functionsKF-3 Begin initial therapy:NS, vitB, glucose, diaz+phenKF-4 Elicit Hx re: causes:alc., meds, drugs, diabetesKF-5 Order imm. exams:lytes, gluc., Ca, ABG,drug, brain CTNendaz & Bordage, 1997, 200332 Test Committee ProcessI- Chair randomly selects problemsII- Assigns a problem to a memberMember select situation & defineKFsDiscussion with committeeIII- Member writes test case & with of KFsClerkship directors from acrossCanada confirmed:-Existing KFs92%- Generating KFs 94%34 III- Write test clinical scenariore: problem, situation, test questionsre: KFs response scoring key35 KFs dictate case scenario Given man w/suspected alcohol dependence brought to ER w/multiple seizures w/oregaining scenario includes:CC, some Hx, initial physical Mr.

6 X, a 36-year-old man, is brought to the emergency room in your hospital by ambulance because he fell to a sidewalk unconscious while waiting for the bus. A witness immediately called an ambulance and reported to the ambulance crew that before falling to the ground, he seemed confused, agitated, and was arguing with some invisible person. After falling, he began to twitch for a shortwhile, his face became blue, and then he began to have jerky movements all over his body for about a minute. He did not recover consciousness after the episode. During the 10-minute ambulance trip, he presented two other similar episodes, without recovering consciousness, and a third episode that you witnessed on arrival. His temperature is C. He looks neglected and is unconscious. No relatives or friends accompanied Mr. X. 37 III-b Test questions KFsKF-1 Generate provisional Dx status epilepticusKF-2 Secure & maintain cardio-resp.

7 FctsKF-3 Begin initial therapy:NS, vitB, glucose, diaz+phenKF-4 Elicit Hx re: causes:alc., meds, drugs, diabetesKF-5 Order imm. exams:lytes, gluc., Ca, ABG,drug, brain CTGenerally 1 question/ KF38 Test questions Question 1: What is (are) your leading working diagnosis(es) at this point in time? You may list up to two. Question 2: What is your immediate management at this point in time? List as many things as you feel are & 339 Test questions Question 3: Ten minutes after arrival, Mr. X is still unconscious. The nurse found a telephone number in his wallet that you decide to call immediately. What questions will you ask the person answering the phone assuming he/she knows the patient? You may select up to six questions. Select option 35 if you think that it is not appropriate to call at this point in 319. Lung infection20. Medication history21. Muscular disease22. Nausea23. Palpitation history24. Pet in household25.

8 Previous similar problem26. Profession27. Sexual history28. Smoking history29. Social integration difficulties30. Surgery31. Travel history32. Viral infection33. Visual impairment34. Vomiting35. Not appropriate to call at this pointin Abdominal pain2. Alcohol history3. Back pain history4. Benzodiazepine5. Cancer history6. Cocaine abuse7. Coronary bypass history 8. Diabetes history9. Diarrhea10. Dizziness11. Drug allergy12. Family history13. Food allergy14. Headache15. Hearing disability16. Heroin abuse17. Joint pain18. LSD abuse41 Test questions Question 4: It has been 15 minutes since Mr. X's arrival. What ancillary exams would you order at this point? You may select as many as you feel appropriate. Select option 35 if you think that ancillary exams are not needed at this point in KFs matrixKF1KF2KF3KF4KF5Q1 Q2 Q3 Q4 43 III-c Response format Write-in (WI): short answer Short menu (SM): 15 - 35 options Long menu (LM): bookletWhich is bestAfter initial management of this patient, what will you do?

9 WISM GI consultation1 12 Wash hands376 Notify h. authorities2563 Stop work482 ..50607080 FailBorder line FailBorder line PassPassPercentage ScoreSMWIP erformance on Open Response (WI) vs. Selected Response Items (SM)46WI vs. SM Nbr responses WI < SM (-14%; cueing) DifficultyWI > SM (-18pts; 54 72) VarianceWI > SM Discrimination WI > SM Marginal cand. WI > SMSM:H&P, Lab. & InvestigationWI:Dx & Rx, Management47 III-d Scoring (points) Dichotomous : 0 / 1 Partial credit (fraction):-Equal weights (3 resp.: .33 each)- Differential w. (.50 +.33 +.33)48 Scoring: Partial creditKF - 3 Begin initial +Phen . mentioning above123 Scoring Problems:(unit of measurement)- item independence -Average KF scores (KFs equal weight) , (1 + .5 + .84 + .5 + .58)/5 = Test as a whole:Average problem scores (equal weight) , (.67 + . )/32 = .78 = 78% Passing score:Content (criterion) - based approach(modified Angoff; 30 )50 Reliability PMPs: -day.

10 3 - .5 KF exams (1/2-day; 32-36 cases): ~.65 Spearman Brown Formula:.80 45-50 cases = 1 dayConsistency, reproducibility51Qs/ case Reliability went downwith single-q. cases Generalizability study;maximize reliabilitywith2 -3 q. / case1 question/ case, not enough>3 redundant, wasting testing timeNorman et al, 200652 References*Sirven, JI & Waterhouse, E. Management of status Fam Physician2003;68:469-76* EBM: Evidence based medicine53 KFs : Case specificity ..problem-solving skills are specific to the case or problem are contingent on the effective manipulation of those few elements of the problem that are critical to its successful the problem s key Features . Page & Bordage, , 1995 Why KFs? High fidelitytest of applied knowledge Focus on case-specific decisions: KFs Broader sampling:adequate & reliability& validity Simple & focused scoring: only re: KFs Varied formatsfit purpose(avoid cueing) Defensible pass-fail decisions Best predictor of future complaintsDissemination Medical schools acrossCanada 1991 Coll ge des M decins du Qu bec(SOI) 1993 College Physicians & Surgeons of Pakistan 1995 Amer.