New evidence pyramid - BMJ Evidence-Based Medicine

1 New evidence pyramidM Hassan Murad, Noor Asi, Mouaz Alsawas, Fares AlahdabAbstractA pyramid has expressed the idea of hierarchy ofmedical evidence for so long, that not all evidence is thesame. Systematic reviews and meta-analyses have beenplaced at the top of this pyramid for several goodreasons. However, there are several counterarguments tothis placement. We suggest another way of looking atthe Evidence-Based Medicine pyramid and explain howsystematic reviews and meta-analyses are tools for con-suming evidence that is, appraising, synthesising andapplying and earliest principle of Evidence-Based medi-cine indicated that a hierarchy of evidence exists. Notall evidence is the same. This principle became wellknown in the early 1990s as practising physicians learntbasic clinical epidemiology skills and started to appraiseand apply evidence to their practice.

2 Since evidence wasdescribed as a hierarchy, a compelling rationale for apyramid was made. Evidence-Based healthcare practi-tioners became familiar with this pyramid when readingthe literature, applying evidence or teaching versions of the evidence pyramid have beendescribed, but all of them focused on showing weakerstudy designs in the bottom (basic science and caseseries), followed by case control and cohort studies inthe middle, then randomised controlled trials (RCTs), andat the very top, systematic reviews and description is intuitive and likely correct in manyinstances. The placement of systematic reviews at the tophad undergone several alterations in interpretations, butwas still thought of as an item in a ver-sions of the pyramid clearly represented a hierarchy ofinternal validity (risk of bias).

3 Some versions incorpo-rated external validity (applicability) in the pyramid byeither placing N-1 trials above RCTs (because their resultsare most applicable to individual patients2) or by separat-ing internal and external version (the 6S pyramid ) was also developedto describe the sources of evidence that can be used byevidence-based Medicine (EBM) practitioners foranswering foreground questions, showing a hierarchyranging from studies, synopses, synthesis, synopses ofsynthesis, summaries and hierarchy mayimply some sort of increasing validity and applicabilityalthough its main purpose is to emphasise that the lowersources of evidence in the hierarchy are least preferredin practice because they require more expertise and timeto identify, appraise and apply.

4 , Minnesota, USAC orrespondence to:Dr M Hassan Murad, Evidence-Based PracticeCenter, Mayo Clinic,Rochester, MN 55905, Based MedAugust 2016| volume 21| number 4|125 Perspective on June 24, 2022 by guest. Protected by Based Med: first published as on 23 June 2016. Downloaded from The traditional pyramid was deemed too simplistic attimes, thus the importance of leaving room for argumentand counterargument for the methodological merit ofdifferent designs has been barrierschallenged the placement of systematic reviews andmeta-analyses at the top of the pyramid . For instance,heterogeneity (clinical, methodological or statistical) isan inherent limitation of meta-analyses that can be mini-mised or explained but never methodo-logical intricacies and dilemmas of systematic reviewscould potentially result in uncertainty and of 163 meta-analyses demonstrated that theestimation of treatment outcomes differed substantiallydepending on the analytical strategy being , we suggest, in this perspective, two visualmodifications to the pyramid to illustrate two contempor-ary methodological principles (figure 1).

5 We provide therationale and an example for each for modification 1In the early 2000s, the Grading of RecommendationsAssessment, Development and Evaluation (GRADE)Working Group developed a framework in which the cer-tainty in evidence was based on numerous factors andnot solely on study design which challenges the design alone appears to be insufficienton its own as a surrogate for risk of bias. Certain meth-odological limitations of a study, imprecision, inconsist-ency and indirectness, were factors independent fromstudy design and can affect the quality of evidencederived from any study design. For example, ameta-analysis of RCTs evaluating intensive glycaemiccontrol in non-critically ill hospitalised patients showeda non-significant reduction in mortality (relative risk (95% CI to )9).

6 Allocation concealment andblinding were not adequate in most trials. The quality ofthis evidence is rated down due to the methodologicalimitations of the trials and imprecision (wide CI thatincludes substantial benefit and harm). Hence, despitethe fact of havingfive RCTs, such evidence should not berated high in any pyramid . The quality of evidence canalso be rated up. For example, we are quite certain aboutthe benefits of hip replacement in a patient with disab-ling hip osteoarthritis. Although not tested in RCTs, theFigure 1 The proposed new Evidence-Based Medicine pyramid . (A) The traditional pyramid . (B)Revising the pyramid : (1) lines separating the study designs become wavy (Grading ofRecommendations Assessment, Development and Evaluation), (2) systematic reviews are chopped off the pyramid .

7 (C) The revised pyramid : systematic reviews are a lens through whichevidence is viewed (applied).126 Evid Based MedAugust 2016| volume 21| number 4|Perspective on June 24, 2022 by guest. Protected by Based Med: first published as on 23 June 2016. Downloaded from quality of this evidence is rated up despite the studydesign (non-randomised observational studies).10 Therefore, thefirst modification to the pyramid is tochange the straight lines separating study designs in thepyramid to wavy lines (going up and down to reflect theGRADE approach of rating up and down based on thevarious domains of the quality of evidence ).Rationale for modification 2 Another challenge to the notion of having systematicreviews on the top of the evidence pyramid relates tothe framework presented in theJournal of the AmericanMedical AssociationUser s Guide on systematic reviewsand meta-analysis.

8 The Guide presented a two-stepapproach in which the credibility of the process of a sys-tematic review is evaluatedfirst (comprehensive litera-ture search, rigorous study selection process, etc). If thesystematic review was deemed sufficiently credible, thena second step takes place in which we evaluate the cer-tainty in evidence based on the GRADE words, a meta-analysis of well-conducted RCTs atlow risk of bias cannot be equated with a meta-analysisof observational studies at higher risk of bias. Forexample, a meta-analysis of 112 surgical case seriesshowed that in patients with thoracic aortic transection,the mortality rate was significantly lower in patientswho underwent endovascular repair, followed by openrepair and non-operative management (9%, 19% and46%, respectively, p< ).

9 Clearly, this meta-analysisshould not be on top of the pyramid similar to ameta-analysis of RCTs. After all, the evidence remainsconsistent of non-randomised studies and likely subjectto numerous , the second modification to the pyramid isto remove systematic reviews from the top of thepyramid and use them as a lens through which othertypes of studies should be seen (ie, appraised andapplied). The systematic review (the process of selectingthe studies) and meta-analysis (the statistical aggrega-tion that produces a single effect size) are tools toconsume and apply the evidence by and limitationsChanging how systematic reviews and meta-analyses areperceived by stakeholders (patients, clinicians and stake-holders) has important implications.

10 For example, theAmerican Heart Association considers evidence derivedfrom meta-analyses to have a level A (ie, warrants themost confidence). Re-evaluation of evidence usingGRADE shows that level A evidence could have beenhigh, moderate, low or of very low qualityof evidence drives the strength of recommendation,which is one of the last translational steps of research,most proximal to patient of the limitations of all pyramids and depic-tions of evidence hierarchy relates to the underpinningof such schemas. The construct of internal validity mayhave varying definitions, or be understood differentlyamong evidence consumers. A limitation of consideringsystematic review and meta-analyses as tools toconsume evidence may undermine their role in new dis-covery (eg, identifying a new side effect that was notdemonstrated in individual studies13).