Methodological notes
Medwave 2021;21(4):e8182 doi: 10.5867/medwave.2021.04.8182

The GRADE approach, Part 2: Evidence to decision frameworks—outlining decision-making in health

Carlos Quilodrán, Matías Kirmayr, Bárbara Valente, Javier Pérez-Bracchiglione, Luis Garegnani, Juan Víctor Ariel Franco

Abstract

The GRADE (Grading of Recommendations Assessment, Development and Evaluation) methodology provides a framework for assessing the certainty of the evidence and making recommendations. The Evidence to Decision Framework (EtD) is a transparent and structured system for formulating health recommendations. Once the problem is identified and the certainty of the evidence is assessed, EtD provides several criteria for formulating a recommendation. These criteria include the trade-off between benefits and harms, patients’ values and preferences, acceptability, feasibility, resource use, and impact on equity. The resulting recommendations may differ in strength (strong or weak) and direction (for or against). The process is transparent, allowing other users to adjust the framework of recommendations by modifying the criteria to fit the desired context through an adaptation-adoption process. Given the extensive information available on EtD and the GRADE methodology in general, this narrative review seeks to explain the main concepts involved in decision-making in health by using simplified and friendly descriptions, accompanied by practical examples, thus facilitating its understanding by inexperienced readers.


 

Main messages

  • In addition to assessing the certainty of the evidence, the GRADE methodology seeks to generate recommendations by drawing on Evidence to Decision (EtD) frameworks. These consider the balance between benefits and harms, people’s values and preferences, acceptability, feasibility, and costs associated with the recommendations.
  • Emerging recommendations do not necessarily directly relate to the certainty of the evidence; a high level of certainty in the evidence does not necessarily translate into a strong recommendation and vice versa.
  • Due to the transparency characteristic of the process, adaptation and adoption of recommendations made in other settings are feasible.
  • This article seeks to provide a simplified explanation of the EtD framework in a user-friendly language, oriented to the training of undergraduate and graduate students.

Introduction

This article aims to provide a simplified explanation of the Evidence to Decision (EtD) frameworks to novice readers, as was done in the first part[1] in which the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach to assessing the certainty of evidence was developed. In this second part, the EtD frameworks and their general concepts will be addressed, without particular emphasis on more complex scenarios, detailed at length in the GRADE Handbook itself[2] and other bibliographic resources. EtD frameworks emerge to integrate the three key components of evidence-based medicine in an efficient, systematized, and structured way: best available evidence, clinical expertise/experience, and patient values and preferences[3]. In turn, they seek to standardize, based on transparent and explicit frameworks, how to develop recommendations for health-care decision-making, whether for prevention, diagnosis, or therapy[3]. Should patients with sepsis receive early antibiotics? Should we screen for prostate cancer with prostate-specific antigen? Should we implement a breast cancer screening program with mammography? To make recommendations on questions such as these, one must not only evaluate the best available evidence but also consider additional factors that will be developed throughout this article.

It is important to mention that, although these recommendations can be generated de novo, due to the transparency of the process, they can also be adapted/adopted from others already developed, adapting the criteria to the desired context through an adaptation–adoption process according to the GRADE methodology[4].

This article is the fourth in a methodological series of narrative reviews on general biostatistics and clinical epidemiology topics, which explore and summarize in user-friendly language published articles available in the main databases and specialized reference texts. The series is oriented to the training of undergraduate and graduate students. It is carried out by the Chair of Evidence-Based Medicine of the School of Medicine of the University of Valparaiso, Chile, in collaboration with the Research Department of the University Institute of the Italian Hospital of Buenos Aires, Argentina, and the UC Evidence Center of the Catholic University, Chile. This article aims to describe the EtD frameworks, a component of the GRADE methodology useful to homogenize and improve healthcare decision-making.

Overview of the GRADE methodology

Broadly speaking, the GRADE methodology consists of two main processes: the assessment of the certainty of the evidence and the generation of recommendations. The first is to clearly formulate the question in PICO format (patient/problem, intervention, comparison, and outcomes), considering the different perspectives (relevant subgroups and important context for understanding the question) and why a recommendation or decision is necessary[5]. Subsequently, the evidence for outcomes should be evaluated according to their corresponding studies (e.g., whether they are interventions in clinical trials or observational studies); and synthesize this evidence, ideally in systematic reviews, using the GRADE approach to assess their certainty[6]. This process has been described in the first part of this installment[1]. In a second instance, EtD frameworks seek to formulate recommendations in a structured and transparent manner, considering explicit criteria that support why a particular health decision is proposed. Figure 1 summarizes the complete process of applying the GRADE methodology, from the assessment of the certainty of the evidence to the application of the EtD frameworks.


<b>Figure 1.</b> Summary of the GRADE methodology.
Figure 1. Summary of the GRADE methodology.

Evidence to Decision frameworks and their structure

EtD frameworks are schematic and transparent frameworks for formulating health recommendations, which show, using explicit criteria, the reasons for generating such recommendations[4]. In addition to the certainty of the evidence, they consider the balance between benefits and harms, patients’ values and preferences, and more concrete aspects such as acceptability, feasibility, and costs. The process of applying these criteria for the formation of recommendations is shown in a simplified manner in Figure 2.


<b>Figure 2.</b> Graphical summary of the recommendation formulation process proposed by the EtDs.
Figure 2. Graphical summary of the recommendation formulation process proposed by the EtDs.

The first two criteria (A and B) are closely related because the balance depends on the evidence of benefits and harms, as well as the relative importance of each, defined by the values and preferences of patients. The following two (C and D) are focused on the practical application of the recommendations.

Application of the criteria in the evidence frameworks to the decision

A. The balance between benefits and harms

The GRADE approach evaluates the certainty of the evidence for each of the outcomes (e.g., mortality or quality of life, with their respective certainty). However, to determine the net balance between benefits and harms, several outcomes must be evaluated simultaneously. Therefore, it is important to know the certainty of the evidence for each outcome of benefit or harm and the weight of each outcome in decision-making, according to the perspectives of individuals[8]. The evaluation of this balance between benefits and harms, as well as the other criteria of the EtD framework, can indicate the strength and direction of the recommendation. For example, the early use of antibiotics in sepsis is based on a net benefit balance for patients suffering from this condition (Example 1).

Example 1. The balance between benefits and harms: use of antibiotics in sepsis[9]

Sepsis is a serious medical condition characterized by an inflammatory response to an infection that can affect the entire body, with a very high mortality rate without treatment. Most patients with sepsis require antibiotics and admission to an intensive care unit. In this context, the aim is to reduce the mortality outcome in patients with sepsis. The administration of antibiotics far outweighs the benefit/harm balance, with the adverse effects of antibiotics playing a lesser role in this balance, regardless of the exact time of antibiotic application.

B. People’s values and preferences

The values and preferences of individuals are defined as the relative importance of the outcomes, that is, the value that each individual places on each outcome from his or her perspective. These can be determined directly in the clinical encounter with patients, involving people in making decisions about their health, or developing clinical guidelines, using the available evidence on preferences. The evidence on preference, in turn, can have different levels of certainty, for which the GRADE approach has developed a specific methodology[10],[11].

Systematic study of people’s values and preferences is scarce, and evidence is often not available. However, clinical experience can provide considerable information about typical values and preferences. An example is the knowledge of the strong aversion of pregnant women to the risk of major fetal anomalies when receiving a drug, even when the associated risk is considered small[2]. However, there may be evidence reporting large variability or discordance among individuals’ preferences, and a single recommendation is less likely to apply uniformly to all patients. Therefore, when there is large uncertainty or variability, it is impossible to make a strong recommendation for or against, as this would not apply to the entire target population (Example 2, regarding prostate cancer screening)[8].

Example 2. Weak recommendation: prostate cancer screening according to net balance values and preferences

Prostate-specific antigen screening for prostate cancer screening slightly decreases prostate-specific mortality. However, it causes harm through false positives, overdiagnosis, and overtreatment. For example, treatment of indolent prostate cancers leaves sequelae in urinary and sexual function. Some men may value more the effect on specific mortality, and others may prefer preservation of urinary and sexual function. Considering the magnitude of harms evidenced by this practice and the variability in values and preferences, a weak recommendation against it has been made[12] that clinicians should not routinely consider screening except in specific cases that will require shared decision-making to respect individual preferences.

C. Acceptability and feasibility

An intervention considered beneficial may not be acceptable for various reasons, whether personal, community, or religious; or it may be considered unfeasible because it is incompatible with the local reality for which it was originally intended (Examples 3a and 3b).

Example 3a. Acceptability: the case of blood transfusions[13]

In most cases, a blood transfusion can be a beneficial intervention (with a positive net balance). However, respecting individual autonomy, a person may not accept it on religious grounds, despite recognizing that the balance of benefits and harms is favorable. By considering the acceptability of an intervention, possible barriers to its implementation are anticipated and allow, eventually, to adapt the recommendation for subgroups of people (e.g., by means of autologous blood transfusion).

Example 3b. Feasibility: the case of new technologies[14]

The use of electronic devices can be associated with substantial health benefits in certain scenarios. However, they depend on contingent factors such as telephone and data networks. In turn, an intervention involving the use of electronic devices aimed at people with low digital literacy (e.g., some groups of older adults) will be considered difficult to implement, even if the benefits are clear.

D. Resource use, equity, and cost-effectiveness

There are circumstances in which resource use is a key factor in the development of a recommendation, especially when the balance between benefits and harms is narrow or the implementation of the recommendation has important budgetary implications (Example 4). Similar to values and preferences, there is often little information on the economic implications of recommendations, especially considering the direct and indirect costs involved. The EtD frameworks indicate that some key factors should be considered in estimating resources for the formulation of recommendations:

a) The funder’s perspective in estimating costs.

b) The funder’s perspective on resource use.

c) Local (jurisdictional) variations in cost estimation.

d) The perspective in defining the opportunity cost (what is left out to finance the implementation of the recommendation).

e) Political implications in the allocation of resources.

There are two tools of health economics that allow this criterion to be assessed:

  1. Cost-effectiveness: This domain has the purpose of evaluating what is the relative or incremental benefit of an intervention with respect to another in relation to the associated costs (incremental cost-effectiveness ratio or ICER). The ICER is a relative measure, but it does not estimate the total cost of implementing the recommendation.
  2. Budgetary impact: There are interventions that are cost-effective but involve a large absolute cost, either because the disease is very prevalent or because they involve prolonged treatments, among other reasons.

Finally, when resources are used for one intervention, they are not used for others (opportunity cost), potentially increasing inequalities between individuals, affecting equity. For example, suppose there is a highly effective or moderately effective drug for a group of people within a country whose target population is small, and the drug is expensive. In that case, this intervention can displace the budget allocated to measures with a greater impact on society (vaccines and water sanitation), increasing social gaps by affecting more vulnerable groups[15].

Example 4. Application of criterion D: mammographic screening[16]

In Argentina, the National Cancer Institute used the decision framework questionnaire to establish recommendations on whether or not to perform mammographic screening from a public health perspective, separated by age group.

For women aged 40 to 49 years, the expert panel determined that the resource costs required were extensive (low certainty of evidence), and the cost-effectiveness of the intervention likely benefits the comparison (i.e., do not perform). Therefore, the direction of the intervention recommendation was against screening (strength of recommendation: conditional). Meanwhile, for women aged 50 to 69 years, although the panel considered the resources required involve extensive costs (moderate certainty of evidence), the cost-effectiveness of the intervention favors the intervention. Consequently, the direction of the intervention recommendation was in favor (strength of recommendation: strong).

Overlapping of the different domains

Acceptability, feasibility, and cost-effectiveness depend on efficacy. An intervention will be more acceptable if this same intervention is very effective. In turn, if the intervention is very effective, the cost-effectiveness ratio will be better. There is interdependence between the evidence and the criteria or domains to be considered, which justifies their separation within the EtD framework.

Recommendations

The recommendations formulated by the EtD frameworks can be classified according to their strength (strong or weak) and according to their direction (for or against). To summarize, the strength of a recommendation reflects the degree of confidence that the desired effects of an intervention outweigh the undesirable effects or vice versa. In strong recommendations, the balance between benefits/harm, considering also costs, and patient values and preferences, is clearly determined. In weak recommendations, the opposite is true. It is important to clarify that the term “weak recommendation” can be interpreted negatively by associating the word weak with weak evidence. In order to avoid misunderstandings, the GRADE group proposes to describe weak recommendations with other terms, such as conditional and discretionary. If any of these variations are used, authors must be consistent throughout their guidelines and the rest of the guidelines they produce[2].

It should be noted that even when confidence in the estimate of effect is low and/or the desirable and undesirable consequences are closely balanced, GRADE encourages expert panels to overcome doubts and formulate recommendations, which will inevitably be weak[2]. Table 1 summarizes the significance of these recommendations for patients, clinicians, and policymakers.


<b>Table 1.</b> Summary of interpretations of EtD recommendations.
Table 1. Summary of interpretations of EtD recommendations.

If we take the case of the strong recommendation in favor of early antibiotic use in patients with sepsis (Example 1), it is expected that most patients will receive this medication and that most (if not all) clinicians will take this course of action. Indeed, the proportion of patients with sepsis receiving antibiotics early could be a definite quality-of-care indicator for hospital policy developers.

In contrast, in the case of the weak recommendation against prostate cancer screening (Example 2), there is variability in patient values and preferences, so the decision is expected to be individualized. In this case, decision support tools (decision aids) could be incorporated to assist the dialogue with patients[17], with the use of such tools for communication being a possible quality indicator. The proportion of men screened with PSA could not be an indicator of quality because variable implementation is reasonably anticipated.

Recommendations should not be regarded as rules or rulings. Even strong recommendations based on evidence of high certainty will not apply to all circumstances or all patients. Guideline users may reasonably conclude that following particular recommendations based on high certainty of evidence will be wrong for some patients considering their unique characteristics[18].

Strength and direction of recommendations

There is no direct relationship between the certainty of evidence and the strength of recommendation. The certainty of the evidence is assessed for each outcome; usually, several outcomes are estimated for an intervention, and in addition, all the other criteria already mentioned should be considered. For example, an intervention with high certainty of evidence for its effectiveness may end up with a recommendation against it due to uncertainty about its safety (about its adverse effects). An example of this is the use of quinolones to treat urinary tract infections during pregnancy[19]. Conversely, an intervention for which there is low certainty of evidence for its effectiveness may end up with a strong recommendation in favor of, for example, the duration of handwashing[20].

Adaptation and adoption frameworks

There are many recommendations on prevention, diagnosis, and treatment of health or public health problems developed by teams that systematically apply this methodology. To avoid unnecessary efforts, the GRADE group created the adaptation–adoption frameworks, making it possible to adapt or adopt recommendations made in other countries. For example, in Argentina, the breast cancer screening recommendation (Example 4) was developed from other current recommendations that used the GRADE method using the adaptation–adoption process developed for this purpose[4] (Figure 3).

Initially, we search for relevant guidelines with GRADE methodology, and we can update the evidence that supports them. Then the EtD framework is reassessed in the local context. If the recommendation is similar, it is called “adoption”, if it is different, “adaptation”, and in the case of not having GRADE guidelines, one can choose to develop new recommendations from scratch or “de novo”.


<b>Figure 3.</b> Summary of the process of adaptation/adoption of recommendations.
Figure 3. Summary of the process of adaptation/adoption of recommendations.

Conclusions

The available evidence on the effectiveness of interventions should be considered together with evidence of other types, mainly people’s values and preferences. Specific aspects of the applicability of an intervention, such as acceptability, feasibility, and costs, should also be included. As this process is developed transparently, the EtD frameworks are adoptable and adaptable by other territories.

The certainty of the evidence is not directly proportional to the strength of the recommendation, given the multiple facets of health problems.

As in the first part of our description of the GRADE method, using simple descriptions accompanied by examples, we seek to provide a simplified and summarized approach to this methodology for its understanding and application by the general public, students, and novice methodologists.

Notes

Contributor roles
CQ: conceptualization, methodology, research, writing original draft, review and editing, visualization, project management. MK, BV: conceptualization, methodology, research, writing original draft, reviewing and editing, visualization. JPB, LG: conceptualization, methodology, review and editing, visualization. JAF: conceptualization, methodology, writing original draft, review and editing, visualization.

Acknowledgments
We thank the Chair of Evidence-Based Medicine of the School of Medicine of the University of Valparaíso, Chile, for promoting this series and the collaboration of the Research Department of the University Institute of the Italian Hospital of Buenos Aires, Argentina.

Competing interests
The authors completed the ICMJE conflict of interest statement and declared that they did not receive funding for this article; they have no financial relationships with organizations that may have an interest in the article published in the last three years and they have no other relationships or activities that may influence the publication of the article. Forms can be requested by contacting the responsible author or the Editorial Board of the Journal.

Funding
The authors declare that they have no external sources of financing associated with the preparation of this article.

Language of submission
Spanish.

Referencias
  1. Kirmayr M, Quilodrán C, Valente B, Loezar C, Garegnani L, Franco JVA. The GRADE approach, Part 1: how to assess the certainty of the evidence. Medwave. 2021 Mar 17;21(2):e8109. | CrossRef | PubMed |
  2. Schünemann H, Brożek J, Guyatt G, Oxman A. The GRADE handbook. Cochrane Collaboration. 2013. [On line]. | Link |
  3. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016 Jun 28;353:i2016. | CrossRef | PubMed |
  4. Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017 Jan;81:101-110. | CrossRef | PubMed |
  5. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995 Nov-Dec;123(3):A12-3. | PubMed |
  6. Nussbaumer-Streit B, Grillich L, Glechner A, Affengruber L, Gartlehner G, Morche J, et al. GRADE: Von der Evidenz zur Empfehlung oder Entscheidung - ein systematischer und transparenter Ansatz, um gut informierte Entscheidungen im Gesundheitswesen zu treffen. 1: Einleitung [GRADE: Evidence to Decision (EtD) frameworks - a systematic and transparent approach to making well informed healthcare choices. 1: Introduction]. Z Evid Fortbild Qual Gesundhwes. 2018 Jul;134:57-66. | CrossRef | PubMed |
  7. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011 Apr;64(4):383-94. | CrossRef | PubMed |
  8. Fretheim A, Munabi-Babigumira S, Oxman AD, Lavis JN, Lewin S. SUPPORT tools for evidence-informed policymaking in health 6: Using research evidence to address how an option will be implemented. Health Res Policy Syst. 2009 Dec 16;7 Suppl 1(Suppl 1):S6. | CrossRef | PubMed |
  9. Siddiqui S, Razzak J. Early versus late pre-intensive care unit admission broad spectrum antibiotics for severe sepsis in adults. Cochrane Database Syst Rev. 2010 Oct 6;2010(10):CD007081. | CrossRef | PubMed |
  10. Zhang Y, Coello PA, Guyatt GH, Yepes-Nuñez JJ, Akl EA, Hazlewood G, et al. GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences-inconsistency, imprecision, and other domains. J Clin Epidemiol. 2019 Jul;111:83-93. | CrossRef | PubMed |
  11. Zhang Y, Alonso-Coello P, Guyatt GH, Yepes-Nuñez JJ, Akl EA, Hazlewood G, et al. GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness. J Clin Epidemiol. 2019 Jul;111:94-104. | CrossRef | PubMed |
  12. Tikkinen KAO, Dahm P, Lytvyn L, Heen AF, Vernooij RWM, Siemieniuk RAC, et al. Prostate cancer screening with prostate-specific antigen (PSA) test: a clinical practice guideline. BMJ. 2018 Sep 5;362:k3581. | CrossRef | PubMed |
  13. Besio M, Besio F. Testigos de Jehová y transfusión sanguínea: Reflexión desde una ética natural. Rev Chil Obstet Ginecol. 2006;71(4):274–9. [On line]. | Link |
  14. Schad M, John J. Towards a social discount rate for the economic evaluation of health technologies in Germany: an exploratory analysis. Eur J Health Econ. 2012 Apr;13(2):127-44. | CrossRef | PubMed |
  15. Oxman AD, Fretheim A, Lavis JN, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP) 12: Finding and using research evidence about resource use and costs. Health Res Policy Syst. 2009 Dec 16;7 Suppl 1(Suppl 1):S12. | CrossRef | PubMed |
  16. Ministerio de Salud de Argentina, Instituto Nacional del Cáncer de Argentina, Di Sibio A. Efectividad del tamizaje mamográfico en la reducción de la mortalidad por cáncer de mama. Buenos Aires: INC, MINSAL; 2018. [On line]. | Link |
  17. Riikonen JM, Guyatt GH, Kilpeläinen TP, Craigie S, Agarwal A, Agoritsas T, et al. Decision Aids for Prostate Cancer Screening Choice: A Systematic Review and Meta-analysis. JAMA Intern Med. 2019 Jun 24;179(8):1072–82. | CrossRef | PubMed |
  18. Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, et al. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med. 2006 Sep 1;174(5):605-14. | CrossRef | PubMed |
  19. Glaser AP, Schaeffer AJ. Urinary Tract Infection and Bacteriuria in Pregnancy. Urol Clin North Am. 2015 Nov;42(4):547-60. | CrossRef | PubMed |
  20. Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013 Jul;66(7):719-25. | CrossRef | PubMed |

 

Licencia Creative Commons Esta obra de Medwave está bajo una licencia Creative Commons Atribución-NoComercial 3.0 Unported. Esta licencia permite el uso, distribución y reproducción del artículo en cualquier medio, siempre y cuando se otorgue el crédito correspondiente al autor del artículo y al medio en que se publica, en este caso, Medwave.
Address: Villaseca 21, Of. 702, Ñuñoa, Santiago de Chile.
Phone: 56-2-22743013
ISSN 0717-6384