Introduction

Over the last few decades, the use of evidence in the practice of medical science as a decision-making tool from a clinical or public health perspective has become increasingly common, resulting in a progressive and considerable increase in primary research in recent years. The amount of primary evidence that currently exists (and continues to be produced) is impossible for a clinician to address to make decisions. For this reason, it became necessary to develop systematic reviews as a methodological design, which made it possible to organize and synthesize the evidence. Over time, systematic reviews have made it possible to compile much of the primary evidence [1]. However, the number of systematic reviews has increased significantly, with an estimated average of 48 000 publications per year in the last three years [2], often addressing the same clinical question, occasionally with discordant results [3,4]. In addition, it can often be difficult to conduct comprehensive systematic reviews of the literature to obtain orderly information on a broad topic, among other things, because of the time involved and the associated methodological difficulties [5].

The "overviews of reviews", also known as "umbrella reviews", "review of reviews", and "meta-reviews" (which we will simply call "overviews" from now on), arise from this need to systematize and summarize information. An overview is a study design that takes systematic reviews as the unit of analysis instead of primary studies [6]. Overviews synthesize the body of evidence on a topic of interest, often to answer research questions that are broader in scope than those examined in individual systematic reviews, presenting the findings in an orderly and summarized manner [7]. Overviews aim to make information more accessible to clinicians, decision-makers, researchers, policymakers, and healthcare providers [8].

This article is the twelfth in a methodological series of narrative reviews on general biostatistics and clinical epidemiology topics, which explore and summarize published articles in a user-friendly language in the main databases and specialized reference texts. The series is oriented to the training of undergraduate and graduate students. It is carried out by the Chair of Evidence-Based Medicine of the School of Medicine of the University of Valparaiso, Chile, in collaboration with the Research Department of the University Institute of the Italian Hospital of Buenos Aires, Argentina, and the UC Evidence Center of the Pontificia Universidad Católica de Chile. This manuscript aims to describe overviews, their usefulness, and their particularities as a methodological design.

What are overviews?

To facilitate the understanding of overviews, we must first define what a systematic review is. A systematic review is a secondary study design that uses explicit and systematic methods to search for and to identify primary studies related to the research question, intending to synthesize information [9].

Overviews can be defined as a review (an evidence synthesis design) that uses explicit and systematic methods to search for and identify systematic reviews (not primary studies) related to the research question in the same area, to extract and analyze their results through important outcomes [8]. Overviews can describe the current body of evidence from systematic reviews on a topic of interest or address a new review question that the included systematic reviews did not address as a specific objective. In addition, they may present the results exactly as they appear in the included systematic reviews, or they may choose to reanalyze the data [10].

Differences between overviews and other types of evidence synthesis

Earlier in this methodological series, we have addressed other types of evidence syntheses, such as systematic reviews [9], scoping reviews [11], and evidence gap maps [12], but what sets them apart from an overview?

Overviews are, in many ways, similar to systematic reviews [13]. Both methodological designs use systematic information search and selection methods similarly. Both designs generally assess the risk of bias in the included studies and present a narrative and/or statistical analysis of results. However, the fact that systematic reviews use primary studies (e.g., clinical trials) as the unit of analysis and overviews use the systematic reviews themselves brings fundamental differences that are important to consider.

A systematic review differs from an overview on two key points: its scope and its unit of study. In terms of scope, an overview is much broader than a systematic review. While the latter generally focuses on analyzing outcome differences in the same population for two interventions, the overview can examine the evidence from two or more systematic reviews.

This allows us to evaluate the results of different interventions applied to the same population, the same intervention applied to different populations, or the outcomes measured through the reviews [8]. Regarding the units of analysis, systematic review synthesizes primary studies (e.g., randomized clinical trials or cohort studies), whereas, in overviews, secondary studies are the units of analysis, i.e., systematic reviews.

A scoping review is an extensive literature review that answers broad research questions, focusing mainly on exploring the literature, sizing its size, and potential scope in a specific area. Scoping reviews aim to identify key concepts, theories, sources of evidence, and research gaps [11,14]. A gap map can be defined as a thematic collection of evidence structured around a framework that graphically and schematically represents the types of interventions and outcomes relevant to a particular problem [12].

While both scoping reviews and evidence gap maps aim to cover the evidence on a particular topic broadly, their focus is more on reporting in a general way on the body of (primary and secondary) evidence available, either by identifying what has been studied on a topic in the case of scoping reviews, or what remains to be studied in the case of evidence gap maps. In contrast, an overview makes a detailed analysis and summary of specific findings from systematic reviews, attempting to answer structured clinical questions. Table 1 compares overviews and systematic reviews, evidence gap maps, and scoping reviews.

Overviews' particular methodological aspects.

Risk of bias assessment in an overview

Considering that an overview analyzes systematic reviews and systematic reviews, in turn, analyzes primary studies, the assessment of bias risk in the context of an overview can be performed at two levels: on the systematic reviews or the primary studies.

Different tools are available to assess the bias risk of systematic reviews. The most commonly used tools are ROBIS (Risk of Bias in Systematic Reviews) [15], which assesses the risk of bias itself, and AMSTAR-2 (Assessing the Methodological Quality of Systematic Reviews) [16], which assesses the overall quality of reviews [10]. In particular, ROBIS is a tool with three phases detailed in Table 2.

On the other hand, AMSTAR-2 allows the evaluation of systematic reviews of randomized and non-randomized trials where a total of 16 domains are considered, seven of which are critical. After answering the questions of these domains, an overall assessment of the weaknesses is made, and conclusions are drawn regarding the confidence of the review [16].

In addition to assessing the risk of bias in systematic reviews, overviews could assess the bias risk of the included primary studies. This can be done in two ways [10]:

1. Through assessing and verifying the evaluations made by the authors of the included systematic reviews. When conducting an overview, this is a faster method of assessing the risk of bias of the primary studies. It allows contrasting authors' evaluations from different reviews and comparing judgments regarding risk of bias.

2. By assessing the primary studies directly, the authors of an overview may prefer to individually assess the primary studies included in the systematic reviews, using specific tools according to study design (e.g., Cochrane Tool for randomized clinical trials) [17].

For example, in a recently published overview on the utility of using systemic oncologic treatments versus supportive care for patients with advanced hepatobiliary cancer, the authors performed the critical appraisal of the systematic reviews using the AMSTAR-2 tool but additionally reported the risk of bias of the primary studies according to the authors' assessment of each systematic review [18].

Assessment of primary study overlap between systematic reviews

An important feature of the overviews is that they allow us to assess the level of overlap of the primary studies included in the systematic reviews analyzed by the overview. Overlap refers to the multiple inclusion of the same primary study in different systematic reviews within the same overview. This can lead to overestimation (or underestimation) of the true effect of an intervention. Figure 1 provides a graphical representation of how overlap can overestimate the results of a specific primary study.

Figure 1

Representation of overlapping primary studies within an overview.

RCT: Randomized Clinical Trial. SR: Systematic Review.
Green circle: patients in whom treatment was beneficial.
Red circle: patients in whom treatment was not beneficial.
It could be erroneously concluded that drug A is superior to drug B because RCT 3 is overrepresented as it is included in several SR, thus overestimating its real effect.
Source: Prepared by the authors.

Full size

To address this problem, there are multiple proposals [19]. One of them is using evidence matrices, which is a graphic way of presenting the overlap (Figure 2). Matrices are tables or grids where systematic reviews are ordered in the columns and primary studies in the rows (or vice versa), accounting for the number of times a primary study was repeated throughout the different systematic reviews included. Evidence matrices are quite useful for plotting the overlap of primary studies; however, they become more difficult to interpret as the evaluated body of evidence becomes larger.

Figure 2

The included matrix made with the GROOVE (Graphical Representation of Overlap for OVErviews) tool shows the studies included in the analyzed overview, randomized clinical trials (RCTs) in the rows, and systematic reviews (SRs) in the columns. Green boxes marked with a 1 indicate that the randomized clinical trial was included in the intersected systematic reviews.

Source: Prepared by the authors.

Full size

Calculating the covered area and corrected covered area [8,20,21] is another tool proposed to assess the degree of overlap. The overlap of included primary studies can be calculated from an evidence matrix as a percentage measure. Figure 3 provides the calculation of primary study overlap from an evidence matrix. Recently, GROOVE (Graphical Representation of Overlap for OVErviews) has been developed, a tool that, based on the calculation of the corrected covered area, provides a visual representation of the primary study overlap between the systematic reviews included in an overview, both in general and for each pair of systematic reviews (Figure 4) [22,23].

Figure 3

Table included in the GROOVE tool showing the calculation of the covered area and the corrected covered area for the example we have developed in Figure 2.

Source: Prepared by the authors.

Full size

Figure 4

Final GROOVE graph comparing all the systematic reviews included in the overview and showing the percentage of overlap between them. The figure shows the degree of overlap for the example shown in Figure 2.

Source: Prepared by the authors.

Full size

For example, in an overview of the effectiveness of non-pharmacological interventions to prevent adverse events in intensive care units, the overlap was assessed using the GROOVE tool for each outcome separately, and a high degree of overlap was found overall [24]. This helped the interpretation of the overview results, avoiding overestimation of effects.

Assessing the certainty of the evidence

To determine an overview’s certainty of the evidence, aspects such as the limitations of the included systematic reviews, the presence of overlapping primary studies, and the lack of relevant information not presented should be considered [10,25,26]. The certainty of the evidence is generally estimated using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) methodology [27]. Even though working groups are already formed, there is still no widely accepted methodological adaptation for assessing the certainty of evidence specific to overviews [25,26].

Thus, one way to determine the certainty of evidence in an overview is to report the certainty of evidence estimated by the authors of the included systematic reviews. Eventually, there may be inconsistency in assessing the certainty of evidence among different groups of systematic review authors, which should be addressed in the results discussion of the overview authors. Another way to address the certainty of the evidence is to perform a new evidence certainty assessment, analyzing all unique primary studies that report data for the same outcome. This can be especially useful in cases where the certainty of evidence has been evaluated using different methods among the different included reviews or in cases where the evaluations are not consistent with the objectives of the overview [10].

For example, an overview of the Cochrane collaboration on the effectiveness of assisted reproductive therapies for subfertile couples chose to describe the results of each systematic review, giving the certainty of the evidence for each finding as reported by each review [28]. On the other hand, in a series of overviews regarding the effectiveness of systemic oncologic treatments for various advanced cancers, the authors decided to analyze the certainty of evidence de novo, performing a GRADE approach from the primary studies included in all systematic reviews [18,29,30].

Remaining challenges

Although overviews are useful tools, they are still a relatively new methodological design, and they present some unmet challenges that must be addressed.

Until recently, there were no widely accepted reporting guidelines for conducting overviews. These guidelines are needed to systematize how these studies are published. The CONSORT (Consolidated Standards of Reporting Trials) guidelines for clinical trials [31] or PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) for systematic reviews [32] have been widely used and accepted by the scientific community. For overviews, interesting new proposals are under development and improvement, such as the checklist of Onishi & Furukawa [33] or Li et al. [34]. Recently, the PRIOR (Preferred Reporting Items for Overviews of Reviews) guide [35,36] has been published, which promises to be the most complete. Given that its publication is very recent, it remains to be seen whether its use is widely accepted and used by overview authors. Likewise, a pending challenge is the development of specific reporting guidelines for the integration of qualitative findings in overviews, in a similar way to the SRQR (Standards for Reporting Qualitative Research) [37] and COREQ (Consolidated Criteria for Reporting Qualitative Research) [38] guidelines that provide guidelines for the correct writing of other types of qualitative studies.

Secondly, the absence of tools to assess the certainty of evidence designed specifically for this type of study can be considered an unmet challenge [37]. Among the options that some authors have chosen are to adapt the GRADE tool for use in overviews, to use the GRADE tool directly in the overview as if it were a systematic review, and to report the certainty of the evidence for each review included in the overview [10].

Regarding overview overlap, although progress has been made and methods have been developed to estimate and plot it (such as evidence matrices, the calculation of the corrected covered area, and the GROOVE tool), there is still a long way to go. For example, the extent to which a high degree of overlap could modify the magnitude of an intervention’s effect within an overview has not yet been studied.

Another gap that overviews, as an emerging type of study, must bridge is that of making themselves known. The dissemination of this type of study should grow over time to encourage more researchers to conduct them and become more accessible. It would be useful to have a centralized database where all the available overviews could be easily consulted and, therefore, all the reviews and summaries of the available information.

Generating a single site could be useful, as the Cochrane Library did in the past with systematic reviews and is doing again now with overviews. Another interesting option would be to generate new specific search filters for overviews, considering that large databases such as MEDLINE, EMBASE, or Epistemonikos already have the vast majority (if not all) of the published overviews [38,39].

Finally, in addition to the challenge of disseminating overviews, the spectrum of their use remains to be broadened. There is still a long way to go in terms of positioning them as an important synthesis tool. As part of the scientific community and as healthcare personnel, we still need to define the main usefulness of overviews, which can range from support for specific clinical decision-making or information for patients or decision-makers in general to a tool for meta-epidemiologic assessment of evidence [40].