Introduction

According to evidence-based medicine, systematic reviews are the most reliable evidence synthesis method to make healthcare decisions. For that reason, clinical practice guidelines use systematic reviews when elaborating their recommendations[1],[2]. However, producing and publishing a systematic review takes an average of over 67 weeks (or a total of one year and three months) because of its high methodological standards[3]. Furthermore, many trials are published and registered daily[4], so a systematic review may be outdated shortly after publication[5]. Therefore, the evidence sustaining the recommendations to clinicians may not be the current evidence on a topic[6]. In 2014, Elliot et al. proposed a new way of updating systematic reviews to overcome this problem without losing methodological rigor: the living systematic review[7]. Since then, multiple living systematic reviews have been published[8],[9],[10],[11],[12],[13],[14],[15],[16],[17], and the Cochrane Collaboration even has a guide for development and publication[18].

This is the second article of a methodological series of narrative reviews about general topics on biostatistics and clinical epidemiology that explore and summarize several published articles available in the main databases and specialized reference texts. The series aims to reach undergraduate and graduate students. The Evidence-Based Medicine Department of the School of Medicine of Universidad de Valparaíso, Chile, collaborated with the Research Department of Instituto Universitario Hospital Italiano de Buenos Aires, Argentina, and the Evidence Center UC, of the Universidad Católica, Chile to elaborate the series. This article’s main objective is to indicate when it is accurate to perform a systematic review with a living approach and comment on its considerations and challenges.

When to choose the “living” format for a systematic review?

It is important to know when it is convenient to choose the “living” format over the traditional one. There are three requirements that a clinical question must fulfill in order to consider the “living” format[19]:

1) The question addressed in the review is a priority in decision making. If the question is not a priority, the review’s effects or conclusions will not improve patient care.

2) The certainty of existing evidence must be low or very low since if there already exists a high certainty of the evidence for the question, it would not be necessary to invest time or resources to update the evidence continuously.

3) New evidence for the same question is likely to appear in the future. If we anticipate that new evidence will not appear, then it is not worth investing resources in a continuous update.

Authors of a living systematic review should justify the use of the “living” format in their protocol (declaring that it will stay in that format) and mention pre-established criteria (in case there are any) to abandon the “living” format for the conventional method. If a living systematic review does not fulfill any of the points mentioned above, it would be a reason to abandon the living format[19].

The knowledge translation tools used by the authors of the systematic review to identify the best model for their revision have not included the “living” aspect of the review in their proposed flow diagram as of yet[20].

What is new in a living systematic review?

As mentioned, a living systematic review preserves the methodological rigor and quality of a conventional systematic review[18],[21],[22], but proposes a new way to update the review as new evidence becomes available[7],[19]. Authors may conduct a living systematic review from scratch, or they may turn an already published systematic review into a “living” format. To accomplish the continuous update—turning the systematic review into a living systematic review—multiple changes in the production and publication process are recommended. We address some of them below:

1. Workload and distribution

In a conventional systematic review, the workgroup is often small and has a substantial workload for a limited period. A living systematic review aims to generate a larger workgroup with a lighter, but permanent, workload[19],[23]. Moreover, the team working on the review may change over time, integrating and excluding people according to participants’ availability and the workload demanded by the living systematic review[19].

2. Automated and continuous search

To ensure the systematic review’s living nature, the search strategy must run periodically, fulfilling what was established for each review without compromising its sensitivity—remembering that the living systematic review must maintain its rigorous methodology. To accomplish this, we have to change the conventional model in which the authors create the search strategy, run the search, and later screen the results for a “push model.” This model requires an expert to create the search strategy but later automates the search in different databases and then notifies the authors every time there are new results to screen—making the author’s job easier[23].

The automation presents two limitations: first, some databases are not compatible with automation (which may be fundamental for specific living systematic reviews), and second, the search for unpublished articles require human resources[19],[23]. To palliate these limitations and accomplish a sensitive and continuous search, authors may appeal for collaborative work in the scientific community and establish a spaced-in-time search a priori in the databases (including a search for unpublished articles) for those where automation is not possible[19].

3. Updating scenarios

The updating process of a conventional systematic review is arduous and often requires the same effort as the original publication. To avoid this in the living format, authors must define how they will update and integrate the new evidence to the livening systematic review. They will face three possible scenarios when updating the search and screening; Elliot et al. proposed an approach for each of them[19] (Figure 1): 

• The search and screening are updated, and there is no new evidence: Authors should indicate the date of the last search and state that no new evidence was found. 
• The search and screening are updated, and there is new evidence, but it is improbable that this new evidence would change the results and conclusions of the living review: Authors should indicate the date of the last search, state that new evidence was found, describe the new evidence, and justify why it will not yet be integrated into the living systematic review. 
• The search and screening are updated, and there is new evidence that will probably change the results and conclusions of the living review: Authors should indicate the date of the last search, state that new evidence was found, describe the new evidence, and announce that they are currently incorporating the evidence into the living systematic review. They must carry out data extraction, synthesis, and quality assessment to publish an update of the living systematic review.

Figure 1.

Updated scenarios of a living systematic review.

Full size

The authors must a priori define and make explicit these scenarios and the methods to address each one in the living systematic review and its protocol.

4. Statistical methods considerations

When conducting a living systematic review, the central statistical conflict is that every update of the meta-analysis increases the type 1 error: the update increases the probability of finding an effect when there is no effect[24],[25]. In a living systematic review, it is also relevant to know when there is enough evidence to say that there is no difference between interventions or expositions; not knowing this difference will create unnecessary work (control type 2 error).  Heterogeneity, which may change with every update, is another factor to consider[24].

Several methods can control these factors when updating the meta-analysis. The trial sequential analysis and the sequential meta-analysis correct type 1 and 2 errors and adjust results and sample size according to heterogeneity[24],[26],[27]. Other methods like the Shuster method or the law of the iterated logarithm may correct type 1 error[24],[28],[29]. We will not describe each method because it is not the objective of this review.

Authors of a living systematic review should specify which methodology they will use while considering the pros and cons of each. They should also consider that none of the mentioned methods correct systematic errors (just the aleatory ones: type 1 and some type 2 errors), so they address the publication and selective report bias that may exist[24].

5. Editorial and peer review process

Conventional systematic review updates are subjected to the same editorial and peer review process as the original publication of the review, which consumes much time (this process may vary between journals). The first version of a living systematic review must undergo the same editorial and peer review process as a conventional systematic review, but it is impractical to perform this process for every living review’s update because authors are frequently updating the search and screening process of the review (e.g., monthly for Cochrane LSRs). Elliot et al. proposed an approach for each of the three update scenarios described above (see point 3)[7],[19]:

• The search and screening are updated, and there is no new evidence: It would only require editorial review. 
• The search and screening are updated, and there is new evidence, but it is unlikely that this new evidence would change the results and conclusions of the living systematic review: It would require editorial review, but peer review could be optional. 
• The search and screening are updated, and there is new evidence that will probably change the results and conclusions of the living review: It would require editorial and peer review. If results and conclusions remain the same after integrating the new evidence in the review, editorial review only may be considered.

Elliot et al. proposed considering a post-publication peer-review, as some journals such as F1000 perform currently[30], especially in the third scenario. The authors also proposed that a living systematic review should undergo both editorial and peer reviews every one to two years to ensure methodological quality and that authors periodically update the search and screening[7],[9].

6. Publication

The publication process is essential to fulfilling the living review’s objective of providing updated and rigorous evidence to users and decision-makers. This is one of the main differences with the conventional approach of systematic reviews: publication should be dynamic, persistent, and online; it should allow fast updates, and the platform should be accessible[7]. Authorship of the living systematic review and its updates may vary over time according to each researcher’s contributorship in the process[19].

A controversial point is the DOI identifier of the living review and its updates; if every update has its identifier, the review’s impact factor may be underestimated[31]. On the other hand, if each update has the same identifier as the original living systematic review, identifying the project would be easily accomplished, but the authors’ academic productivity may be underestimated[19].

7. Use of new technologies

When we talk about technologies, we refer to computational advances, artificial intelligence, and the improved and efficient use of human resources with collaboration models[23]. All the points displayed above share the use of these new technologies. They should be considered in all steps of a living systematic review as they are crucial to its sustainability. However, we emphasize that none of the tools that we present below are exclusive to the living model: all of them can significantly speed up any systematic review performance.

First, some programs and platforms promote collaboration and simplify conducting systematic reviews (e.g., Rayyan QCRI, Covidence). Regarding technologies improving each step of a systematic review, more tools are available to streamline the initial steps (searching and selecting studies). Automation in different databases and a “push model” when performing screening (see point 3) are possible thanks to new technologies.

In the selection step, artificial intelligence becomes more important every day, as it spares investigators time and improves the efficiency of the process. There are different ways to harness artificial intelligence in this process, depending on each living systematic review. One of these ways is the generic classification based on machine learning techniques—the more the machines learn, the more accurate they become. This classification excludes articles by assessing a generic condition (e.g., it only selects randomized clinical trials), and, by doing so, spares much time in the screening process. Another way is the specific classification based on machines that learn directly from a human expert screening of an original review. Using any of these classifiers to make a living systematic review production (or a conventional review production) more efficient, researchers should consider that specific classifiers have much less learning data than generic classifiers and are less accurate. However, not all revisions support generic classification[23].

The next steps of a living systematic review are less intensive on new technologies, but new tools could speed them up, make data extraction, risk bias assessment, data synthesis, and report more efficiently. Some programs extract data directly from tables and graphs in PDF format (e.g., Graph2Data), and others automate the risk of bias assessment (e.g., RobotReviewer). Speeding up synthesis and reporting of a systematic review could be achieved from template-based sections[23]. The impact of these tools is still unknown, and we will hopefully see the role they play in producing a living systematic review in the future.

Finally, collaboration within the scientific community is essential. We have to promote and create systems that avoid duplication, and we have to generate large teams of researchers to bring updated and high-quality evidence to health decision-makers[19],[23].

What are the challenges of a living systematic review?

One of the challenges that a living systematic review will face, due to it being a new proposal and still under development, is the need to adapt to new tools and mechanisms that make them more efficient, implying that they may change in the next few years[7],[19]. The authors of the living review from the Cochrane Collaboration pointed out that there are many challenges that this model will have to face, such as keeping the review alive considering the resources and amount of time they demand[31]. They also identified specific needs in the process, including the acceleration of the interface that notifies new findings, improved tools (e.g., RevMan), and the PRISMA flowchart, showing the search as alive, among others[31]. There are many possible ways to improve the tools and publication mechanisms of the living systematic reviews, so we should keep an eye on upcoming changes.

Another important challenge is to use this update method in other types of reviews. The possibility of creating living clinical practice guides is vital because it would shorten the amount of time until the evidence is available for decision-makers, and thus the update of these guides would be more efficient[32]. Instead of updating the whole review at once, each recommendation could be updated separately as the living review associated with the guide includes new evidence[19],[32].

Finally, living systematic reviews should not only synthesize updated evidence for decision-makers but also guide future primary investigations diminishing the distance between primary researchers (of controlled trials) and secondary researchers (producing reviews, including living reviews)[33]. Furthermore, the broad search strategy that identifies registered clinical studies’ records will allow living reviews to halt redundant trials and recommend possible improvements to other studies[33]. Thus, living systematic reviews could contribute to avoiding human and material resource waste in primary investigation.

Conclusions

Living systematic reviews are frequently updated and incorporate new evidence as it becomes available. They preserve the methodological rigor of the conventional systematic reviews, but there are multiple considerations that researchers must consider when conducting them. Living systematic reviews have promising applications, as the living model can be applied in several kinds of reviews.

Notes

Authorship contributions

Competing interests

Funding

Ethics 

From the editors