Introduction

Characteristics of the condition

COVID-19 is an infection caused by the SARS-CoV-2 coronavirus[1]. It was first identified in Wuhan, China, on December 31, 2019[2]; three months later, almost half a million cases of contagion had been identified across 197 countries[3]. On March 11, 2020, the World Health Organization characterized the COVID-19 outbreak as a pandemic[1].

Proper decision-making and communication is an ongoing challenge worldwide. As data evolve, both lack of scientific research and overstated information can lead to inappropriate actions. It is vital to differentiate promptly the true epidemic from an epidemic of false claims and potentially harmful actions[4].

Only a few studies have been completed, and those with published results provide results with very low certainty of evidence. However, the speed with which the information is breaking out is unprecedented. For instance, a systematic review published on March 3 that identified only preclinical evidence summarized the evidence regarding the efficacy of hydroxychloroquine for the treatment of COVID-19 pneumonia[5]. On March 16, the first comparative study in humans, a non-randomized study, went viral and hit the news worldwide[6]. On March 24, the first randomized trial, with small sample size and serious limitations, appeared[7]. And there are more than a dozen ongoing trials that should provide relevant information in the next weeks or months[8].

At these levels of uncertainty, the sound advice is to interpret evidence with caution. And yet, government officials, healthcare professionals, and patients seem to be opting for “impulsive actions,” which “can indeed cause major harm”[4].

If COVID-19 is indeed the worst pandemic of the last 100 years, we need the best evidence to handle it. If COVID-19 is not as grave as it is depicted, high-quality research evidence is equally relevant[4].

Systematic reviews are the gold standard to collect and summarize the available evidence regarding a scientific question. Experience, however, has uncovered several limitations in the current evidence ecosystem, which underpins their production[9]. Some of the shortfalls of systematic reviews are:

• A restricted scope, which hampers their use by decision-makers[10].
• Inefficiency, in terms of time and resources[11].
• Unnecessary duplication of efforts, resulting in waste and potential confusion among users[12].
• Rapid obsolescence—a systematic review is as up to date as its literature search[13].
• Suboptimal quality—not all systematic reviews live up to their promise of rigor and transparency, yet decision-makers might ignore how to handle that[14].

Amidst the COVID-19 crisis, these shortfalls risk thwarting our efforts to reduce the effects of the pandemic. It seems appropriate, then, to put into practice a novel approach that tackles them, thus providing the scientific community and other interested parties with evidence that is actionable; rapidly and efficiently produced; up to date; and of the highest quality.

Using a traditional approach to concurrently produce multiple systematic reviews would require a similar number of expert teams conducting each one of them in parallel and starting independent evidence searches de novo. Our approach, in contrast, is to begin the process with the creation of a living collection of all the evidence relevant to COVID-19, organize this evidence in clusters, with each cluster grouping the evidence pertinent to a specific question, and only then distribute the remaining tasks to individual teams. Following a standard protocol, organizing the different review tasks in an innovative and agile way, and taking advantage of technological tools, we expect to increase efficiency without compromising on quality substantially.

This collection of COVID-19 evidence will be available to researchers through an open, easy-to-use web platform that will be continuously updated. New evidence will be organized immediately and mapped and shared with the teams working on the individual reviews for timely update of the evidence synthesis.

Objective

To systematically assess the evidence for multiple questions relevant to COVID-19.

Methods

This protocol was registered in PROSPERO (submitted, awaiting PROSPERO ID allocation) and was designed in line with Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P)[15].

The protocol states the shared objectives and methodology of multiple evidence syntheses (systematic reviews and overviews of systematic reviews) to be conducted in parallel for different questions relevant to COVID-19. It will be adapted to the specificities of each question to be addressed. We expect the final manuscripts based on this protocol to be transparent and with a reduced likelihood of a reviewer’s biased interpretation.

A central team will coordinate the tasks that cut across multiple questions, provide methodological support, assist the editorial process, lead the dissemination and communications efforts, and monitor the need for updates to the individual evidence syntheses.

Identification and prioritization of relevant questions

The central team will collate and prioritize the questions using the following approach:

• Liaison with local and international organizations and research groups.
• Review of guidelines and relevant documents.
• Analysis of the systematic reviews and studies not included in any of the questions already identified.

The relevant populations/conditions, interventions, diagnostic tests, and predicting or prognostic variables will be arranged in a taxonomy, which will be periodically updated.

Search strategies

Electronic searches. We will use Epistemonikos database as the primary source of information. The methods of Epistemonikos are described elsewhere[16], but, in brief, it is maintained by screening ten databases regularly to identify systematic reviews relevant for health decision-making. It contains more than 300 000 systematic reviews and over 400 000 studies included in those reviews.

Additionally, for this project, we will upload information from different sources to Epistemonikos database to supply any content relevant to COVID-19[17].

Additional searches will be conducted using highly sensitive searches in PubMed/MEDLINE, the Cochrane Central Register of Controlled Trials (CENTRAL) and EMBASE, without language or publication status restriction. The searches will cover from the inception of each database and will be updated regularly. 

The main component of the search strategy (PubMed format) will be as follows:

#1        coronavir* 
#2        coronovirus* 
#3        "corona virus"
#4        "virus corona" 
#5        "corono virus" 
#6        "virus corono" 
#7        hcov* 
#8        "covid-19" 
#9        covid19* 
#10      "covid 19" 
#11      “2019-nCoV“
#12      cv19* 
#13      "cv-19" 
#14      "cv 19"
#15      "n-cov" 
#16      ncov* 
#17      "sars-cov-2" 
#18      (wuhan*[tiab] AND (virus OR viruses OR viral OR coronav*))
#19      (covid* AND (virus OR viruses OR viral)) 
#20      "sars-cov" 
#21      "sars cov" 
#22      "sars-coronavirus" 
#23      "severe acute respiratory syndrome" 
#24      "mers-cov" 
#25      "mers cov" 
#26      "middle east respiratory syndrome" 
#27      "middle-east respiratory syndrome"

We will adapt this strategy to the syntax of the other databases and develop additional components of the search for the individual reviews/overviews, following the same approach. For questions where the search does not retrieve any relevant evidence, we will develop searches that include a broader population (e.g., influenza, other respiratory infections).

In cases where indirect evidence of adverse effects from other conditions is considered relevant, we will run searches without a population component (e.g., adverse effects of antimalarials).

Other sources. We will screen the reference lists of other systematic reviews and will evaluate in the full text all the articles they included. We will check the reference lists of selected guidelines, narrative reviews, and other documents. We will conduct a cross-citation search in Google Scholar and Microsoft Academic, using each included study as the index reference. We will review websites from pharmaceutical companies producing drugs with claims to effectiveness for COVID-19 drugs, websites or databases of major regulatory agencies, and other websites specialized in COVID-19. We will email the contact authors of all of the included studies to ask for additional publications or data on their studies and other studies on the topic. We will search trial registers not included in Epistemonikos Database. We will search with the reference lists of the included studies.

Types of studies/reviews

Systematic reviews. We will search for any type of primary study. For intervention questions, we will prioritize randomized controlled trials, but other types of comparative study designs will also be included. The rationale for including study designs beyond randomized trials will follow the Cochrane EPOC Group guidance[18].

Overviews of reviews. We will include systematic reviews according to Epistemonikos database definition[16].

Types of conditions

We will primarily search for direct evidence on patients with COVID-19 or at risk of developing it (e.g., prevention questions), as defined by the authors of the studies. If we find substantial clinical heterogeneity on how the condition was defined, we will explore it using a sensitivity analysis. We will exclude studies evaluating the effects on animal models or in vitro conditions.

Treatment and prognosis questions. Also, we will routinely analyze indirect evidence from previous coronavirus pandemics (SARS-CoV and MERS-CoV). If no sufficient evidence is available from studies on coronavirus infection, we will broaden our criteria to include other respiratory viruses (e.g., influenza). For adverse effects of interventions, we will also include any condition relevant to the specific question.

Diagnostic questions. We will include participants without symptoms, with specific symptoms suggesting the presence of COVID-19 infection and patients with COVID-19, where the test aims to identify complications or risk of transmission.

Type of interventions, tests, exposures or predictive factors

Intervention questions. We will include studies evaluating interventions for prevention or treatment in people at risk or infected with COVID-19, including but not limited to: 

• Pharmacological interventions (e.g., antivirals, antimalarials, anti-IL-6, vaccination, macrolides, antivirals, convalescent sera).
• Personal protective measures (hand hygiene, facemasks, respiratory etiquette, etc.); 
• Public health measures (e.g., social distancing measures).
• Health system interventions (e.g., task shifting, after-hour care models).
• Complementary and alternative medicine (herbs and other natural ingredients, acupuncture, traditional Chinese medicine).
• Nutritional interventions and nutraceuticals (e.g., vitamin C supplementation).
• Behavioral interventions (e.g., psychological first aid).

For treatment questions, our primary interest will be in studies evaluating the effect of an intervention plus the optimal treatment versus placeboplus optimal treatment. However, we will also include studies comparing against no treatment, against another active intervention or comparing different schemes or doses.

Diagnostic questions. We will include studies evaluating any diagnostic test defined in a broad sense (e.g., including symptoms and signs as ‘tests’).

Prognostic questions. We will include studies assessing the course of the disease in different populations and under different contexts. We will also include studies assessing factors, or a combination of factors, predicting the risk of developing COVID-19, predicting the outcome of the disease, or predicting treatment response[19].

Questions about the magnitude of the problem. We will include any study providing a measure of frequency (e.g., incidence), burden or cost, or qualitative measures of the problem (barriers and facilitators, values and preferences).

Type of outcomes

We will use the following approach to inform which outcomes will be selected, which will be considered as primary outcomes and secondary outcomes, and which, and in what order, will be presented in the Summary of Findings table.

Firstly, we will give priority to validated core outcome measures for COVID-19 (e.g., COS-COVID[20]) or other relevant conditions, if available.

Secondly, we will search for studies informing about the relative importance of the outcomes[21]. 

Thirdly, we will select outcomes that are critical for decision-making (including patient-reported outcomes for those conditions in which they are relevant) according to the judgment of the authors of the individual reviews or overviews. The information from available guidelines and other related documents will be considered to inform this judgment.

We will not routinely consider surrogate outcomes, but the authors of the individual reviews will decide about the pertinence of doing so in specific cases.

We will prioritize up to seven critical outcomes for the development of ‘Summary of Findings’ tables[22],[23].

We will consider grouping outcomes according to the time point in which they were measured in categories (e.g., short term, medium term, long term).

We will not use the outcomes as an inclusion criterion during the selection process. Any article meeting all the criteria except for the outcome criterion will be preliminarily included, and the corresponding author will be contacted. We will make two attempts to reach the author, separated by one week, before excluding the study.

Measures of effect

Intervention questions/efficacy. For dichotomous outcomes, we will express the estimate of the treatment effect of an intervention as risk ratios along with 95% confidence intervals. For continuous outcomes, we will use mean difference and standard deviation to summarize the data using a 95% confidence interval. Whenever continuous outcomes are measured using different scales, the treatment effect will be expressed as a standardized mean difference with 95% confidence interval. When possible, we will multiply the standard mean difference by a standard deviation that is representative from the pooled studies, for example, the standard deviation from a well-known scale used by several of the studies included in the analysis on which the result is based. In cases where the minimally important difference is known, we will also present continuous outcomes as minimally important difference units or inform the results as the difference in the proportion of patients achieving a minimal important effect between intervention and control[23]. Then, these results will be displayed on the 'Summary of Findings Table' as to mean difference.

Intervention questions/adverse effects. We will summarize information from the studies used in the efficacy estimation. If the information obtained from these studies provides low or very low certainty, we will consider the use of the estimates from the relevant systematic reviews identified in the overviews if they provide higher certainty of evidence (e.g., high-quality evidence from a different population).

Diagnostic questions. We will express diagnostic interventions’ accuracy as sensitivity and specificity, together with a 95% confidence interval.

Prognostic questions. For questions on fundamental prognosis (course of COVID-19 infection in the context of the nature and quality of current care), we will express the risk of developing a particular outcome as a proportion.

For questions on prognostic factors, we will standardize the units of measurement for each variable, unifying the direction of the predictors, adjusting the weights of the studies, and calculating crude effect estimates when not provided. We will present the effect estimate as odds ratio and their corresponding 95% confidence intervals. In studies reporting the measure of association as a hazard ratio or risk ratio, we will convert them to odds ratio using the baseline risk reported in the studies.

For questions on prognostic models, we will summarize the predictors included in the final model; how the included predictors where coded; what the specification of the model was, and how it produces an individual outcome probability or risk score; the reported predictive accuracy of the model; and whether or not the model was validated internally and externally and, if so, how.

Data extraction (selection and coding)

Study selection. The results of the literature search in Epistemonikos database will be automatically incorporated to the L·OVE platform[24](automated retrieval), where they will be de-duplicated by an algorithm comparing unique identifiers (database ID, DOI, trial registry ID), and citation details (i.e., author names, journal, year of publication, volume, number, pages, article title, and article abstract).

The additional searches will be uploaded to the screening software Collaboratron™[25].

In both L·OVE and Collaboratron™, two researchers will independently screen the titles and abstracts yielded by the search against the inclusion criteria. We will obtain full reports for all the article titles that appear to meet the inclusion criteria or require further analysis to decide on their inclusion.

We will record the reasons for excluding trials in any stage of the search.

We will outline the study selection process in a PRISMA flow diagram adapted for this project.

Using standardized forms, two reviewers will extract data independently from each included study/review. We will collect the following: information on study design and setting, participant characteristics (including disease severity and age), and study eligibility criteria; details about the administered intervention(s), the outcomes assessed, the source of study funding, and conflicts of interest disclosed by the investigators; and risk of bias assessment for every individual study (e.g., ROBINS).

We will resolve disagreements by discussion, and one referee will adjudicate unresolved disagreements.

Risk of bias (quality) assessment

Two reviewers will independently assess the risk of bias using the following tools:

• For Intervention studies: Cochrane Collaboration tool for assessing the risk of bias, version 2 (RoB 2) for randomized trials, and the ROBINS-I tool[26] for other primary study designs.
• For diagnostic accuracy studies: Quality Assessment of Diagnostic Accuracy Studies (QUADAS‐2) tool[27].
• For prognostic factors: QUality In Prognostic Studies (QUIPS) tool[28].
• For fundamental prognosis questions: We will adapt the QUIPS tool and consider the study population, study attrition, outcome measurement, statistical analysis, and report[28].
• For prognostic models: Prediction model Risk Of Bias ASsessment Tool (PROBLAST)[29].
• For systematic reviews: A MeaSurement Tool to Assess systematic Reviews- 2 (AMSTAR-2)[30].

Handling of missing information

We will perform a complete case (only includes participants for which we have no missing data on the variables of interest) as the primary analysis. We will test the robustness of the results performing sensitivity analysis with different imputation strategies to include missing participants[31].

In the event of a study/review reporting insufficient details, we will judge the risk of bias as ‘unclear’ and contact the investigators of the original study for more information. Disagreements will be resolved first by discussion and then by consulting a third author for arbitration.

When possible and applicable, we will compute graphic representations of potential bias within and across studies using RevMan 5.3.5 (Review Manager 5.3.5)[32], but other software might be used if preferred by the authors of the individual reviews.

Strategy for data synthesis

We will only conduct a meta-analysis if the included studies are sufficiently homogeneous in terms of design, population, interventions, and comparators reporting the same outcome measures.

The results for clinically homogeneous studies will be meta-analyzed using RevMan[32] or alternative software if individual systematic reviews authors consider it appropriate. As general guidance, meta-analyses will be conducted using the inverse variance method with random effects model. However, authors of individual systematic reviews will be able to use alternative analytical strategies if they consider it appropriate. Separate meta-analyses will be presented for specific populations or interventions if statistically significant heterogeneity is explained by some of these, or whenever a convincing subgroup effect is found.

For any outcomes where data is insufficient to perform a meta-analysis, a narrative synthesis will be presented.

Analysis of subgroups or subsets

The individual reviews will assess specific subgroup analyses relevant to the condition of interest. Those analyses will be based on an a prioriheterogeneity hypothesis.

Living evidence synthesis

An artificial intelligence algorithm will be created for each question and deployed in the L·OVE platform to provide instant notification of articles with a high likelihood of being eligible. These will be analyzed by the authors of the individual reviews to decide if the whole search strategy needs to be updated.

After updating the evidence, and as a general guidance, we will resubmit for publication any review or overview in which there is a change in the direction of the effect on the critical outcomes or a substantial modification on the certainty of the evidence. However, authors of individual systematic reviews will decide, case by case, on the pertinence of resubmitting.

To provide end-users with the latest, most up-to-date evidence, we will maintain a living, web-based version of the review/overview in the following website: https://www.epistemonikos.cl/living-evidence/