20 June 2013
Evidence-Based Medicine applied to the control of communicable disease incidents when evidence is scarce and the time is limited
Control of acute communicable disease incidents demands rapid risk assessment, often with minimal peer-reviewed literature available but conducted in the public’s view. This paper explores how methods of evidence-based medicine (EBM) can be applied in this scenario to improve decision making and risk communication. A working group with members from EBM organisations, public health institutions and the European Centre for Disease Prevention and Control used a six-stage framework for rapid risk assessments: preparation, risk detection/verification, risk assessment, development of advice, implementation, and evaluation. It concluded that data from observational studies, surveillance and modelling play a vital role in the evidence base. However, there is a need to further develop protocols and standards, to perform, report and register outbreak investigations more systematically and rigorously, and to allow rapid retrieval of the evidence in emergencies. Lack of evidence for risk assessment and advice (usual for new and emerging diseases) should be made explicit to policy makers and the public. Priorities are to improve templates for reporting and assessing the quality of case and outbreak reports, apply grading systems to evidence generated from field investigations, improve retrieval systems for incident reports internationally, and assess how to communicate uncertainties of scientific evidence more explicitly.
Public health agencies responsible for the control of public health emergencies are expected to work according to the best standards of scientific evidence. They need to be explicit about the source, type, quality, scope and completeness of the evidence, so that policy makers, politicians and the public can understand the evolving nature of evidence, its strengths and limitations . Even in the acute situation of infectious disease emergencies such as an influenza pandemic, agreed protocols for developing policy and advice should be followed. However, there are two important challenges: reliance upon limited field investigations and population surveillance data, and the speed with which evidence has to be identified and synthesised.
In 2010 the European Centre for Disease Prevention and Control (ECDC) set up a working group to review the potential utility of currently used evidence-based medicine (EBM) tools and risk assessment tools in realistic communicable disease outbreak scenarios, and to propose new tools . A group of experts from 12 countries working in EBM and public health institutions or at ECDC, with a broad range of experience in public health methodology and infectious diseases, were appointed to give guidance on how to strengthen the scientific work at ECDC by adapting and applying EBM methods that were practical and applicable in the environment of infectious diseases and public health.
In this paper we report the conclusions on how to apply the principles of EBM in situations where rapid risk assessment is needed.
Methods and results
The working group presented the experiences of Member States in providing evidence-based guidance in circumstances when time was short, including the influenza pandemic in 2009  and the Q fever epidemic in the Netherlands . Consensus within the group was reached through informal group processes, through plenary and smaller group discussions, and by review of draft texts by the members and work colleagues in their institutions. The group members are listed at the end of the article.
The development of evidence for control of any incident, outbreak or pandemic was conceptualised as a knowledge cycle in which data are collated from surveillance and field investigation reports and peer-reviewed literature, rapidly appraised and used to assess risks, develop advice and implement control measures. Continued surveillance, monitoring and auditing further consolidate the evidence base and allow refinement of risk assessment and evaluation of the effectiveness of interventions (Figure 1). Usually in the acute incident the knowledge cycle is entered at the risk assessment stage, when a report of an incident has to be verified, evidence collated and synthesised, and the risk assessed.
Figure 1. Evidence cycle in outbreak recognition, investigation, control and review
We identified six stages that need to be considered when preparing a rapid risk assessment under time constraints, and the need for improvement in each. They are summarised in the Table and described in detail below.
Table. Conceptual stages in rapid risk assessment and proposed evidence-based medicine tools
Stage 1: Preparatory phase
Alerting and surveillance systems should be set up that are regularly reviewed for fitness for purpose . For newly emerging infections, the published data available to carry out systematic reviews will necessarily be very limited. It is therefore vital that critical summaries of evidence about epidemiology and control of these diseases are kept up to date and accessible internationally, including specifying key gaps in knowledge and suggesting appropriate models for risk assessment. Outbreak investigations are vital for defining epidemiological characteristics of specific pathogens (e.g. reproduction number) and can be used to evaluate the success of interventions [6,7]. However, to the best of our knowledge there are no agreed international standards for outbreak investigation and reporting. The value of field investigations would be greatly improved if a standardised framework for conducting, reporting, and synthesising data from outbreak investigations was used. Such standards exist for strengthening the reporting of observational studies in epidemiology (STROBE) , for the transparent reporting of evaluations of non-randomised designs (TREND) , and for meta-analysis of observational studies (MOOSE) . Fine-tuning and evaluation for their application to outbreak situations has been undertaken for hospital outbreaks (the outbreak reports and intervention studies of nosocomial infection (ORION) statement) . For outbreak reports to be useful to others in a timely way, there needs to be an international repository of such reports and international agreement to make data rapidly available to investigators.
We identified tools and decision aids that we think would greatly improve public health decision making in acute outbreak situations.
- Up-to-date critical summaries of evidence from epidemiology and control of infectious diseases;
- Quality standards for performance and reporting of surveillance and field investigations;
- An international database of outbreak reports, accessible for all and with a user-friendly search function.
Stage 2: Incident verification
The critical step at this stage is to recognise the alert signal among the background noise of information. The agreed terminology outlining the epidemic intelligence process is the following:
- A signal needs to be filtered;
- An event needs to be validated;
- A validated event needs to be analysed.
In order to reduce the risk of bias, reproducible, transparent and explicit incident verification protocols should be followed. The process of verification requires rapid international communication networks of communicable diseases units. Algorithms should include trigger levels for upscaling, and stopping rules, to allow control agencies to agree that further investigation or more detailed risk assessment are not considered appropriate so that resources can be prioritised efficiently .
Tools required for this stage:
- International alerting and verification systems (e.g. the European Union’s Early Warning and Response System ),
- Effective communication platforms (e.g. The European Union’s Epidemic Intelligence Information System ).
Stage 3: Assessment of risk
This stage follows the verification of a threat and should address specific population groups at risk of more severe disease/outcome (e.g. pregnant women, the elderly, young children and immune-compromised individuals), and those at increased risk of exposure (e.g. healthcare workers). For rare, new and emerging infections there may be little or no peer-reviewed literature, and assessments will depend on field investigations, data from ongoing surveillance, and communication with experts in other countries. A comprehensive international database of outbreaks does currently not exist. Systematic methods for rapid searching and appraisal need to be developed that are appropriate to the time scales involved.
In order to reduce bias and to provide transparent quality assurance, risk assessment protocols and algorithms should be followed, and these should explicitly include frameworks for the synthesis of different types of evidence in relation to public health questions (e.g. risk of influenza A(H1N1) infection to pregnant women at different stages of pregnancy), admit to gaps and uncertainties in the evidence and possible alternative explanations of findings. Evidence should be classified by type (e.g. case report, population surveillance, field investigation) and study quality assessed through evidence-based checklists or tools such as the graphic approach to epidemiology (GATE) instrument for critical appraisal  and rapid risk assessment algorithms .
Tools required for this stage:
- A protocol for rapid searching for relevant peer-reviewed and grey literature,
- Checklists and templates for rapid appraisal of the evidence,
- An international database on disease incidents and outbreak reports,
- A rapid risk assessment procedure and tool.
Stage 4: Developing advice
Guidance will need to recognise explicitly the situational context and the population groups to which it is applied, but should seek to follow agreed EBM principles as embodied in, for example, the guidelines evaluation tool AGREE II (appraisal of guidelines for research and evaluation) . The grading of recommendations applicability, development and evaluation (GRADE) instrument was developed to evaluate and make explicit the steps from evidence to recommendations about treatments of diseases, but these principles also apply when a public health decision is to be made under time constraints . An essential part of developing advice is to state clearly what are the options for interventions and the expected relative merits of different options, as well as openness in dealing with uncertainty . Following the principles of EBM under pressure of time will usually reveal a higher level of uncertainty about the conclusions and recommendations than medium- or long-term risk assessments. We are aware that it is difficult, especially for public health agencies, to translate scientific uncertainty into policy advice . Stakeholders expect certainty and clear answers. However, we also believe that scientific uncertainty should be included in the assessment and the decision-making process as information, not ignored .
The working group considered the added value to communicable disease incident control of integrating principles from the discipline of risk analysis, as embodied, for example, in the Codex Alimentarius . If we consider the Public health decision making process as a predictive model, uncertainties can arise both from the potential errors associated with the structure of the model (such as the context of the outbreak, modes of transmission and potential control measures for new infections) and from uncertainties in the values of the model parameters (incomplete data or measurement errors) . These uncertainties are an integral part of scientific judgment and should be reflected in communication with policy makers and the public.
- Guidance on developing advice, including assessment of the quality of evidence;
- Uncertainty tables addressing uncertainties arising directly from the data and from the model/ process used to capture and interpret the data.
Stage 5: Implementation
For effective implementation, advice must be framed by requirements of the target groups. Public perception and communication of risk must therefore be considered. Various governments and international organisations have published guidelines on risk communication which embrace the need for consistent, credible and high-quality information to be shared with the public [24,25]. In acute scenarios, the rapidly changing picture and accumulation of intelligence needs to be explained, and caveats about interim advice clearly admitted.
- A checklist of key points to address in risk communication.
Stage 6: Monitoring and evaluation
The last stage is monitoring the implementation of control measures. It is increasingly recognised by public health agencies that they should have in place systems for learning lessons from incidents and continuously improving performance . Therefore, incidents should be reviewed systematically to identify the lessons for better management of future incidents, and to identify new knowledge about the causative agent and the risks to the population. This would be aided by the use of standardised audit tools  and protocols  that should be followed to give a rapid but systematic approach to identifying lessons within a framework of organisational accountability.
- Protocols for review and audit of lessons to be learned from of incidents.
The validity, credibility and success of public policy and risk management of public health threats are increasingly being seen as dependent upon the use of the best available scientific evidence developed through a transparent and open process . To this end, a working group set up by ECDC has assessed the potential value of a more widespread use of strategies from evidence-based medicine in communicable disease control.
The EBM movement started as an application of epidemiological and public health principles in clinical practice; the application to public health threats is a more recent trend . We recognise that there are important distinctions between evidence-based strategies applied to the review and appraisal in clinical medicine and the reality of public health policy making and communicable disease control, not least the lack of a strong evidence base and the pressure of time. In the sister discipline of risk analysis it is also increasingly being recognised that public health decision making is generally a result of a more complex interaction of the best available evidence from research and other epidemiological sources, with judgements made on needs, resources, local circumstances, and ethical, legal and societal implications .
We see considerable merit in an integrative approach bringing risk analysis methods together with the epidemiological principles of EBM. For example, the EU Scientific Committee for New and Emerging Health Risks uses the expression ‘lines of evidence’ to characterise different sources and levels of evidence and information . They consider lines of evidence that lie at the bottom of the EBM hierarchy. The highest levels of evidence from systematic reviews of randomised trials are seldom available in acute communicable disease incidents and advice has to be derived from observational studies underpinned by microbiological and virological principles. Sometimes advice has to be based on analogy and modelling, using laboratory research, animal experiments and mathematical modelling of outbreak data. When empirical data in an outbreak emerge, they first appear in expert committee papers and conference presentations, well before peer-reviewed publication, making it difficult to identify that knowledge systematically and quickly. But as with higher-level forms of evidence, the quality of such studies, their collation and interpretation should be guided by EBM methods. This demands the application of rigorous, standardised and systematic ways of handling evidence so that the risk of bias is minimised and assumptions are made explicit.
The application of risk analysis methods is particularly important when dealing with the uncertainties implicit in rapid decision making. It is important to acknowledge that the level of confidence in the conclusions reached is typically inversely related to the time that has passed since the start of the event (Figure 2).
Figure 2. Conceptual model of the relationship between uncertainty and time in risk assessments
The confidence level which can be achieved for short-term risk assessments is largely dependent upon the preparatory work done. “Constraints, uncertainties and assumptions having an impact on the risk assessment should be explicitly considered at each step in the risk assessment and documented in a transparent manner. Expressions of uncertainty or variability in risk estimates may be qualitative or quantitative, but should be quantified to the extent that is scientifically achievable” . The applicability and relevance of standard EBM methods increases with time as the outbreak investigations proceed, but at any particular time there is also the necessity to consider the application of the precautionary principle, and to be clear that lack of evidence of harm is not interpreted as evidence for no harm . The principles of EBM, working rigorously, systematically and transparently and according to best available evidence, should apply at all times.
In order to improve the management of outbreaks of communicable disease across Europe, the working group developed a conceptual framework and a potential set of tools and checklists that need to be developed to deal with the twin pressures of timeliness of risk assessment and lack of evidence. We hypothesise that these tools would improve outbreak management and thereby reduce the human and resource costs of outbreaks. They would also provide a clear auditable trail of decision making that would allow continuous learning from outbreaks. We envisage that the tools described above, collected together with worked examples in the format of a work book, could provide a uniform, consistent methodology for health protection practitioners. The international health protection community should work together to take this agenda forward and in particular identify leadership and responsibilities for developing the tools and for setting up and managing the archives and databases identified as a necessary part of EBM applied to outbreak control. Led by the Robert Koch Institute and based on a tender from ECDC, a multidisciplinary team has started to develop and pilot a systematic, transparent and comprehensive evidence assessment framework for rating the evidence and strength of recommendations in the area of infectious disease prevention and control.
Below are listed all members of the working group.
Roberta James, Scottish Intercollegiate Guidelines Network (SIGN), Scotland, UK;
Signe Flottorp, Norwegian Knowledge Centre for the Health Services and GRADE working group, Norway; Antony Morgan, National Institute for Health and Clinical Excellence (NICE), England, UK; Simon Ellis, National Institute for Health and Clinical Excellence (NICE), England, UK; Françoise Hamers, Haute Autorité de Santé (HAS), France; Jacek Mrukowicz, Institute for Evidence Based Medicine, The Jagiellonian University Krakow, Poland and the GRADE working group (participated in two meetings).
Public health institutions:
Alex Sanchez-Vivar, Health Protection Scotland, Scotland, UK; Anders Tegnell, Swedish Board of Health, Sweden (Member of ECDC Advisory Forum); Gérard Krause, Robert Koch Institute, Germany (Member of ECDC Advisory Forum); Heather Murdoch, Health Protection Scotland, Scotland UK; Hanne Nøkleby, Norwegian Institute of Public Health, Norway (Member of ECDC Advisory Forum); Lukas Murajda, Jessenius Faculty of Medicine, Comenius University, Slovakia; Stephen Palmer, Health Protection Agency, UK and Cardiff University, Wales, UK; Pentti Huovinen, National Institute for Health and Welfare, Finland (participated in one meeting).
Helena de Carvalho Gomes, Scientific Advice Unit; Piotr Kramarz, Scientific Advice Unit; Paulo Moreira, Health Communication Unit, replaced by Ines Steffens, Health Communication Unit; Andrew Amato, Surveillance Unit; Howard Needham, Scientific Advice Unit; Katrin Leitmeyer, Preparedness and Response Unit; Ana-Belén Escriva, Scientific Advice Unit; Marc Struelens, Scientific Advice Unit; Andreas Jansen, Scientific Advice Unit; Frode Forland, chair, Scientific Advice Unit.
- Hine D. The 2009 influenza pandemic. An independent review of the UK response to the 2009 influenza pandemic. London: Cabinet Office; 2010. Available from: http://www.dhsspsni.gov.uk/the2009influenzapandemic_acc.pdf
- European Centre for Disease Prevention and Control (ECDC). Evidence-based methodologies for public health. Technical Report. Stockholm: ECDC; 2011. Available from: http://www.ecdc.europa.eu/en/publications/Publications/Forms/ECDC_DispForm.aspx?ID=738
- European Centre for Disease Prevention and Control (ECDC). Guide to public health measures to reduce the impact of influenza pandemics in Europe: 'The ECDC Menu'. Stockholm: ECDC; 2009. Available from: http://ecdc.europa.eu/en/publications/publications/0906_ter_public_health_measures_for_influenza_pandemics.pdf
- European Centre for Disease Prevention and Control (ECDC). Risk assessment of Q fever. Technical Report. Stockholm: ECDC; 2010. Available from: http://www.ecdc.europa.eu/en/publications/Publications/1005_TER_Risk_Assessment_Qfever.pdf
- Buehler JW, Hopkins RS, Overhage JM, Sosin DM, Tong V. Framework for evaluating public health surveillance systems for early detection of outbreaks. MMWR Recomm Rep. 2004;53(RR-5):1-11. PMid:15129191
- Voirin N, Barnet B, Netzger MH, Vanhems P. Hospital-acquired influenza: a synthesis using the outbreak reports and intervention studies of nosocomial infection (ORION) statement. J Hosp Infect. 2009;71(1):1-14. http://dx.doi.org/10.1016/j.jhin.2008.08.013 Mid:18952319
- Heijne JC, Teunis P, Morroy G, Wijkmans C, Oostveen S, Dinzer E, et al. Enhanced hygiene measures and norovirus transmission during an outbreak. Emerg Infect Dis. 2009;15(1):24-30. http://dx.doi.org/10.3201/1501.080299 PMid:19116045 PMCid:2660689
- von Elm E, Altman DG, Egger M, Pocock SJ, Gotzscher PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453-7.
- Des Jarlais DC, Lyles C, Crepaz N, TREND Group. Improving the reporting quality of non-randomised evaluations of behavioural and public health interventions. The TREND statement. Am J Public Health. 2004;94(3):361-6. http://dx.doi.org/10.2105/AJPH.94.3.361 PMid:14998794 PMCid:1448256
- Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008-12. http://dx.doi.org/10.1001/jama.283.15.2008 PMid:10789670
- Stone SP, Cooper BS, Kibbler CC, Cookson BD, Roberts JA, Medley GF, et al. The ORION Statement: guidelines for transparent reporting of outbreak reports and intervention studies of nosocomial infection. J Antimicrob Chemother. 2007;59(5):833-40.
- Palmer SR, Brown D, Morgan D. Early qualitative risk assessment of the emerging zoonotic potential of animal diseases. BMJ. 2005;331(7527):1256-60. http://dx.doi.org/10.1136/bmj.331.7527.1256 PMid:16308389 PMCid:1289329
- Cox A, Guglielmetti P, Coulombier D. Assessing the impact of the 2009 H1N1 influenza pandemic on reporting of other threats through the Early Warning and Response System. Euro Surveill. 2009;14(45):pii=19397. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19397 PMid:19941788
- Paquet C, Coulombier D, Kaiser R, Ciotti M. Epidemic Intelligence: a new framework for strengthening disease surveillance in Europe. Euro Surveill. 2006;11(12): pii=665. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=665
- Jackson R, Ameratunga S, Broad J, Connor J, Lethaby A, Robb G, et al. l. The GATE frame: critical appraisal with pictures. Evid Based Med. 2006;11(2):35-8. http://dx.doi.org/10.1136/ebm.11.2.35 PMid:17213070
- European Centre for Disease Prevention and Control (ECDC). Operational guidance on rapid risk assessment methodology. Stockholm: ECDC;2011. Available from: http://ecdc.europa.eu/en/publications/Publications/1108_TED_Risk_Assessment_Methodology_Guidance.pdf
- Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839-42. http://dx.doi.org/10.1503/cmaj.090449 PMid:20603348 PMCid:3001530
- Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64(4):380–2. http://dx.doi.org/10.1016/j.jclinepi.2010.09.011 PMid:21185693
- The Parliamentary Office of Science and Technology. Handling uncertainty in scientific advice. Postnote. 2004;220. Available from: http://www.parliament.uk/documents/post/postpn220.pdf
- Kelly M, Morgan A, Ellis S, Younger T, Huntley J, Swann C. Evidence based public health: A review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010;71(6):1056-62.
- Forland F, De Carvalho Gomes H, Nokleby H, Escriva A, Coulombier D, Giesecke J, et al. Applicability of evidence-based practice in public health: risk assessment on Q fever under an ongoing outbreak. Euro Surveill. 2012;17(3):pii=20060. Available online: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20060 PMid:22297099
- Joint Food and Agriculture Organization (FAO)/World Health Organization (WHO) Food Standards Programme. Codex Alimentarius Commission: procedural manual. Section III. 15th ed. Rome: FAO, WHO; 2005. Available from: ftp://ftp.fao.org/codex/Publications/ProcManuals/Manual_15e.pdf
- Droguett EL, Mosleh A. Bayesian methodology for model uncertainty using model performance data. Risk Anal. 2008;28(5):1457-76.
- Cabinet Office. Communicating Risk. Available from: https://update.cabinetoffice.gov.uk/sites/default/files/resources/communicating-risk-guidance.pdf
- National Institute for Public Health and the Environment (RIVM)/ Environmental Assessment Agency (MNP) Guidance for uncertainty assessment and communication: detailed guidance. Utrecht: Utrecht University; 2003. Available from: http://www.nusap.net/downloads/detailedguidance.pdf
- Late Lessons from early warnings: the precautionary principle 1896–2000. Environmental issue report no 22. Copenhagen: European Environmental Agency, Copenhagen, 2001. Available from: www.eea.europa.eu/publications/environmental_issue_report_2001_22/Issue_Report_No_22.pdf
- Kipping RR, Hamilton S, Roderick M, Alexander K. Developing audit standards required for outbreaks of communicable diseases – lessons from a mumps outbreak. J Public Health. 2006;28(4):347-50. http://dx.doi.org/10.1093/pubmed/fdl065 PMid:17065178
- Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175-201. http://dx.doi.org/10.1146/annurev.publhealth.031308.100134 PMid:19296775
- European Commission. The Scientific Committees on Consumer Safety (SCCS), Health and Environmental Risks (SCHER), Emerging and Newly Identified Health Risks (SCENIHR). Rules of procedure. Brussels: DG Health and Consumers. Available from: http://ec.europa.eu/health/ph_risk/documents/rules_procedure_en.pdf
- Rutter M. Identifying the environmental causes of disease: how should we decide what to believe and when to take action? London: Academy of Medical Sciences, 2007. Available from: http://www.acmedsci.ac.uk/download.php?file=/images/publication/119615475058.pdf