Evaluation of the Health Protection Event-based Surveillance for the London 2012 Olympic and Paralympic Games

The Health Protection Agency (HPA) (currently Public Health England) implemented the Health Protection Event-Based Surveillance (EBS) to provide additional national epidemic intelligence for the 2012 London Olympic and Paralympic Games (the Games). We describe EBS and evaluate the system attributes. EBS aimed at identifying, assessing and reporting to the HPA Olympic Coordination Centre (OCC) possible national infectious disease threats that may significantly impact the Games. EBS reported events in England from 2 July to 12 September 2012. EBS sourced events from reports from local health protection units and from screening an electronic application 'HPZone Dashboard' (DB). During this period, 147 new events were reported to EBS, mostly food-borne and vaccine-preventable diseases: 79 from regional units, 144 from DB (76 from both). EBS reported 61 events to the OCC: 21 of these were reported onwards. EBS sensitivity was 95.2%; positive predictive value was 32.8%; reports were timely (median one day; 10th percentile: 0 days – same day; 90th percentile: 3.6 days); completeness was 99.7%; stability was 100%; EBS simplicity was assessed as good; the daily time per regional or national unit dedicated to EBS was approximately 4 hours (weekdays) and 3 hours (weekends). OCC directors judged EBS as efficient, fast and responsive. EBS provided reliable, reassuring, timely, simple and stable national epidemic intelligence for the Games.


Introduction
Between July and September 2012, the 2012 Olympic and Paralympic Games (the Games) took place in London and in 10 other United Kingdom (UK) locations.The Games involved 15,000 athletes, 70,000 volunteers and over 10 million tickets were sold [1,2].
Inherent in the characteristics of such mass gathering (MG) events is the increased risk of communicable diseases (e.g.large number of visitors, highly concentrated and mobile population, increased pressure on infrastructure, mass catering) and, due to the high profile of the event, an increased risk of a bioterrorist threat [3][4][5][6].Although communicable diseases have not been a significant cause of health events during recent major sporting MGs [7,8], and those events that have occurred have often been of low risk and low consequence and have not impacted on the success of the event, the increased risk remains.
Effective and timely communicable disease control relies on effective and timely disease surveillance.Epidemic intelligence (EI) encompasses all activities related to detection of public health threats through the early identification of potential health hazards, their verification, assessment and investigation in order to prompt timely public health action [9,10].EI sources information through traditional and routine indicatorbased components (centred on routine reporting of cases of disease) and other event-based components (i.e.unstructured data collection from screening of any kind of source).
Following a risk assessment and gap analysis performed by the UK Health Protection Agency (HPA) (Public Health England since 1 April 2013, but referred to throughout this article as the former organisation) as part of the Games preparedness, a number of potential shortcomings were identified in existing routine indicator-based surveillance systems, leading to the development of some new surveillance approaches for the Games [1,11].One of the new systems established was Health Protection Event-Based Surveillance (EBS), described as 'an organised process to detect, validate, analyse, rapidly assess and report on significant infectious disease events of potential public health risk that may have an impact on the Games' [1], i.e. effectively a 'safety net' system for routine infectious disease reporting systems, as distinct from the traditional understanding of event-based surveillance (which is more community based).While an 'all-hazards' approach was taken to surveillance across the organisation, EBS did not include non-infectious environmental hazards, which were reported through a different surveillance system.This 'national' EI would complement the routine global infectious disease situational analysis (scanning and risk assessment) for public health protection ('international EI') [12], with the aim that the various indicator-and event-based surveillance systems would work as an integrated public health surveillance network.EBS was established in part by building on existing systems in place in the HPA.These existing systems included weekly reports from nine regional offices to the national infectious diseases centre regarding incidents or cases considered to be of national interest.The regional teams sourced this information from 25 local health protection units.
In a time when a growing number of EI systems are being developed [10] and the science of MG health is relatively new, this study aims to describe the evaluation of EBS, in order to identify lessons and contribute to the knowledge-and evidence-base for planning of future MG events.

Methods
The approach to the evaluation of EBS was based broadly on the framework defined by the Updated guidelines for evaluating public health surveillance systems from the United States Centers for Disease Control and Prevention (CDC) [13].The evaluation described the system and processes of EBS (aims and objectives, description, operation of the system staffing, surveillance data flows) as well as EBS performance (case and outbreak detection, and system experience).As there is no guidance internationally on evaluating surveillance systems specifically in a MG context or on evaluating event-based surveillance systems, we focussed on measuring system attributes particularly important in a MG context and/or in providing lessons for planning for future MG events -i.e.timeliness, sensitivity, positive predictive value (PPV), completeness, usefulness, acceptability, simplicity and system stability.
Definitions for an 'EBS event' operated at a number of levels.We defined an EBS event as any event in England related to an infectious agent affecting an individual or

Table 1
Definition of attributes evaluated for Health Protection Event-Based Surveillance, Regional Operation Centres and HPZone Dashboard reporting systems, 2 July-12 September 2012

Attributes
Health Protection Event-Based Surveillance Regional Operation Centres HPZone Dashboard

Sensitivity
The percentage of all OCC new infectious disease outbreak/incident reports that were reported by EBS as new significant events The percentage of all EBS new significant events that were reported by ROCs as new events of interest (same day or day before) The percentage of all EBS new significant events that were identified as new events of interest from analysis of DB

Positive predictive value
The percentage of new significant events reported by EBS that were subsequently included in the OCC report as new infectious disease outbreak/incident reports/ The percentage of new events of interest reported by ROCs that were subsequently reported by EBS as new significant events The percentage of new events of interest identified by analysis of DB that were subsequently reported by EBS as new significant events EBS events were classified as follows: • 'new events' when the event was reported for the first time; • 'update events' when the event had been previously reported; • 'events of interest' were events reported by ROCs to EBS or those identified on HPZone DB by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (since 1 April 2013, Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone); • 'significant events' were those events reported by EBS to the OCC in the daily EBS situation report.
a group of individuals that (i) could have put the health of those participating, visiting or working at the Games at significant risk; or (ii) was likely to be/had been the subject of media scrutiny that would harm the perception of the Games; or (iii) may have resulted in widespread public concern that needed to be addressed.EBS events were classified as follows: • 'new events' when the event was reported for the first time; • 'update events' when the event had been previously reported; • 'events of interest' were events reported by Regional Operation Centres (ROCs) to EBS or those identified on HPZone Dashboard (DB) by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (HPUs -since 1 April 2013 Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone); • 'significant events' were those events reported by EBS to the Olympic Coordination Centre (OCC) in the daily EBS situation report (SitRep).
We described EBS events by time, place and source of reporting, and by implicated infectious agent and number of cases involved.
To gather information for the evaluation, we undertook a mixture of quantitative and qualitative approaches.EBS, DB, ROC and OCC reports were analysed to assess the completeness, sensitivity, PPV and timeliness of the EBS system.Definitions for the various system attributes measured can be seen in Table 1.
System experience was evaluated via (i) three different web-based surveys of surveillance system participants and/or stakeholders between September and December 2012, which included front-line Olympic focal points in each HPU and ROC directors; and (ii) semi-structured interviews of OCC directors (n=3) (conducted by a single researcher).These focussed on assessing the acceptability, simplicity and usefulness of EBS, and on assessing system costs in terms of staff resources and time.

Results
The main Games-monitoring period for the HPA extended from 2 July to 23 September 2012, i.e. from two weeks before the Olympic Village opening (on 16 July) to two weeks after the finish of the Paralympic Games (on 9 September).EBS activities were conducted on a daily basis for 69 days between 2 July 2012 and 12 September 2012, apart from 7 to 8 July and 18 to 19 August when national Olympic surveillance activities were on an exception report-basis only.EBS was co-located with the OCC based in HPA Victoria, London, and was staffed by a daily duty regional epidemiologist and either a scientist or a public health trainee.

System description and data flows
EBS reported significant events related to infectious diseases for the Games in England between July and September 2012 to the OCC.EBS identified events of interest in two ways.
Firstly, on a daily basis, local HPA staff at each local HPU reported events of interest to their ROC.HPUs used all local intelligence available to identify these events of interest, including notifications from clinicians, laboratories and reports from institutions, e.g.schools, and members of the public.The ROCs then emailed a daily report of events of interest to the EBS team (Box).
Secondly, the EBS team used DB to screen and filter all cases and situations (incidents or outbreaks) entered on HPZone by HPA staff in England.Information was obtained using DB in two ways.The application was programmed so that whenever a case or situation was flagged with an 'Olympic' context, an email with

New reports:
Update from previous reports: C. SIGNIFICANT REGIONAL: Extremely severe or unusual disease occurring anywherein the region or events largely occurring outside the Olympic area but likely to affect populations within the Olympic areas.

New reports:
Update from previous reports: Thanks for reporting to the EBS team to ebs2012@hpa.org.ukEHO: environmental health officers; HPU: Health Protection Unit.
relevant information was sent from DB in real time to the EBS team.Furthermore, the DB was manually screened three times a day using three queries: all situations reported across England; all cases of particular interest (e.g.anthrax or poliovirus infection); and all cases or situations that had been flagged by health protection staff with an Olympic context.
The EBS team screened, filtered, analysed and assessed those events of interest reported by ROCs and identified on DB.The team then reported those assessed as significant events to the OCC by emailing an EBS SitRep by 16:00 each day.Those reports not considered significant, e.g. they were not located near to Olympic areas or were unlikely to impact on people involved in the Games, were not included.Reports on significant events included essential details about infectious agent, number of cases involved, severity of illness, control measures in place and implications for the Games.Overlapping or duplication of reports between the different HPA members collaborating in the Games' surveillance was avoided through a daily teleconference and a preview of the reports by the surveillance teams in Victoria, London, and in the national surveillance centre in Colindale, London.
The OCC issued a daily public health SitRep by 18:00 each day to a range of stakeholders including the UK Department of Health and the London Organising Committee of the Olympic and Paralympic Games including selected information from all HPA Olympic surveillance streams.The OCC SitRep included a section 'Outbreaks and Incidents' where EBS reports (those EBS significant events selected by the OCC) were included.

Detection of events
During the EBS Games-monitoring period, 343 events of interest were reported to the EBS team, of which 11 were discarded as they related to non-infectious hazards.Of the remaining 332 events of interest (mean: 5 per day; standard deviation: 3), 147 (44%) were new events and 185 (56%) were updates.All nine ROCs reported at least one event of interest, with London reporting most events (Figure 1).The median number New events (n=147) Number of reports Regional Operation Centre "EBS events were classified as follows: • 'new events' when the event was reported for the first time; • 'update events' when the event had been previously reported; • 'events of interest' were events reported by Regional Operation Centres to EBS or those identified on HPZone Dashboard (DB) by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (since 1 April 2013, Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone).
of updates per event was two, ranging from 1 to 64 updates, the largest being received for a large regional measles outbreak in the north of the country.
The largest daily number of events of interest reported to EBS was during and immediately after the Olympic Games (27 July to 12 August) (Figure 2).There was also an increase in the number of events of interest reported at the beginning of EBS (early July) and at the end of the Paralympic Games (29 August to 7 September).Most of the troughs in reporting occurred during weekends and bank holidays.Only 18 of 147 new events of interest were reported at weekends and bank holidays, which accounted for 17 of the 69 days of EBS activity.
The most commonly reported events of interest were those related to possible food-borne diseases/pathogens, followed by those related to vaccine-preventable diseases (Table 2).Of the 147 new events of interest reported to EBS, 112 (76.2%) were related to one case and eight (5.4%) did not involve a case, e.g. they were related to an exposure.The remaining 27 events of interest reported (18.4%) were related to a median number of four cases; the maximum number of cases related to a single event was 520 (a regional measles outbreak) and the minimum was two cases.
Of the 147 new events of interest reported to EBS, ROCs reported 79, including three new events of interest not identified in DB by the EBS team (Figure 3).The vast majority of the new events of interest were identified by review of DB (144/147 events of interest).
The EBS staff assessed all the EBS events of interest and identified 61 as EBS significant events, which were then included in the EBS SitRep for reporting to the OCC.These most commonly related to food poisoning (n=16), Escherichia coli infection (n=7) and chickenpox (n=7).This represents a mean of less than one EBS significant event reported each day.
During the Games, the OCC included 21 new reports classified as 'outbreaks or incidents' within the UK, most commonly related to gastroenteritis (n=9) and chickenpox (n=4).

Figure 2
Health 'Events of interest' were events reported by Regional Operation Centres to EBS or those identified on HPZone Dashboard (DB) by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (since 1 April 2013, Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone).

Health Protection Event-Based Surveillance attributes
The The median time period from data entry on HPZone at HPU level to reporting to EBS (EBS timeliness) was one day (10th percentile: 0 days -same day; 90th percentile: 3.6 days).Three events were not identified in HPZone and were therefore excluded from the timeliness analysis.The median time period between a new event being entered in HPZone and the same onset of the event (DB timeliness) was two days (10th percentile: 0 days -same day; 90th percentile: 14.8 days).
Regarding completeness, all but two ROC reports were received out of the 621 expected (99.7% completeness) and all but 25 reports were received by the expected time (96.0%).

System experience
Regarding system stability, during the entire Games period, EBS was always able to collect, manage and provide electronic reports and no downtime or system failures were reported.The daily time dedicated to run EBS at ROC and national EBS level was about 4 hours per unit during weekdays and slightly more than 3 hours per unit at weekends.This time was distributed between different staff, with trainees and consultants bearing the largest proportion of this time -week days 57.9%; weekends 83.1%.
All ROCs responding (eight of nine) rated the simplicity of the EBS events reporting process from HPU to ROCs as good (very good was the highest of five values).Six ROCs rated the EBS level of integration with the other Olympic surveillance systems as fair, two of them as good.
All three OCC directors were interviewed.They were satisfied that EBS met both the EBS objectives and the OCC needs: EBS was judged as an efficient information management system able to gather all information from local and regional levels in a single flow to the OCC.The work was undertaken in a fast, reliable and • 'new events' when the event was reported for the first time; • 'events of interest' were events reported by ROCs to EBS or those identified on HPZone DB by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (since 1 April 2013, Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone); • 'significant events' were those events reported by EBS to the OCC in the daily EBS situation report.
responsive way, and was reported as providing reassurance to the directors that nothing significant would be missed.They regarded EBS as a valuable addition to the overall Games surveillance.

Discussion
Providing early warning signals of potential infectious disease and/or threats of non-infectious environmental hazards is a main objective of public health surveillance systems, which must balance the risk/probability of those threats, the value of early intervention and the finite resources for investigation.This balance becomes more delicate in a MG context -a period of heightened risk with intense political and media scrutiny of the hosting country.Disease surveillance for the Games was built on existing robust routine surveillance systems both locally and nationally in the UK, adding enhancements/ additions to routine systems to improve (primarily) sensitivity and timeliness, and significantly, to provide the added reassurance required in a time of increased scrutiny.
Traditional event-based surveillance is generally recommended as an addition to the basic systems of indicator-based surveillance in order to fill potential gaps and to detect cases or outbreaks that did not enter the basic surveillance net or were not detected in it [14], using external sources of information regarding clusters or cases of diseases, e.g. sales of over-the-counter drugs or media screening.However, while the type of EBS implemented during the Games provided a 'safety net' for existing systems, it used indicator-based as well as event-based reporting sources, and thus did not follow the traditional model.
Evaluations of parts of surveillance systems have been reported from previous Olympic and Paralympic Games and in other sporting MGs, e.g.timeliness (evaluated in the World Cup in Germany, 2006 [15] and the Olympic Games in Barcelona, Spain, 1992 [16], data completeness (Cricket World Cup, West Indies, 2007 [17]), acceptability (Winter Olympic Games, Torino, Italy, 2006 [18-20], and system costs (Olympic Games in Atlanta, United States, 1996) [21].However, there is little guidance on specific system attributes to evaluate for a MG surveillance system, how to measure those attributes or on appropriate indicators for evaluating the effectiveness of surveillance systems in MG [22], neither for indicator-based nor event-based systems.
Thus, our study was an attempt to suggest attributes for evaluation as well as to describe and evaluate the national EBS in place in England during the 2012 Games.
The surveillance system evaluation showed that EBS met its objectives, was timely and sensitive (key attributes in a MG context) and was considered a useful, reliable, stable and acceptable reporting system that met the daily reporting and reassurance needs of the OCC.
The EBS system had over 90% sensitivity.The only new event reported by the OCC and not reported by EBS, a regional Legionnaires' disease outbreak, had been reported by the ROC to EBS, but was not considered significant by the EBS team.The OCC had been informed about it by a different HPA reporting system.
For this analysis, OCC reports were used as the sensitivity analysis denominator, therefore OCC reports were considered as a proxy for identifying all significant events occurring during the Games.It may be possible that one or more significant events were missed by the OCC; however, we consider this unlikely due to the widespread and intense media scrutiny surrounding the Games.Nonetheless, it is possible that some Games participants did not report their illness and if "EBS events were classified as follows: • 'new events' when the event was reported for the first time; • 'update events' when the event had been previously reported; • 'events of interest' were events reported by ROCs to EBS or those identified on HPZone Dashboard (DB) by the EBS team (HPZone is an electronic public health case management tool used by all local Health Protection Units (since 1 April 2013, Health Protection Teams) in England [1] and DB is an application that provides access to summary information on HPZone); • 'significant events' were those events reported by EBS to the OCC in the daily EBS situation report (SitRep).
so, the EBS sensitivity would be over-estimated due to single cases under-reporting.Under-reporting is a common challenge in most surveillance systems.
EBS had a low PPV, i.e. most of what was reported as significant by EBS was not considered significant by the OCC for inclusion in the final SitRep.This was perhaps not surprising as the significant event definition used by the team was very wide and the guidance was to report if unsure, i.e. to focus on a high sensitivity, so that the OCC were kept informed of issues, even if the OCC did not report these events as part of the final OCC SitRep.Furthermore, this was the first time this system had been established and there was little time for systematic refinement of reporting during the Games period.
Two different systems were used to inform EBS, daily emailing from health protection staff and screening by the EBS team of summary information entered into the health protection case management system (DB).Both systems sourced information mainly by infectious disease notifications and local laboratory reporting, but also, thanks to the presence of the HPU in the territory, through local media: therefore EBS was mainly built on established indicator-based surveillance, but also had some components of event-based surveillance.
The DB system had a higher sensitivity, a lower PPV and contained less tailored and detailed information for the EBS team than the emailed reports from Health Protection staff.The DB required no extra local Health Protection staff resources to identify events of interest; however, as little information was available to aid risk assessment, if the EBS team were relying on DB alone, they would have had to contact HPUs for more information to understand the significance of the events identified on DB.This made DB more useful as a screening tool to reassure the EBS team that relevant events were being reported by the ROCs, rather than being able to replace active reporting from Health Protection Staff via email.The analysis showed high acceptability of the system from ROCs.
The risk of using multiple overlapping and parallel systems is that they will interface, to a greater or lesser extent.Participating stakeholders judged EBS as a simple system 'fairly well' integrated with the rest of the Games surveillance.However, running EBS at national level took a substantial amount of time.It is important to be aware that the time calculated does not take into account either the time spent for training and preparation in the two years before the Games, or the time spent at HPU level.
Training, preparation and exercising were crucial and the time needed to do this should not be underestimated.Unlike other surveillance systems, quality could not be improved gradually.EBS had to be robust from the start of the Games.The quality of ROC reporting varied considerably, with some reports lacking the required level of information to allow the EBS team to conduct a robust risk assessment or supply the OCC with sufficient information.Therefore further communication was often needed between EBS and both ROCs and HPUs, and this was at the times when HPUs were already busy responding to the incident in question.More training on the level of information needed within reports may have helped.
OCC directors evaluated EBS as a useful and supportive reporting system, able to provide confidence to the OCC that they were aware of significant events.This was despite the low PPV analyses.This may indicate that although a lot of EBS reported events were not subsequently reported in the OCC SitRep, the OCC appreciated being made aware of them.
The guidelines for evaluating public health surveillance systems by the United States CDC [13] proved to be very useful in our study; however, there is a need to build specific guidance for the evaluation of EI surveillance systems, possibly looking at new attributes better describing the priorities of these systems.
In conclusion, during the EBS surveillance period, there were no significant events related to infectious diseases and no major threats were detected.In this context, EBS acted as a reliable, reassuring, timely, simple and stable national EI tool for the 2012 Games.

London Date: XXXXXX Name: XXXXXX Please
include in the report: a brief description of the incident including the agent, the number of cases, the geographical location, relation to the Olympics (if any), response (control measures) and if there is media interest.
BoxHealth Protection Event-Based Surveillance (EBS) significant event reporting form by Health Protection Agency Regional Operation Centres, England, 2 July-12 September 2012Olympics Event Based Surveillance Daily regional reportPlease refer to Guidance Notes about what constitutes a significant event to report.Region:

volunteers working around the Olympic Live site at Olympic Park were reported by Human Resources with diarrhoea and vomiting. Environmental Health investigating at the Park. Questionnaires are being distributed and public health advice being offered. Risk to the Games assessed as low but investigations underway to confirm this. Update from previous reports: Journalist with suspected food poisoning at Olympic park (initial report from XXX HPU). Information from food history questionnaires for this case and two other journalists reportedly also ill do not indicate any common food link and EHOS have not identified any issues with premises. Two journalist contacts to be followed up with Olympics EHO. No clinical samples are available. However, these are being requested. Risk to the Games assessed as low.
B.LOCAL: Events occurring in the area local to Olympic venues (including training camps) that although not impacting directly on the Olympics, have the potential for spread to involve Olympics personnel or visitors during the time period of Olympic activity in that area (one week before to one week after).
sensitivity of EBS was 95.2%.Of the 21 new reports included in the OCC daily SitRep under 'outbreaks and incidents', 20 were identified by EBS.The new report not previously reported by EBS was a regional outbreak of Legionnaires' disease.The sensitivity of the ROC reports was 91.8%.Of the 61 new significant events included in the EBS daily SitRep, 56 were previously reported by ROCs.The DB sensitivity was 96.7%.Of the 61 new significant events included in the EBS daily SitRep, 59 were identified using DB.

Table 2
Distribution of new events reported by disease/pathogen by the Health Protection Event-Based Surveillance and the Health Protection Olympic Coordination Centre, 2 July-12 September 2012