Successful methodology for large-scale surveillance of severe events following influenza vaccination in Canada, 2011 and 2012

J A Bettinger (jbettinger@cfri.ca)1, I Rouleau2, M C Gariépy2, W R Bowie3, L Valiquette4, O G Vanderkooi5, J D Kellner5, B L Coleman6, S A McNeil7, A McCarthy8, G De Serres2,9, On behalf of the Public Health Agency of Canada/Canadian Institutes for Health Research Influenza Research Network 1. Vaccine Evaluation Center, BC Children’s Hospital, University of British Columbia, Vancouver, British Columbia, Canada 2. Centre de Recherche du CHU de Québec, Laval University, Canada 3. Vancouver General Hospital, University of British Columbia, Vancouver, British Columbia, Canada 4. Centre Hospitalier Universitaire de Sherbrooke, Sherbrooke, Quebec, Canada 5. Alberta Children’s Hospital, University of Calgary, Calgary, Alberta, Canada 6. Mount Sinai Hospital, University of Toronto, Toronto, Ontario, Canada 7. Canadian Center for Vaccinology, IWK Health Centre, Queen Elizabeth II Health Sciences Centre and Dalhousie University, Halifax, Nova Scotia, Canada 8. Ottawa Hospital, Ottawa, Ontario, Canada 9. Institut National de Santé Publique du Quebec, Quebec City, Quebec, Canada

In 2011 and 2012, a nationwide Canadian vaccine safety surveillance network rapidly collected safety data from healthcare workers (HCW) during the first weeks of the annual influenza vaccination campaign. This network provided the first available post-marketing safety data on seasonal influenza vaccines with information on background rates as a comparator. In 2012, these data were used to investigate a possible safety concern regarding a particular vaccine. An online questionnaire was provided to participating HCW two weeks before the annual influenza vaccination campaign for controls, and eight days after influenza vaccination for vaccinees. Control and vaccinees were requested to report health events occurring in the seven days prior to receiving the questionnaire. Control data were used to calculate background rates. HCW reporting a severe event were followed-up by telephone within 48 hours of the online report to validate the report and check on their health status. More than 22,000 vaccinated HCW were enrolled and surveyed over two seasons and > 90% reported no severe event following vaccination. Validated severe event rates were similar in vaccinated HCW and unvaccinated HCW (2.2% vs 2.3%;

Introduction
Influenza vaccines are modified yearly to include the influenza viral strains most likely to circulate during the next influenza season. Starting in 2015, European regulatory requirements to evaluate the safety and immunogenicity of seasonal influenza vaccines in small scale clinical trials will be withdrawn [1]. Such trials had insufficient power to adequately evaluate safety concerns arising from annual formulation changes (e.g. adverse events occurring at a rate of 1-2%). These clinical trials are to be replaced by enhanced, preferably active, safety monitoring and vaccine effectiveness assessments [2].
Recognising the need for timely information supporting the seasonal vaccines' safety profiles early in the annual immunisation campaign, a sentinel network was established in Canada in 2009 [3] to conduct online safety monitoring. The goals of the online surveillance are to detect any safety signals and provide an estimate of severe events reported in vaccinated and non-vaccinated individuals.
As part of the sentinel network, methodology for online, active, safety monitoring was further refined and tested and is described here. This study of the network surveillance aimed to assess health events reported following vaccination, by healthcare workers NA: not applicable. a P-value for difference in characteristics between controls and vaccinees who respectively responded to the online questionnaire in 2011. b P-value for difference in characteristics between controls who responded to the online questionnaire and controls who initially did not respond to the questionnaire (controls initial non-responders) in 2012. c P-value for difference in characteristics between controls and vaccinees who respectively responded to the online questionnaire in 2012. d P-value for difference in characteristics between vaccinees who responded to the online questionnaire and the vaccinees who initially did not respond to the questionnaire (vaccinees initial non-responders) in 2012.    (2011,2012). Significance ≤ 0.002 adjusted for multiple comparisons using Bonferroni's correction. b P-value for difference in health event rate between controls who responded to the online questionnaire and controls who initially did not respond (controls initial non-responders) in 2012. Significance ≤ 0.002 adjusted for multiple comparisons using Bonferroni's correction. c P-value for difference in health event rate between vaccinees who responded to the online questionnaire and vaccinees who initially did not respond (vaccinee initial non-responders) in 2012. Significance ≤ 0.002 adjusted for multiple comparisons using Bonferroni's correction. d Severe event is defined as a health event preventing daily activities or causing work absenteeism or requiring a medical consultation, or any combination of these effects. e Validated events are self-reported severe events in the online questionnaire, which remained the primary diagnosis after a nurse follow-up. (HCW) participating in an online questionnaire. To establish background rates for health events, nonvaccinated HCW were also recruited to respond to the questionnaire. Telephone follow-up of participants reporting severe health events allowed estimation of the validity of such self-reported events. The representativeness of health events reported by online responders was also assessed by comparing the rates of health events in participants who responded to the online questionnaire to those who did not. The study was conducted during the two immunisation seasons of 2011 and 2012, whereby in 2012, prior to the temporary suspension of a seasonal influenza vaccine [4], some data were collected. The brief, voluntary suspension of the vaccine offered a valuable opportunity to assess the capacity of the Canadian network's ability to detect any signal of severe events post-vaccination and to rapidly provide safety data to public health decision makers.

Online surveillance system
HCW who received the influenza vaccine in 2011 or 2012 were recruited to participate in an online survey from seven and eight Canadian acute care hospital sites respectively, in Alberta (2012 only), British Columbia, Nova Scotia, Ontario, and Quebec. The HCW were invited to enrol in the study when presenting for vaccination at a participating hospital, and provided their email address, telephone number(s) and informed consent. Enrolled vaccinated HCW were sent an email eight days after vaccination with a link to an online health event questionnaire. Vaccinee non-responders were sent a reminder email three days later.
Two weeks before the start of the 2011 and 2012 vaccination campaigns respectively at seven of the eight sites, HCW immunised in the previous year were invited to serve as a control group to establish the background rates for health events. Conducting the control survey before the start of influenza vaccination allowed for compliance with national recommendations for all HCW to receive the influenza vaccine and provided a comparable control group for vaccinees. Controls were sent an email with an embedded link to the online surveillance questionnaire which remained active until the day before the start of their institution's influenza vaccination programme (SimpleSurvey v2.17.0, OutSideSoft Solutions inc., Saint-Jean-sur-Richelieu, Quebec). Nonresponders in the control group were sent a reminder email three days after the initial email link was sent.
Participants were identified by a unique study code and email addresses were not linked with the questionnaire responses. The study was approved by the research ethics boards at each site.
The online questionnaires collected information on demographics (i.e. age, sex, occupation), past influenza vaccination history and occurrence of health events of interest. Health events occurring in the seven days before receiving the questionnaire link were documented by broad categories: local injection site reactions (vaccinated HCW only), systemic symptoms (fever as temperature ≥38.5°C, fatigue, myalgia), respiratory symptoms suggestive of allergy-like events, bronchitis, cold, gastrointestinal symptoms (diarrhoea, nausea, vomiting), influenza, pharyngitis, pneumonia, sinusitis, tonsillitis and any other health event. Symptoms of oculorespiratory syndrome (ORS) and numbness (anaesthesia/paraesthesia) were also solicited [5][6][7]. ORS was defined according to the National Advisory Committee on Immunization definition [7]. All events were considered severe if they prevented daily activities, resulted in work absenteeism, or required a medical consultation.

Observed health events and capacity for signal investigation
For each study year, rates of health events reported following vaccination were compared with those observed among controls. Symptoms of ORS or paraesthesia beginning within a seven-day observation period for controls in 2011 or during the previous 24 hours for 2012 controls or 24 hours after vaccination for 2011 and 2012 vaccinees were reported (the wording for these questions was changed in 2012 to make the time period in controls and vaccinees comparable). Before comparison, the number of ORS and paraesthesia events in 2011 controls were divided by seven to adjust for the difference in reporting period in 2011 controls. Characteristics and events were compared using chi-squared and Fischer exact tests.
Following a safety signal issued by Italian authorities and a request from the Public Health Agency of Canada, Novartis Vaccines temporarily suspended Canadian distribution of Agriflu on 26 October 2012 [8,9]. Clumping of virus-like particles in the vaccine prompted concerns about the potential for an increase in ORS or allergylike reactions. The vaccine had already been in use at some of our sites; therefore, we had safety information on the implicated product prior to its temporary suspension. In response to this safety signal, our network compared the event rates observed following vaccination with Agriflu (the signal vaccine) with those observed in controls and after receipt of other seasonal vaccines (Vaxigrip and Fluviral). Individual level data on the vaccines used for HCW vaccination were not available, although 7/8 centres vaccinated HCW with a single product (2 used Fluviral, 4 Vaxigrip and 1 Agriflu exclusively). Only one centre used both Agriflu and Vaxigrip, in unknown proportions and was excluded from this sub-analysis.

Validity of self-reported events by healthcare personnel
Participants who reported any severe health event (i.e. prevented daily activities/work or required a medical consultation) were contacted within 48 hours by a nurse trained in adverse events following vaccination (AEFI) reporting who verified and documented the health event. Participants were allowed to enter several health events of interest occurring during the week, but nurses were required to choose a single primary event (i.e. the main complaint) based on their clinical judgement after speaking with the participant for each individual case report. Respiratory or gastrointestinal symptoms were considered the primary event when reported in conjunction with systemic symptoms. We excluded acute health events that had an onset before vaccination (for vaccinees) or > 1 week before the survey (for controls), as well as events that did not meet the reporting criteria for a severe event, and scheduled medical visits. All events were reviewed and validated centrally by members of the research team (IR, MCG).

Representativeness of events reported by responders
In 2012, 10% of study participants (vaccinees and controls) who did not complete the online questionnaire after the reminder email (non-responders) were randomly selected and contacted by telephone five to 10 days after the reminder email was sent. A minimum of five attempts to contact each non-responder was made on different days and at different times before another non-responder was selected. We compared characteristics and rates of events observed between online responders and non-responders using Bonferonni's correction for the chi-squared and Fischer exact tests.

Study participants
Over the two seasons, 22,080 vaccinated HCW enrolled in the surveillance network (Table 1). Overall, 15,163 (68.7%) responded to the online questionnaire sent eight days following vaccination, although the response rate was statistically higher in 2011 compared with the following year (74.4% vs 63.8%; p < 0.001) ( Table 1). The characteristics of HCW who responded to the online surveillance remained stable over the two study years: about three quarters of participants were women, two thirds were between 30 and 59 years of age and the majority ( Demographic characteristics of controls and vaccinees are shown in Table 1. In both years, respondents younger than 30 years of age were slightly more represented in the vaccinated group than in the controls, and there were minor differences with regard to profession/occupation.

Observed health events and capacity for signal investigation
Over the two study years, 1,388 (9.2%) health events were self-reported by vaccinated HCW compared with 396 (9.7%) health events by controls (p = 0.31) (  Table 2).
The hospitals surveyed in this study conducted their yearly vaccinations campaigns earlier than most jurisdictions in Canada and HCW at one of these hospitals were vaccinated with the vaccine implicated in the safety concern before it was temporarily suspended. This allowed a comparative safety review of the signal vaccine severe event rates with the other seasonal influenza vaccines and the background rates observed in controls within 48 hours of the vaccine suspension.
A sub-analysis of the 2012 dataset was conducted including the vaccinees who responded to the questionnaire (n=7,667), the vaccinated initial online nonresponders who subsequently provided information (n=994), but excluding the one site with mixed vaccine use (n=1,217). This confirmed the interim findings. A total of 1,084 of the 7,444 (15%) vaccinees received the signal vaccine. The rate of self-reported severe events among HCW vaccinated with the signal vaccine was 2.9% (31/1,084), which was similar to 2.4% (151/6,360; p = 0.40) in HCW vaccinated at institutions using other seasonal vaccines and to all controls at 3.9% (133/3,400; p = 0.11). The clinical nature of severe health events reported by HCW vaccinated with the signal vaccine was similar to those reported after other seasonal vaccines (Figure).

Validity of self-reported events
Over the two study seasons, nurses were able to complete follow-up calls with 93% (500/537) of participants reporting severe events online (Table 3). This resulted in 90% (134/149) of controls and 94% (366/388) of vaccinated HCW being followed-up (p = 0.09). Following the nurse interviews, 30% (40/134) of controls and 9% (33/366) of vaccinees reporting severe events were excluded, leaving a total of 427 participants with eligible severe health events (94 controls and 333 HCW).
Reasons for exclusions were that these events (i) started > 1 week before the survey (for controls) or prior to vaccination (for vaccinees), (ii) did not prevent daily activities/work or require a medical consultation or (iii) were previously scheduled medical visits. In both years, the proportion of events that were excluded was significantly higher among controls than that observed among vaccinated individuals (30% vs 9%; p < 0.001).
Participants who were excluded did not vary according to the type of event or clinical presentation, with the notable exception of paraesthesia, which was a preexisting condition in all controls not considered.
The accuracy and validity of the online reported severe health events are shown in Table 3. Among 427 participants reporting eligible severe health events, 45% (n=193) had respiratory symptoms. For 79% (n=153) of these, respiratory symptoms remained the primary diagnosis after talking to the nurse (i.e. validated event). Gastrointestinal symptoms were reported by 28% (121/427) of participants with eligible severe health events, and of those reports, 54% (65/121) remained as the primary diagnosis. Eligible severe systemic symptoms were frequently reported (59% in controls and 71% in vaccinated HCW) (Table 3). However, systemic symptoms reported by controls were more often secondary to another health problem (most often respiratory or gastrointestinal symptoms), and only 15% of controls reporting such symptoms were validated. In contrast, for vaccinated HCW one third (76/281) of systemic events remained the primary diagnosis. While 27 vaccinated HCW (8%) who reported an eligible severe health event had a local reaction ( Table 3), 21 of them missed work or consulted a physician for other health issues.

Observed health events
The overall reporting rate for validated severe events was similar (p=0.7) between vaccinees (2.2%) and controls (2.3%) (

Representativeness of events reported
Study participants who did not respond to the 2012 online questionnaire (i.e. initial non-responders) were contacted by telephone. No difference in age (p=0.3) and sex (p=0.84) could be observed for controls between initial non-responders and responders. Vaccinee initial non-responders were slightly older than vaccinee online responders (p<0.001) but similar in sex (p=0.21) (Table 1). Overall, initial non-responders differed by hospital occupation group (p<0.001). Vaccinated participants who responded online reported more health events (9.0%) across all types than vaccinated non-responders (6.1%), although this difference was not observed among controls ( Table 2). The rate of severe events, however, was generally similar in both responders and initial non-responders. Vaccinee nonresponders reported not answering the online questionnaire due to circumstantial factors, mainly because they reported being too busy (38.2%; 380/994), did not recall receiving the email (30.6%; 304/994), reported that the embedded link to the questionnaire did not work (11.9%; 118/994) or were away or did not check email regularly (8.1%; 80/994).

Discussion
Online monitoring offers an economical and sustainable platform to conduct large-scale electronic surveillance of vaccinated individuals, allows rapid identification of AEFI and minimises human resource needs. However, rapid large-scale surveillance of vaccine safety poses challenges which require a careful balance between information needs and feasibility. The quantity and validity of the information collected must be sufficient to allow stakeholders to detect and interpret safety signals in a timely manner, while requesting a minimal amount of information to obtain sufficient response rates from participants. Self-reported severe events offer the advantage of improved efficiency, but unless validated, may under-or overestimate AEFI reporting rates GI: gastrointestinal; NA: not applicable; ORS: oculorespiratory syndrome. a The event reported was not considered, because it either did not prevent daily activities/work or require a medical consultation, or a medical consultation was pre-existing, or symptoms started prior to vaccination for vaccinees or prior to the reporting period for controls. b Eligible events are events remaining after taking into account reporting errors. c The denominators for the percentages are the total persons for the column in question. d Validated events are events self-reported in online questionnaire that remained the primary diagnosis after a follow-up with a nurse. e Significant difference between validated events in controls and vaccinees at p < 0.05. f Each person could report more than one health event, so the total number of persons is not equal to the total of reported events.
Our results demonstrate that online safety surveillance can be used to effectively monitor influenza vaccine safety in a large number of vaccinees, despite the methodological limitations of relying on self-reported health events. As shown during the influenza A(H1N1)pdm09 pandemic, internet-based safety questionnaires are uniquely suited to rapid collection and analyses and can be adapted to provide monitoring for seasonal influenza vaccines [3,[10][11][12]. The rapid collections of data, early in the mass vaccination campaigns that occur simultaneously across Canada allow for ongoing monitoring and analysis throughout the first weeks of activities and provides an opportunity to detect signals before widespread vaccine use. The ability of online surveillance to detect rare events will depend on the total number of respondents. Our study was able to detect events with a frequency of 1 per 1,000.
Although public health officials were concerned about the possibility of an increase in oculorespiratory syndrome among Agriflu recipients in 2012, event rates observed among HCW vaccinated in centres using this vaccine were similar to the rates observed in centres using other seasonal vaccines and to rates observed in the control group. This was later confirmed by passive surveillance results from the United States Vaccine Adverse Event Reporting System and the Canadian Adverse Event Following Immunization Surveillance System [4,13]. The 2012 interim analysis of data, in response to the temporary suspension of the Agriflu vaccine, confirmed that the network can provide timely evaluation of safety signals and adequately support decision makers. At this time, our network remains the largest able to provide active monitoring of influenza vaccine safety both nationally and internationally. Our findings confirm that influenza vaccines used in Canada for both the 2011 and 2012 seasons were safe and that their safety profiles were consistent with those expected following influenza vaccination.
We also showed that most of the eligible severe events self-reported by vaccinated HCW were consistent with the nurse interviews and had indeed prevented daily activities, resulted in missed work or required a medical consultation. The higher error rate in the control questionnaire reflects the difficulty controls may have in identifying the time period under surveillance and indicates a reference point, through a reminder email, may be needed for this group. Vaccinees have the advantage of a well-defined observation period starting at the vaccination event from which to start tracking any new or exacerbated symptoms. This discrepancy was particularly evident for the paraesthesia questions where control symptoms starting more than one week before the questionnaire period were frequently reported, indicating background rates for chronic conditions or illness may be more difficult to separate from new events using an online questionnaire. This shortcoming was addressed in the severe event follow-up where the difference between controls and vaccinees disappeared when more accurate questioning elicited precise event windows. Reassuringly, most primary diagnoses had indeed been reported by participants, but the main difficulty we encountered in validating health events reported by both controls and vaccinated HCW was in distinguishing the primary complaint from all other health events that occurred during the observation period. This problem was particularly evident for local reactions and systemic symptoms, which often accompanied respiratory and gastrointestinal symptoms, but which alone did not prevent daily activities or lead to absenteeism or medical consultations. The more specific events or symptom questions on the online questionnaire (respiratory symptoms, gastroenteritis, etc.) were more likely to accurately capture a true event than nonspecific event or symptom questions (fever, myalgia, etc.).
The inclusion of a control group in our study is an added strength of the network. It provides background rates for health events just before the start of the influenza vaccination campaign in a similar population and enables precise calculation of risk estimates. Moreover, age and sex specific background rates can be estimated. Importantly background event rates can be compared over multiple years to address fluctuations in events or temporal variations, a potential weakness of the staggered data collection periods of controls and vaccinees.
The similarity in severe event rates between initial non-responders and online responders indicates our online survey participants were representative of their respective vaccine and control groups. This suggests the rates of severe events elicited with our online survey is representative of the group overall.

Limitations
We did not track the total number of individuals who presented at each institution for vaccination or the characteristics of those who were vaccinated but did not enrol in our study. Therefore, we cannot determine whether selection bias occurred at recruitment. Even if our sample is not representative of all HCW, we would not expect the rate of severe events to occur differentially among those who participated and those who did not. In our control group, we had fewer controls that were under the age of 30 years, but the proportion in the remaining age categories was similar, therefore we would not expect this to affect our estimates for severe events. Moreover our severe event rates mirror those seen in other studies collected by different methods [11,14]. The importance of individual-level vaccine information became immediately apparent with the temporary suspension of one vaccine product. Fortunately for our study, only one among the healthcare centres considered used multiple influenza vaccines, so we were able to infer which product individuals received based upon where they were immunised. However, our experience from institutional vaccination of HCW using a single product or vaccine lot may not hold true for children and adults vaccinated in the community. In subsequent years, as a wider range of vaccine products become available, individual-level vaccine product data will be necessary.
HCW who participated in our surveillance constitute a unique group of vaccine recipients, which may not be representative of community vaccinees. Almost 70% of our participants have medical training or are involved in patient care. This likely enables them to better evaluate health problems and communicate chief complaints which may have improved the validity of the online survey. The validity of self-reported events by non-HCW populations may not be similar. Evaluation of this methodology in cohorts of children and non-HCW adults are needed.

Conclusions
Online surveillance can provide rapid assessment of influenza vaccine safety and is highly acceptable to the HCW participating in this activity. The addition of a control group enhances internal validity and establishes background rates for common events of interest. This methodology works particularly well in a mass vaccination setting where large numbers of individuals can be rapidly enrolled and followed-up and meets the new enhanced surveillance requirements as outlined by the EMA.