Eurosurveillance remains in the updated list of the Directory of Open Access Journals (DOAJ). It was first added to the DOAJ on 9 September 2004. Eurosurveillance is also listed in the Securing a Hybrid Environment for Research Preservation and Access / Rights MEtadata for Open archiving (SHERPA/RoMEO) [2], a database which uses a colour‐coding scheme to classify publishers according to their self‐archiving policy and to show the copyright and open access self-archiving policies of academic journals. Eurosurveillance is listed there as a ‘green’ journal, which means that authors can archive pre-print (i.e. pre-refereeing), post-print (i.e. final draft post-refereeing) and archive the publisher's version/PDF.

Follow Eurosurveillance on Twitter: @Eurosurveillanc

Read our articles on Zika virus infection

Note of concern published for 'Epidemiological investigation of MERS-CoV spread in a single hospital in South Korea, May to June 2015',

In this issue

Home Eurosurveillance Edition  2015: Volume 20/ Issue 16 Article 5
Back to Table of Contents
Previous Download (pdf)

Eurosurveillance, Volume 20, Issue 16, 23 April 2015
Research articles
Strengths and limitations of assessing influenza vaccine effectiveness using routinely collected, passive surveillance data in Ontario, Canada, 2007 to 2012: balancing efficiency versus quality
  1. Public Health Ontario, Toronto, Ontario, Canada
  2. Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
  3. Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
  4. British Columbia Centre for Disease Control, Vancouver, British Columbia, Canada
  5. School of Population & Public Health, University of British Columbia, British Columbia, Canada

Citation style for this article: Savage RD, Winter AL, Rosella LC, Olsha R, Gubbay JB, Skowronski DM, Crowcroft NS. Strengths and limitations of assessing influenza vaccine effectiveness using routinely collected, passive surveillance data in Ontario, Canada, 2007 to 2012: balancing efficiency versus quality. Euro Surveill. 2015;20(16):pii=21100. Article DOI:
Date of submission: 28 March 2014

Prompt evaluation of annual influenza vaccine effectiveness (IVE) is important. IVE is estimated in Ontario using a test-negative design (TND) within a national sentinel surveillance network (SPSN). To explore alternative approaches, we applied the screening method (SM) during five seasons spanning 2007 to 2012 to passive surveillance data to determine whether routinely collected data could provide unbiased IVE estimates. Age-adjusted SM-IVE estimates, excluding 2008/09 pandemic cases and cases with missing immunisation status, were compared with TND-IVE estimates in SPSN participants, adjusted for age, comorbidity, week of illness onset and interval to specimen collection. In four seasons, including the 2009 pandemic, the SM underestimated IVE (22–39% seasonal; 72% pandemic) by 20 to 35% relative to the TND-IVE (58–63% seasonal; 93% pandemic), except for the 2010/11 season when both estimates were low (33% and 30%, respectively). Half of the cases in the routine surveillance data lacked immunisation information; imputing all to be unimmunised better aligned SM-IVE with TND-IVE, instead overestimating in four seasons by 4 to 29%. While the SM approach applied to routine data may offer the advantage of timeliness, ease and efficiency, methodological issues related to completeness of vaccine information and/or case ascertainment may constitute trade-offs in reliability.


Estimates of influenza vaccine effectiveness (IVE) are vital to measuring the impact of annual immunisation efforts and ideally can provide timely information to guide the response to epidemics and pandemics. Starting in 2004, IVE has been monitored in several Canadian provinces through application of a test-negative design (TND) whereby vaccination status is compared among test-positive vs test-negative specimens systematically collected from patients presenting with influenza-like illness (ILI) to designated practitioners within a national sentinel physician surveillance network (SPSN) [1-9]. The TND approach has been validated theoretically and in relation to per-protocol analysis of the same randomised controlled trial datasets, generating IVE estimates within ranges published by other observational and clinical trial designs [10-14]. Importantly, by standardising for healthcare-seeking behaviour and indication to request ILI testing, this design reduces potential selection biases that may arise from differential testing of patients with ILI at the discretion of a clinician in a passive surveillance system, and additionally allows for collection and adjustment of relevant confounding variables.

The screening method (SM) is another commonly used approach for estimating IVE that compares immunisation status of influenza cases to that of an external reference group such as the general population [15,16]. Key advantages of the SM include timeliness, ease and efficiency given that individual-level information is only required for cases, for whom information is routinely collected as part of public health surveillance in jurisdictions where influenza is reportable, such as Ontario. Despite this, valid individual-level data on cases’ immunisation status and confounding variables, along with timely population-level coverage data, is difficult to obtain and may result in biased estimates. In light of these competing considerations, the objective of this study was to evaluate whether the SM approach applied to routinely collected surveillance data could be a reliable and timely alternative for annual IVE estimation relative to the TND approach applied to SPSN data in Ontario, Canada.


Since 2000, Ontario has provided publicly funded trivalent inactivated influenza vaccine (TIV) to all people six months and older who live, work or attend school in Ontario [17]. Particularly recommended recipients of the seasonal vaccine include those defined by the National Advisory Committee on Immunization (NACI) as at high risk for influenza-related complications owing to underlying health conditions or age, as well as Aboriginal peoples, those capable of transmitting influenza to high-risk individuals and those who provide essential community services [18]. During the 2009/10 pandemic, an influenza A(H1N1)pdm09 AS03-adjuvanted monovalent vaccine was available, along with a limited supply of non-adjuvanted vaccine and seasonal TIV [6].

The majority (≥60%) of influenza testing is performed by Public Health Ontario (PHO) laboratories; respiratory samples from ambulatory settings undergo culture-based testing only, while samples from inpatient, institutional and outbreak settings are tested by PCR [19]. Laboratory-confirmed cases detected by PHO and other laboratories are reported to regional public health units in Ontario, who then use the integrated Public Health Information System (iPHIS) to report cases, including a minimum set of required epidemiological information obtained through case follow-up, to provincial health authorities; iPHIS therefore captures all influenza cases reported to public health authorities.

The Sentinel Physician Surveillance Network (SPSN), in contrast, is an active surveillance system that exists in the five most populous provinces of Canada, including Ontario, and is designed to assess vaccine effectiveness. As previously described [1-9], this system builds on routine public health infrastructure, namely a sentinel network of primary healthcare practitioners used to monitor weekly ILI consultations at a national and provincial level. Participating sentinel practitioners are encouraged to offer influenza testing to all patients who present within one week of ILI onset meeting a standard case definition. Samples are then tested by real-time RT-PCR for influenza at provincial reference laboratories. Epidemiological information is obtained from consenting patients using a standard questionnaire at the time of specimen collection. Since these data are collected for all patients, including those who may ultimately test negative for influenza, the SPSN surveillance protocol is conducted with annual research ethics board review and approval.

Screening method
Data on laboratory-confirmed influenza cases for five influenza seasons from 2007/08 to 2011/12 were extracted on 10 January 2013 from iPHIS. Immunisation status, identified by self-report, is a requested but not mandatory field in iPHIS that is captured in a general ‘relevant immunisations up to date for client’ field. Data entry guidelines stipulate that ‘relevant’ refers to receipt of the influenza vaccine recommended for the current season at least 14 days before symptom onset and with the correct dosing requirements. The accuracy of this field was examined through comparison with free-text case notes, where available in a non-mandatory comments field, to calculate agreement (measured using the kappa statistic), and by examining potential misclassification of immunisation status in ineligible cases (i.e. infants younger than six months). Vaccine coverage in the general population was derived from Ontario-specific data from the annual Canadian Community Health Survey (CCHS); respondents are interviewed throughout the year about immunisation during the prior 12 months. The CCHS uses a multistage, stratified cluster design and provides cross-sectional data representative of 98% of the Canadian population 12 years and older [20]. For children younger than 12 years, an Ontario influenza vaccination coverage survey conducted in 2007 was used and, in the absence of annual data, applied to all subsequent years [21]. Vaccine coverage estimates for these two age groups were weighted according to the age distribution in Ontario and summed to produce overall population-weighted coverage estimates.

IVE was calculated as (PPV − PCV) / [PPV × (1-PCV)] × 100, where PPV was the proportion of the population vaccinated and PCV was the proportion of cases vaccinated [22]. All IVE estimates were stratified by influenza season, defined as 1 September to 31 August to be consistent with Ontario’s respiratory virus surveillance programme. Cases with laboratory-confirmed influenza A(H1N1)pdm09 in the 2008/09 season were excluded in order to examine homologous IVE only. Cases were also excluded from analysis if they were ineligible for influenza immunisation (younger than six months), reported as epidemiologically linked (but not laboratory-confirmed), exposed to influenza in an institutional setting, hospitalised or if they resided outside of Ontario. Both institutionally exposed and hospitalised cases were excluded to ensure comparability of study populations, as the SPSN only includes cases in the community. IVE was calculated using data from cases with a recorded (yes/no) immunisation status only; cases with an unknown/missing status were excluded. IVE estimates were adjusted for age (6 months–11 years, 12–49, 50–64 and ≥ 65 years) and confidence intervals (CI) were calculated using methods described by Farrington in 1993 [15].

Mid-season IVE estimates were also calculated for two seasons. Mid-season was defined as 1 September 2010 to 15 January 2011 for the 2010/11 season, and 1 September 2011 to 15 March 2012 for the 2011/12 season, based on the mid-point of the month after which the peak of the influenza epidemic occurred for the respective season (5 January and 12 March, respectively, based on case reported dates). The mid-point of the month was used instead of the peak date to be more reflective of the earliest time point that a mid-season analysis would be conducted, recognising that the peak would only be identified by public health officials once it had passed. For the 2010/11 season, PPV was determined based on CCHS survey respondents interviewed between September and December 2010; for the 2011/12 season, coverage was based on respondents interviewed between September 2011 and February 2012. In this analysis, no adjustments were made for age.

Cases with recorded (yes/no) and unrecorded (unknown/missing) immunisation status were compared for demographic characteristics such as age, sex, geographic region of residence and underlying illness or chronic condition to determine if differential misclassification was introduced by excluding cases with an unrecorded status. In iPHIS, underlying illness or chronic condition refers to any self-reported chronic medical condition that puts the individual at greater risk of acquiring the disease or having a more severe outcome. Sensitivity analyses were run, for a range of scenarios, to assess the potential impact of missing data on IVE estimates.

Test-negative design
SM-IVE estimates were compared with TND-IVE estimates derived by the SPSN. Ontario-specific data were used for all seasons except 2007/08 and the 2009/10 pandemic. Patients were considered immunised if the vaccine was given at least two weeks before ILI onset. For the current analysis, immunisation status was compared in test-positive (cases) and test-negative patients (controls). IVE was calculated as (1 − OR) × 100 using logistic regression, where OR represents the odds ratio adjusted for age (6 months–11 years, 12–49, 50–64 and ≥ 65 years), presence/absence of specific chronic conditions (including one or more of heart/pulmonary/renal/metabolic/blood/cancer/immune-compromising conditions, conditions that compromise the management of respiratory secretions and increase risk of aspiration, or morbid obesity), specimen collection interval (≤ 4 days or > 4 days from symptom onset) and time (week of illness onset). Influenza seasons were defined as 1 November to 30 April. A delay in obtaining the ethics board’s approval in Ontario in 2009/10 meant that a substantial proportion of Ontario specimens in that season did not have influenza A(H1N1)pdm09 vaccination status recorded [6]. Published pooled Canada-wide estimates for 2009/10 were therefore used for comparison [6]. Similarly, pooled Canada-wide estimates are presented for 2007/08 as Ontario did not participate in the study until January 2008 [4]. All statistical analyses were conducted using Stata version 12 (StataCorp, College Station, TX).
Ethical approval for the Ontario arm of the SPSN data collection was provided by the University of Toronto. Approval from the research ethics committee was not required to use iPHIS data for the SM analysis as PHO has a legislated mandate “to develop, collect, use, analyse and disclose data, including population health, surveillance and epidemiological data (…) in a manner that informs and enhances healthy public policy and public health planning, evaluation and action” [23].


Study population
Screening method
After excluding 3,982 laboratory-confirmed cases of influenza A(H1N1)pdm09 from the 2008/09 season, a total of 22,954 cases were identified in the study period (Figure). The number of confirmed cases reported in Ontario ranged by season from 3,636 to 6,062 (5,168 in 2007/08; 3,636 in 2008/09; 4,143 in 2009/10; 6,062 in 2010/11; and 3,945 in 2011/12). After applying exclusion criteria, 14,282 cases were included in the analysis; 7,194 (50.4%) cases had a recorded immunisation status (yes/no), while status was unrecorded for 7,088 (49.6%) of cases.

Test-negative design
Excluding the 20 patients who tested positive for influenza A(H1N1)pdm09 during the 2008/09 season and the 152 patients from the 2009/10 season, 2,105 patients with a specimen submitted in Ontario were available for analysis (Figure). Overall, 365 eligible specimens were submitted in 2008/09, 802 in 2010/11 and 938 in 2011/12. After applying study exclusion criteria, 1,747 participants were included in the analysis; 680 (38.9%) tested positive for influenza (cases) and 1,067 tested negative (controls).

Figure. Flowchart of study participants by methodological approach, influenza vaccine effectiveness, Ontario, 2007–2012

Characteristics of included study participants are shown in Table 1 by data source. For all data sources used for IVE estimation (iPHIS and CCHS for SM, and SPSN for TND), individuals aged 12–49 years comprised the largest proportion of cases/study participants; however, iPHIS data included a higher proportion of children (< 12 years) and adults 65 years and older relative to the SPSN. A higher proportion of cases in iPHIS reported the presence of a chronic or underlying illness and being vaccinated against influenza than cases in the SPSN. Comparisons of chronic disease prevalence across study populations should be made with caution, however, as definitions vary across sources.

Table 1. Study population characteristics by data source used for influenza vaccine effectiveness estimation, Ontario, 2007–2012  

Validity of immunisation status in routine surveillance data
Immunisation-related, free-text case notes, written by the public health professional who investigated the case, were available in a comments field in iPHIS for 164 (2.3%) of 7,194 cases with a recorded immunisation status. Agreement between recorded immunisation status and status documented in case notes was able to be assessed in 151 cases of this convenience sample of 164 cases and was found to be high (kappa = 0.88: 95% CI: 0.80–0.96). Review of case notes, available for 19 cases with an unrecorded immunisation status, revealed that 10 cases should have been recorded as unimmunised and the remainder as immunised; these cases remained excluded in IVE calculations.

Among the 808 cases younger than six months who were ineligible to receive the influenza vaccine, 26 of 471 (5.5%) with recorded immunisation status were classified as immunised.

Influenza vaccine effectiveness with screening method by season
For the three study seasons (2008/09, 2010/11, 2011/12) for which Ontario-specific SPSN data were available for comparison, unadjusted point estimates of IVE based on the screening method ranged from 7% (95% CI: −4 to 17%) during the 2011/12 season to 49% (95% CI: 43–55%) in 2008/09 (Table 2). After adjustment for age, this range narrowed from 22% (95% CI: 11–31%) in 2011/12 to 39% (95% CI: 31–46%) in 2008/09. For the 2007/08 season for which only national SPSN data were available for comparison, age-adjusted SM-IVE was 37% (95% CI: 30–44%), substantially lower than the 60% (95% CI: 45–71%) identified through TND analysis of SPSN data. Similarly during the 2009 pandemic, the age-adjusted SM-IVE in Ontario was 72% (95% CI: 63–79%) whereas the national TND-IVE for the adjuvanted monovalent pandemic vaccine was estimated at 93% (95% CI: 69–98%).

For the 2010/11 season, the mid-season IVE was estimated at −11% (95% CI: −33 to 8%), based on a PCV of 22% for 678 cases and a PPV of 21%, which was substantially lower than the unadjusted full-season estimate of 18% (95% CI: 9–27%). For the 2011/12 season, the mid-season IVE estimate of 13% (95% CI: −3 to 27%), based on a PCV of 24% for 715 cases and a PPV of 27%, was similar to the unadjusted full-season estimate of 7% (95% CI: −4 to 17%).

Table 2. Unadjusted and adjusted estimates of influenza vaccine effectiveness by influenza season, comparing screening method and test-negative design, Ontario, 2007 to 2012


Comparison of cases with recorded vs unrecorded immunisation status
Relative to unimmunised cases and those with an unrecorded status, immunised cases were more likely to be 65 years or older (52.9% compared with 5.5% and 14.1%, respectively, p value < 0.001), female (62.6% compared with 51.6% and 52.6%, p value < 0.001) and to have reported chronic or underlying illness (44.2% compared with 18.5% and 31.6%, p value < 0.001) (Table 3). Unimmunised cases were generally more similar to cases with an unrecorded immunisation status in terms of age and sex than to immunised cases. The proportion of cases with an unrecorded immunisation status varied substantially by public health unit (range: 12.4–85.2%); however, this proportion was relatively consistent by season (range: 41.6–45.2%) with the exception of the pandemic year (81.3% in 2009/10) (data not shown), suggesting that particular caution should be applied in the interpretation of the SM-IVE for the pandemic year.

Table 3. Comparison of laboratory-confirmed influenza cases from routine surveillance data (iPHIS) with known (n = 7,194) and unknown (n = 7,088) immunisation status, by key characteristics, Ontario, 2007 to 2012 

Influenza vaccine effectiveness with test-negative design method by season
For the three study seasons (2008/09, 2010/11, 2011/12) for which Ontario-specific TND data were available, unadjusted point estimates of overall IVE ranged from 29% (95% CI: −4 to 52%) in 2010/11 to 58% (95% CI: 25–77%) in 2008/09 (Table 2). These estimates were only slightly increased with adjustment for age, comorbidity, week of illness onset and interval to specimen collection to 30% (95% CI: −6 to 54%) and 63% (95% CI: 30–81%), respectively.

Sensitivity analyses
Restricting the seasons to align with the TND-IVE analysis period (1 November to 30 April for all seasons excluding 2009/10) left SM-IVE estimates either unchanged or increased them slightly (≤ 6%) (data not shown).

An additional sensitivity analysis was performed to evaluate the potential impact of missing data on SM-IVE estimates (Table 4). Unadjusted IVE estimates were substantially altered under both extremes (scenario 1: all cases with unrecorded status were considered immunised; scenario 2: all cases with unrecorded status were considered unimmunised). In scenario 3, where all cases with unrecorded status were assigned the same immunisation distribution as cases with known status, and in scenario 4, where cases with missing data were assigned the average population vaccine coverage estimate for the study time period, SM-IVE estimates became lower still, moving farther away from adjusted TND-IVE estimates. In the more likely scenario that cases with an unrecorded status were unimmunised, SM-IVE estimates became more similar to adjusted TND-IVE estimates exceeding the latter by a range of 4–29% in four of five seasons (Tables 2 and 4).

Table 4. Sensitivity analyses of the potential impact of missing immunisation data on influenza vaccine effectiveness estimates in four different scenarios, Ontario, 2007–2012 (n = 14,282)


In this analysis, we highlight the differences between SM-IVE estimates based on routinely collected, passive surveillance data to gauge influenza vaccine performance compared with an active and systematic method using a TND approach applied to SPSN data. Although the SM approach offers the advantage of ease and efficiency in using existing surveillance data, we demonstrate the potentially important trade-off of reliability. In four of five study seasons, including the 2009 pandemic, the SM underestimated IVE by an absolute difference of ca 20–35% relative to the TND-IVE, except for the 2010/11 season when estimates by both approaches were comparably low. The TND-IVE estimates from Canada, including those cited here, were within the expected range of other studies and comparable to a recently published meta-analysis for which overall pooled IVE has been estimated at 59% (95% CI: 51–67%) [24]; lower TND-IVE estimates for the 2010/11 season were also comparable to IVE estimates from the United States, reported as 31% (95% CI: −7 to 55%) [25].

Inaccuracies and missing data in routine surveillance data may result in misclassification and bias, which may have contributed to the lower SM-IVE estimates. Firstly, immunisation status was not reported for half of all eligible cases in iPHIS. Given that vaccinated cases may be more likely to recall their immunisation status, it is likely that iPHIS data selectively bias unvaccinated individuals to be recorded as missing or unknown. Comparison of cases for key confounding variables supports this hypothesis: unvaccinated cases were more similar in age and sex to cases with an unrecorded status than to vaccinated cases. The proportion of cases with an unrecorded immunisation status who reported a chronic or underlying condition was 31.6%, between that of immunised (44.2%) and unimmunised (18.5%) cases, suggesting that this group may comprise both immunised and unimmunised cases. We cannot, however, rule out that this finding may be attributable to the high proportion of missing data. Our sensitivity analysis demonstrated that in the scenario assuming that cases with an unrecorded immunisation status were not vaccinated, SM-IVE estimates increased and became more similar, if somewhat exceeding TND-IVE estimates. This exceedance provides further support that in reality, cases with an unrecorded immunisation status are likely to include also a small proportion of immunised cases. Still, our decision to exclude cases with missing data in SM-IVE calculations probably overrepresented vaccinated cases, contributing to an artificially high proportion of vaccinated cases in the SM approach (25% vs 17% in the TND), which biased IVE estimates downwards. Although the issue of unvaccinated individuals registering as missing is likely to affect also SPSN, the use of a standardised questionnaire completed by a motivated physician who knows the patient and may have administered the vaccine themselves, meant that less than 5% of cases were excluded. This issue is therefore unlikely to have had a significant impact within the SPSN system.

Secondly, iPHIS data capture persons tested and reported in the public health system without standardisation for testing indication or illness severity, unlike the SPSN. In Ontario, specimens are more likely to be collected from hospitalised patients and those at elevated risk for severe disease [19]. We found that cases captured in iPHIS were more likely to report having a chronic condition or underlying illness than cases from the SPSN. Therefore, iPHIS data are more prone to selection bias by capturing cases at the more severe end of the disease spectrum. Because persons with chronic or underlying conditions are more likely to be vaccinated and less likely to respond to vaccine, it is possible that this also led to the higher proportion of cases vaccinated in iPHIS and to lower SM-IVE estimates. Finally, we cannot discount the role of recall bias in iPHIS data since immunisation status was recorded after the influenza test results were known, an added difference from the SPSN.

We do not anticipate that differences in the diagnostic methods used by the two systems (PCR for SPSN vs both PCR and culture for iPHIS) explain the variation we observed in IVE estimates. While culture methods are less sensitive than PCR, both methods are highly specific [26]. Orenstein et al. have shown that although poor test sensitivity can underestimate IVE as true cases that tend to be distributed in the non-vaccinated group are not detected; test specificity has a greater impact on IVE by increasing the number of false positives in both vaccinated and non-vaccinated groups [10].

Arguably the greatest potential of routine surveillance data lies in the timeliness and efficiency with which this system can accrue a large number of cases, drawing as it does on specimens submitted from all province-wide practitioners. In Ontario, influenza cases are required to be entered into iPHIS within five days of case report to local public health authorities [27]; swift data upload further supports in-season estimates. While our mid- and full-season SM-IVE estimates were similar for 2011/12, we noted a discrepancy for the 2010/11 season (−11% vs 18%). It is unclear whether this discrepancy reflects a true phenomenon or is a result of the aforementioned issues with routine surveillance data. During the 2010/11 season, the vaccine was shown to have suboptimal IVE against a genetic variant of influenza A(H3N2) [7] which was the predominant circulating strain; the latter part of the season, however, also included circulation of influenza A(H1N1)pdm09 for which IVE was moderate, which may have led to an improved IVE estimate when the full season was considered. While this analysis demonstrates retrospectively the capacity of routine data for in-season IVE calculation; in practice, the timeliness of population-based influenza vaccine coverage estimates from CCHS would be a limiting factor. It should be noted that the SPSN has been used successfully to generate interim IVE estimates [9,28]; however, this requires early and intense activity to enable sufficient accrual of sample size and statistical power for IVE estimation within the more limited network of sentinel practitioners and their ILI testing indication.

We identified data quality issues that need to be addressed not only for IVE estimation but also for accurately monitoring immunisation coverage; both activities are vital for guiding effective public health response. The proportion of cases with missing immunisation data varied substantially by health unit as data collection procedures are not standardised nor is immunisation status a mandatory field in iPHIS, suggesting that organisational practices can be modified to improve completeness. Contributing factors for this variation in practice should be investigated to identify and minimise barriers.

Linkage of case-level data to physician billing claims for influenza vaccination recorded in the database of the Ontario Health Insurance Plan (OHIP) may be a strategy to improve completeness of this field and additionally offer data on the timing of immunisation in the absence of an immunisation registry. Kwong et al. recently successfully linked laboratory testing data with the OHIP database to ascertain influenza immunisation status and subsequently estimate IVE in elderly adults [29]. This strategy, however, may lead to misclassification of those vaccinated outside of doctor’s offices, e.g. in work, school and community-based clinics, as unvaccinated [30]. Linkage to health administrative data may similarly improve completeness of data on important covariates, e.g. chronic conditions. Lastly, immunisation data captured in iPHIS have not been validated against physician billing or medical records. In a subset of cases, we found strong agreement with free-text data entered into case notes, which was encouraging regarding the accuracy of the immunisation field. These findings, however, should be interpreted cautiously as they were based on a small convenience sample. Additionally, the question of the reliability of the immunisation field does not address the larger concern regarding missing data. Further work is needed to improve the quality of this information, particularly if immunisation registries are considered in the future for vaccine and programme evaluation.


As health organisations search for efficiency, this study highlights potential pitfalls inherent in using readily available, routine surveillance data for the purpose of IVE estimation. Improved data quality, particularly related to immunisation status and its timing as well as important covariates, is needed. Further work is merited to explore whether linkage with health administration data or ideally a vaccine registry could offer solutions to current data limitations. Fundamentally, valid estimation of vaccine effectiveness through any observational design requires consistent and systematic case finding, ascertainment of vaccination status and comparability of study groups, which ultimately may not be possible to achieve through passive surveillance systems alone. These methodological considerations apply not only within Ontario but also in other regions where annual IVE estimation is of interest. Given the significant implications of IVE findings on public perceptions and prevention measures, ensuring timely and reliable results remains an important goal.

The authors thank Michael Lebenbaum for his contribution to the analysis. Funding for the Sentinel Physician Surveillance Network study of vaccine effectiveness was provided by the Canadian Institutes of Health Research (CIHR) - Institute of Infection and Immunity, grant TPA-90193.

Conflict of interest
None declared

Authors’ contributions
RDS participated in the study’s design, extracted data, performed the analysis and drafted the manuscript. AW and LCR participated in the study’s design, assisted in the analysis and interpretation of the data and helped draft the manuscript. RO extracted data, provided coordination for the Ontario arm of the sentinel study, and assisted in the interpretation of the data. JG provided medical oversight for specimen testing, and assisted in the interpretation of the data. DMS provided medical oversight for the sentinel study, assisted in the analysis and interpretation of the data and helped draft the manuscript. NSC conceived of the study, assisted in the analysis and interpretation of the data and helped draft the manuscript. All authors read, provided feedback on and approved the final manuscript.


  1. Skowronski DM, Gilbert M, Tweed SA, Petric M, Li Y, Mak A, et al. Effectiveness of vaccine against medical consultation due to laboratory-confirmed influenza: results from a sentinel physician pilot project in British Columbia, 2004-2005. Can Commun Dis Rep. 2005;31(18):181-91. PMID:16669132
  2. Skowronski DM, Masaro C, Kwindt TL, Mak A, Petric M, Li Y, et al. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005-2006 season of dual A and B vaccine mismatch in Canada. Vaccine. 2007;25(15):2842-51. Available from: PMID:17081662
  3. Skowronski DM, De Serres G, Dickinson J, Petric M, Mak A, Fonseca K, et al. Component-specific effectiveness of trivalent influenza vaccine as monitored through a sentinel surveillance network in Canada, 2006-2007. J Infect Dis. 2009;199(2):168-79. Available from: PMID:19086914
  4. Janjua NZ, Skowronski DM, De Serres G, Dickinson J, Crowcroft NS, Taylor M, et al. Estimates of influenza vaccine effectiveness for 2007-2008 from Canada’s sentinel surveillance system: cross-protection against major and minor variants. J Infect Dis. 2012;205(12):1858-68. Available from: PMID:22492921
  5. Skowronski DM, De Serres G, Crowcroft NS, Janjua NZ, Boulianne N, Hottes TS, et al.; Canadian SAVOIR Team. Association between the 2008-09 seasonal influenza vaccine and pandemic H1N1 illness during Spring-Summer 2009: four observational studies from Canada. PLoS Med. 2010;7(4):e1000258. Available from: PMID:20386731
  6. Skowronski DM, Janjua NZ, De Serres G, Hottes TS, Dickinson JA, Crowcroft N, et al. Effectiveness of AS03 adjuvanted pandemic H1N1 vaccine: case-control evaluation based on sentinel surveillance system in Canada, autumn 2009. BMJ. 2011;342(feb03 1):c7297. PMID:21292718
  7. Skowronski DM, Janjua NZ, De Serres G, Winter AL, Dickinson JA, Gardy JL, et al. A sentinel platform to evaluate influenza vaccine effectiveness and new variant circulation, Canada 2010-2011 season. Clin Infect Dis. 2012;55(3):332-42. Available from: PMID:22539661
  8. Janjua NZ, Skowronski DM, De Serres G, Winter A, Dickinson JA, Mahmud SM, et al. Component-Specific Estimates of 2011-12 Influenza Vaccine Effectiveness Based on the Canadian Sentinel Surveillance System. 10th Canadian Immunization Conference, Vancoouver, BC 3-5 December 2012:P001.
  9. Skowronski DM, Janjua NZ, De Serres G, Dickinson JA, Winter AL, Mahmud SM, et al. Interim estimates of influenza vaccine effectiveness in 2012/13 from Canada’s sentinel surveillance network, January 2013. Euro Surveill. 2013;18(5):20394. PMID:23399422
  10. Orenstein EW, De Serres G, Haber MJ, Shay DK, Bridges CB, Gargiullo P, et al. Methodologic issues regarding the use of three observational study designs to assess influenza vaccine effectiveness. Int J Epidemiol. 2007;36(3):623-31. Available from: PMID:17403908
  11. De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill. 2013;18(37):20585. Available from: PMID:24079398
  12. Frey S, Vesikari T, Szymczakiewicz-Multanowska A, Lattanzi M, Izu A, Groth N, et al. Clinical efficacy of cell culture–derived and egg‐derived inactivated subunit influenza vaccines in healthy adults. Clin Infect Dis. 2010;51(9):997-1004. Available from: PMID:20868284
  13. Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, et al. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7. Available from: PMID:19776407
  14. Fleming DM, Andrews NJ, Ellis JS, Bermingham A, Sebastianpillai P, Elliot AJ, et al. Estimating influenza vaccine effectiveness using routinely collected laboratory data. J Epidemiol Community Health. 2010;64(12):1062-7. Available from: PMID:19910645
  15. Farrington CP. Estimation of vaccine effectiveness using the screening method. Int J Epidemiol. 1993;22(4):742-6. PMID:8225751
  16. Valenciano M, Kissling E, Ciancio BC, Moren A. Study designs for timely estimation of influenza vaccine effectiveness using European sentinel practitioner networks. Vaccine. 2010;28(46):7381-8. Available from: PMID:20851086
  17. Ontario Ministry of Health and Long-Term Care. Influenza Vaccine Fact Sheet. Toronto: Queen’s Printer for Ontario. [Accessed: 20 Apr 2015]. Available from:
  18. National Advisory Committee on Immunization. Statement on Seasonal Influenza Vaccine for 2013-2014. Canada Communicable Disease Report. 2013;39. Available from:
  19. Influenza and other respiratory viral testing for the 2013-2014 influenza season (November 4, 2013 to May 5, 2014). Labstract - November 2013; Toronto: Public Health Ontario. [Accessed: 29 Jan 2014]. Available from:
  20. Statistics Canada. Canadian community health survey - annual component (CCHS). Ottawa: Queen’s Printer for Canada. [Accessed: 19 Nov 2013]. Available from:
  21. Moran K, Maaten S, Guttmann A, Northrup D, Kwong JC. Influenza vaccination rates in Ontario children: implications for universal childhood vaccination policy. Vaccine. 2009;27(17):2350-5. Available from: PMID:19428850
  22. Seyler T, Bella A, Puzelli S, Donatelli I, Rizzo C. Screening method working group. Estimating pandemic vaccine effectiveness in two Italian regions using the screening method, 2009-2010. Vaccine. 2012;30(2):109-11. Available from: PMID:22100893
  23. Ontario Agency for Health Protection and Promotion Act. SO 2007, c 10. Toronto: Queen’s Printer for Ontario. [Accessed: 20 Apr 2015]. Available from:
  24. Osterholm MT, Kelley NS, Sommer A, Belongia EA. Efficacy and effectiveness of influenza vaccines: a systematic review and meta-analysis. Lancet Infect Dis. 2012;12(1):36-44. Available from: PMID:22032844
  25. Ohmit SE, Petrie JG, Malosh RE, Cowling BJ, Thompson MG, Shay DK, et al. Influenza vaccine effectiveness in the community and the household. Clin Infect Dis. 2013;56(10):1363-9. Available from: PMID:23413420
  26. Weinberg GA, Erdman DD, Edwards KM, Hall CB, Walker FJ, Griffin MR, et al. Superiority of reverse-transcription polymerase chain reaction to conventional viral culture in the diagnosis of acute respiratory tract infections in children. J Infect Dis. 2004;189(4):706-10. Available from: PMID:14767825
  27. Ontario Ministry of Health and Long-Term Care. Infectious disease protocol, 2013: Appendix A: Disease-specific chapters (Influenza). Toronto; Queen’s Printer for Ontario; 2009. Available from:
  28. Skowronski D, Chambers C, Sabaiduc S, De Serres G, Dickinson J, Winter A, et al. Interim estimates of 2013/14 vaccine effectiveness against influenza A(H1N1)pdm09 from Canada s sentinel surveillance network, January 2014. Euro Surveill. 2014;19(5):20690. Available from: PMID:24524234
  29. Kwong JC, Campitelli MA, Gubbay JB, Peci A, Winter AL, Olsha R, et al. Vaccine effectiveness against laboratory-confirmed influenza hospitalizations among elderly adults during the 2010-2011 season. Clin Infect Dis. 2013;57(6):820-7. Available from: PMID:23788243
  30. Kwong JC, Manuel DG. Using OHIP physician billing claims to ascertain individual influenza vaccination status. Vaccine. 2007;25(7):1270-4. Available from: PMID:17079057

Back to Table of Contents
Previous Download (pdf)

The publisher’s policy on data collection and use of cookies.

Disclaimer: The opinions expressed by authors contributing to Eurosurveillance do not necessarily reflect the opinions of the European Centre for Disease Prevention and Control (ECDC) or the editorial team or the institutions with which the authors are affiliated. Neither ECDC nor any person acting on behalf of ECDC is responsible for the use that might be made of the information in this journal. The information provided on the Eurosurveillance site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her physician. Our website does not host any form of commercial advertisement. Except where otherwise stated, all manuscripts published after 1 January 2016 will be published under the Creative Commons Attribution (CC BY) licence. You are free to share and adapt the material, but you must give appropriate credit, provide a link to the licence, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

Eurosurveillance [ISSN] - ©2007-2016. All rights reserved

This website is certified by Health On the Net Foundation. Click to verify. This site complies with the HONcode standard for trustworthy health information:
verify here.