Strengths and limitations of assessing influenza vaccine effectiveness using routinely collected , passive surveillance data in Ontario , Canada , 2007 to 2012 : balancing efficiency versus quality

R D Savage (rachel.savage@mail.utoronto.ca)1,2, A L Winter1, L C Rosella1,2, R Olsha1, J B Gubbay1,3, D M Skowronski4,5, N S Crowcroft1,2,3 1. Public Health Ontario, Toronto, Ontario, Canada 2. Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada 3. Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada 4. British Columbia Centre for Disease Control, Vancouver, British Columbia, Canada 5. School of Population & Public Health, University of British Columbia, British Columbia, Canada


Introduction
Estimates of influenza vaccine effectiveness (IVE) are vital to measuring the impact of annual immunisation efforts and ideally can provide timely information to guide the response to epidemics and pandemics.Starting in 2004, IVE has been monitored in several Canadian provinces through application of a test-negative design (TND) whereby vaccination status is compared among test-positive vs test-negative specimens systematically collected from patients presenting with influenza-like illness (ILI) to designated practitioners within a national sentinel physician surveillance network (SPSN) [1][2][3][4][5][6][7][8][9].The TND approach has been validated theoretically and in relation to per-protocol analysis of the same randomised controlled trial datasets, generating IVE estimates within ranges published by other observational and clinical trial designs [10][11][12][13][14]. Importantly, by standardising for healthcareseeking behaviour and indication to request ILI testing, this design reduces potential selection biases that may arise from differential testing of patients with ILI at the discretion of a clinician in a passive surveillance system, and additionally allows for collection and adjustment of relevant confounding variables.
The screening method (SM) is another commonly used approach for estimating IVE that compares immunisation status of influenza cases to that of an external reference group such as the general population [15,16].Key advantages of the SM include timeliness, ease and efficiency given that individual-level information is only required for cases, for whom information is routinely collected as part of public health surveillance in jurisdictions where influenza is reportable, such as Ontario.Despite this, valid individual-level data on cases' immunisation status and confounding variables, along with timely population-level coverage data, is difficult to obtain and may result in biased estimates.In light of these competing considerations, the objective of this study was to evaluate whether the SM approach applied to routinely collected surveillance data could be a reliable and timely alternative for annual IVE estimation relative to the TND approach applied to SPSN data in Ontario, Canada.

Context
Since 2000, Ontario has provided publicly funded trivalent inactivated influenza vaccine (TIV) to all people six months and older who live, work or attend school in Ontario [17].Particularly recommended recipients of the seasonal vaccine include those defined by the National Advisory Committee on Immunization (NACI) as at high risk for influenza-related complications owing to underlying health conditions or age, as well as Aboriginal peoples, those capable of transmitting influenza to high-risk individuals and those who provide essential community services [18].During the 2009/10 pandemic, an influenza A(H1N1)pdm09 AS03adjuvanted monovalent vaccine was available, along with a limited supply of non-adjuvanted vaccine and seasonal TIV [6].
The majority (≥60%) of influenza testing is performed by Public Health Ontario (PHO) laboratories; respiratory samples from ambulatory settings undergo culture-based testing only, while samples from inpatient, institutional and outbreak settings are tested by PCR [19].Laboratory-confirmed cases detected by PHO and other laboratories are reported to regional public health units in Ontario, who then use the integrated Public Health Information System (iPHIS) to report cases, including a minimum set of required epidemiological information obtained through case followup, to provincial health authorities; iPHIS therefore captures all influenza cases reported to public health authorities.
The Sentinel Physician Surveillance Network (SPSN), in contrast, is an active surveillance system that exists in the five most populous provinces of Canada, including Ontario, and is designed to assess vaccine effectiveness.As previously described [1][2][3][4][5][6][7][8][9], this system builds on routine public health infrastructure, namely a sentinel network of primary healthcare practitioners used to monitor weekly ILI consultations at a national and provincial level.Participating sentinel practitioners are encouraged to offer influenza testing to all patients who present within one week of ILI onset meeting a standard case definition.Samples are then tested by real-time RT-PCR for influenza at provincial reference laboratories.Epidemiological information is obtained from consenting patients using a standard questionnaire at the time of specimen collection.Since these data are collected for all patients, including those who may ultimately test negative for influenza, the SPSN surveillance protocol is conducted with annual research ethics board review and approval.

Screening method
Data on laboratory-confirmed influenza cases for five influenza seasons from 2007/08 to 2011/12 were extracted on 10 January 2013 from iPHIS.Immunisation status, identified by self-report, is a requested but not mandatory field in iPHIS that is captured in a general 'relevant immunisations up to date for client' field.Data entry guidelines stipulate that 'relevant' refers to receipt of the influenza vaccine recommended for the current season at least 14 days before symptom onset and with the correct dosing requirements.The accuracy of this field was examined through comparison with free-text case notes, where available in a non-mandatory comments field, to calculate agreement (measured using the kappa statistic), and by examining potential misclassification of immunisation status in ineligible cases (i.e.infants younger than six months).Vaccine coverage in the general population was derived from Ontario-specific data from the annual Canadian Community Health Survey (CCHS); respondents are interviewed throughout the year about immunisation during the prior 12 months.The CCHS uses a multistage, stratified cluster design and provides cross-sectional data representative of 98% of the Canadian population 12 years and older [20].For children younger than 12 years, an Ontario influenza vaccination coverage survey conducted in 2007 was used and, in the absence of annual data, applied to all subsequent years [21].Vaccine coverage estimates for these two age groups were weighted according to the age distribution in Ontario and summed to produce overall population-weighted coverage estimates.IVE was calculated as (PPV − PCV) / [PPV × (1-PCV)] × 100, where PPV was the proportion of the population vaccinated and PCV was the proportion of cases vaccinated [22].All IVE estimates were stratified by influenza season, defined as 1 September to 31 August to be consistent with Ontario's respiratory virus surveillance programme.Cases with laboratory-confirmed influenza A(H1N1)pdm09 in the 2008/09 season were excluded in order to examine homologous IVE only.Cases were also excluded from analysis if they were ineligible for influenza immunisation (younger than six months), reported as epidemiologically linked (but not laboratory-confirmed), exposed to influenza in an institutional setting, hospitalised or if they resided outside of Ontario.Both institutionally exposed and hospitalised cases were excluded to ensure comparability of study populations, as the SPSN only includes cases in the community.IVE was calculated using data from cases with a recorded (yes/no) immunisation status only; cases with an unknown/missing status were excluded.IVE estimates were adjusted for age (6 months-11 years, 12-49, 50-64 and ≥ 65 years) and confidence intervals (CI) were calculated using methods described by Farrington in 1993 [15].
Mid-season IVE estimates were also calculated for two seasons.Mid-season was defined as 1 September 2010 to 15 January 2011 for the 2010/11 season, and 1 September 2011 to 15 March 2012 for the 2011/12 season, based on the mid-point of the month after which the peak of the influenza epidemic occurred for the respective season (5 January and 12 March, respectively, based on case reported dates).The mid-point of the month was used instead of the peak date to be more reflective of the earliest time point that a midseason analysis would be conducted, recognising that the peak would only be identified by public health officials once it had passed.For the 2010/11 season, PPV was determined based on CCHS survey respondents interviewed between September and December 2010; for the 2011/12 season, coverage was based on respondents interviewed between September 2011 and February 2012.In this analysis, no adjustments were made for age.
Cases with recorded (yes/no) and unrecorded (unknown/missing) immunisation status were compared for demographic characteristics such as age, sex, geographic region of residence and underlying illness or chronic condition to determine if differential misclassification was introduced by excluding cases with an unrecorded status.In iPHIS, underlying illness or chronic condition refers to any self-reported chronic medical condition that puts the individual at greater risk of acquiring the disease or having a more severe outcome.Sensitivity analyses were run, for a range of scenarios, to assess the potential impact of missing data on IVE estimates.

Test-negative design
SM-IVE estimates were compared with TND-IVE estimates derived by the SPSN.Ontario-specific data were used for all seasons except 2007/08 and the 2009/10 pandemic.Patients were considered immunised if the vaccine was given at least two weeks before ILI onset.For the current analysis, immunisation status was compared in test-positive (cases) and test-negative patients (controls).IVE was calculated as (1 − OR) × 100 using logistic regression, where OR represents the odds ratio adjusted for age (6 months-11 years, 12-49, 50-64 and ≥ 65 years), presence/absence of specific chronic conditions (including one or more of heart/pulmonary/ renal/metabolic/blood/cancer/immune-compromising

Figure
Flowchart of study participants by methodological approach, influenza vaccine effectiveness, Ontario, 2007-2012 Approval from the research ethics committee was not required to use iPHIS data for the SM analysis as PHO has a legislated mandate "to develop, collect, use, analyse and disclose data, including population health, surveillance and epidemiological data (…) in a manner that informs and enhances healthy public policy and public health planning, evaluation and action" [23].

Validity of immunisation status in routine surveillance data
Immunisation-related, free-text case notes, written by the public health professional who investigated the case, were available in a comments field in iPHIS for 164 (2.3%) of 7,194 cases with a recorded immunisation status.Agreement between recorded immunisation status and status documented in case notes was able to be assessed in 151 cases of this convenience sample of 164 cases and was found to be high (kappa = 0.88: 95% CI: 0.80-0.96).Review of case notes, available for 19 cases with an unrecorded immunisation status, revealed that 10 cases should have been recorded as unimmunised and the remainder as immunised; these cases remained excluded in IVE calculations.
Among the 808 cases younger than six months who were ineligible to receive the influenza vaccine, 26 of 471 (5.5%) with recorded immunisation status were classified as immunised.

Comparison of cases with recorded vs unrecorded immunisation status
Relative to unimmunised cases and those with an unrecorded status, immunised cases were more likely to be 65 years or older (52.9% compared with 5.5% and 14.1%, respectively, p value < 0.001), female (62.6% compared with 51.6% and 52.6%, p value < 0.001) and  d The difference between methods is calculated as the difference between adjusted IVE estimates in the test-negative design relative to the screening method.e National estimate provided due to limited Ontario sample size.
to have reported chronic or underlying illness (44.2% compared with 18.5% and 31.6%,p value < 0.001) (Table 3).Unimmunised cases were generally more similar to cases with an unrecorded immunisation status in terms of age and sex than to immunised cases.The proportion of cases with an unrecorded immunisation status varied substantially by public health unit (range: 12.4-85.2%);however, this proportion was relatively consistent by season (range: 41.6-45.2%)with the exception of the pandemic year (81.3% in 2009/10) (data not shown), suggesting that particular caution should be applied in the interpretation of the SM-IVE for the pandemic year.

Sensitivity analyses
Restricting the seasons to align with the TND-IVE analysis period (1 November to 30 April for all seasons excluding 2009/10) left SM-IVE estimates either unchanged or increased them slightly (≤ 6%) (data not shown).
An additional sensitivity analysis was performed to evaluate the potential impact of missing data on SM-IVE estimates (Table 4).Unadjusted IVE estimates were substantially altered under both extremes (scenario 1: all cases with unrecorded status were considered immunised; scenario 2: all cases with unrecorded status were considered unimmunised).In scenario 3, where all cases with unrecorded status were assigned the same immunisation distribution as cases with known status, and in scenario 4, where cases with missing data were assigned the average population vaccine coverage estimate for the study time period, SM-IVE estimates became lower still, moving farther away from adjusted TND-IVE estimates.In the more likely scenario that cases with an unrecorded status were unimmunised, SM-IVE estimates became more similar to adjusted TND-IVE estimates exceeding the latter by a range of 4-29% in four of five seasons (Tables 2 and 4).

Discussion
In this analysis, we highlight the differences between SM-IVE estimates based on routinely collected, passive surveillance data to gauge influenza vaccine performance compared with an active and systematic method using a TND approach applied to SPSN data.Although the SM approach offers the advantage of ease and efficiency in using existing surveillance data, we demonstrate the potentially important trade-off of reliability.In four of five study seasons, including the 2009 pandemic, the SM underestimated IVE by an absolute difference of ca 20-35% relative to the TND-IVE, except for the 2010/11 season when estimates by both approaches were comparably low.The TND-IVE estimates from Canada, including those cited here, were within the expected range of other studies and comparable to a recently published meta-analysis for which overall pooled IVE has been estimated at 59% (95% CI: 51-67%) [24]; lower TND-IVE estimates for the 2010/11 season were also comparable to IVE estimates from the United States, reported as 31% (95% CI: −7 to 55%) [25].
Inaccuracies and missing data in routine surveillance data may result in misclassification and bias, which may have contributed to the lower SM-IVE estimates.Firstly, immunisation status was not reported for half of all eligible cases in iPHIS.Given that vaccinated cases may be more likely to recall their immunisation status, it is likely that iPHIS data selectively bias unvaccinated individuals to be recorded as missing or unknown.
Comparison of cases for key confounding variables supports this hypothesis: unvaccinated cases were more similar in age and sex to cases with an unrecorded status than to vaccinated cases.The proportion of cases with an unrecorded immunisation status who reported a chronic or underlying condition was 31.6%, between that of immunised (44.2%) and unimmunised (18.5%) cases, suggesting that this group may comprise both immunised and unimmunised cases.We cannot, however, rule out that this finding may be attributable to the high proportion of missing data.Our sensitivity analysis demonstrated that in the scenario assuming that cases with an unrecorded immunisation status were not vaccinated, SM-IVE estimates increased and became more similar, if somewhat exceeding TND-IVE estimates.This exceedance provides further support that in reality, cases with an unrecorded immunisation status are likely to include also a small proportion of immunised cases.Still, our decision to exclude cases with missing data in SM-IVE calculations probably overrepresented vaccinated cases, contributing to an artificially high proportion of vaccinated cases in the SM approach (25% vs 17% in the TND), which biased IVE estimates downwards.Although the issue of unvaccinated individuals registering as missing is likely to affect also SPSN, the use of a standardised questionnaire completed by a motivated physician who knows the patient and may have administered the vaccine themselves, meant that less than 5% of cases were excluded.This issue is therefore unlikely to have had a significant impact within the SPSN system.
Secondly, iPHIS data capture persons tested and reported in the public health system without standardisation for testing indication or illness severity, unlike the SPSN.In Ontario, specimens are more likely to be collected from hospitalised patients and those at elevated risk for severe disease [19].We found that cases captured in iPHIS were more likely to report having a chronic condition or underlying illness than cases from the SPSN.Therefore, iPHIS data are more prone to selection bias by capturing cases at the more severe end of the disease spectrum.Because persons with chronic or underlying conditions are more likely to be vaccinated and less likely to respond to vaccine, it is possible that this also led to the higher proportion of cases vaccinated in iPHIS and to lower SM-IVE estimates.Finally, we cannot discount the role of recall bias in iPHIS data since immunisation status was recorded after the influenza test results were known, an added difference from the SPSN.
We do not anticipate that differences in the diagnostic methods used by the two systems (PCR for SPSN vs both PCR and culture for iPHIS) explain the variation we observed in IVE estimates.While culture methods are less sensitive than PCR, both methods are highly specific [26].cases allocated to the same immunisation distribution as cases with known status for all seasons combined (coverage: 25%).Scenario 4: cases assigned the same distribution as the average CCHS estimate for the study time period (coverage: 34%).
group are not detected; test specificity has a greater impact on IVE by increasing the number of false positives in both vaccinated and non-vaccinated groups [10].
Arguably the greatest potential of routine surveillance data lies in the timeliness and efficiency with which this system can accrue a large number of cases, drawing as it does on specimens submitted from all province-wide practitioners.In Ontario, influenza cases are required to be entered into iPHIS within five days of case report to local public health authorities [27]; swift data upload further supports in-season estimates.While our mid-and full-season SM-IVE estimates were similar for 2011/12, we noted a discrepancy for the 2010/11 season (−11% vs 18%).It is unclear whether this discrepancy reflects a true phenomenon or is a result of the aforementioned issues with routine surveillance data.During the 2010/11 season, the vaccine was shown to have suboptimal IVE against a genetic variant of influenza A(H3N2) [7] which was the predominant circulating strain; the latter part of the season, however, also included circulation of influenza A(H1N1) pdm09 for which IVE was moderate, which may have led to an improved IVE estimate when the full season was considered.While this analysis demonstrates retrospectively the capacity of routine data for in-season IVE calculation; in practice, the timeliness of population-based influenza vaccine coverage estimates from CCHS would be a limiting factor.It should be noted that the SPSN has been used successfully to generate interim IVE estimates [9,28]; however, this requires early and intense activity to enable sufficient accrual of sample size and statistical power for IVE estimation within the more limited network of sentinel practitioners and their ILI testing indication.
We identified data quality issues that need to be addressed not only for IVE estimation but also for accurately monitoring immunisation coverage; both activities are vital for guiding effective public health response.The proportion of cases with missing immunisation data varied substantially by health unit as data collection procedures are not standardised nor is immunisation status a mandatory field in iPHIS, suggesting that organisational practices can be modified to improve completeness.Contributing factors for this variation in practice should be investigated to identify and minimise barriers.
Linkage of case-level data to physician billing claims for influenza vaccination recorded in the database of the Ontario Health Insurance Plan (OHIP) may be a strategy to improve completeness of this field and additionally offer data on the timing of immunisation in the absence of an immunisation registry.Kwong et al. recently successfully linked laboratory testing data with the OHIP database to ascertain influenza immunisation status and subsequently estimate IVE in elderly adults [29].This strategy, however, may lead to misclassification of those vaccinated outside of doctor's offices, e.g. in work, school and community-based clinics, as unvaccinated [30].Linkage to health administrative data may similarly improve completeness of data on important covariates, e.g.chronic conditions.Lastly, immunisation data captured in iPHIS have not been validated against physician billing or medical records.In a subset of cases, we found strong agreement with free-text data entered into case notes, which was encouraging regarding the accuracy of the immunisation field.These findings, however, should be interpreted cautiously as they were based on a small convenience sample.Additionally, the question of the reliability of the immunisation field does not address the larger concern regarding missing data.Further work is needed to improve the quality of this information, particularly if immunisation registries are considered in the future for vaccine and programme evaluation.

Conclusions
As health organisations search for efficiency, this study highlights potential pitfalls inherent in using readily available, routine surveillance data for the purpose of IVE estimation.Improved data quality, particularly related to immunisation status and its timing as well as important covariates, is needed.Further work is merited to explore whether linkage with health administration data or ideally a vaccine registry could offer solutions to current data limitations.Fundamentally, valid estimation of vaccine effectiveness through any observational design requires consistent and systematic case finding, ascertainment of vaccination status and comparability of study groups, which ultimately may not be possible to achieve through passive surveillance systems alone.These methodological considerations apply not only within Ontario but also in other regions where annual IVE estimation is of interest.Given the significant implications of IVE findings on public perceptions and prevention measures, ensuring timely and reliable results remains an important goal.
assisted in the analysis and interpretation of the data and helped draft the manuscript.NSC conceived of the study, assisted in the analysis and interpretation of the data and helped draft the manuscript.All authors read, provided feedback on and approved the final manuscript.

Table 1
Study population characteristics by data source used for influenza vaccine effectiveness estimation, Ontario, 2007-2012 Age unknown for nine cases, and 4,005 cases were not asked about whether they had a chronic illness or underlying condition.Cases with missing data have been excluded from denominators in proportion calculations.b Proportions weighted for the survey design are presented.
After excluding 3,982 laboratory-confirmed cases of influenza A(H1N1)pdm09 from the 2008/09 season, a total of 22,954 cases were identified in the study period (Figure).The number of confirmed cases reported in Ontario ranged by season from 3,636 to 6,062 (5,168 in 2007/08; 3,636 in 2008/09; 4,143 in 2009/10; 6,062 in 2010/11; and 3,945 in 2011/12).After applying exclusion criteria, 14,282 cases were included in the analysis; 7,194 (50.4%) cases had a recorded immunisation status (yes/no), while status was unrecorded for 7,088 (49.6%) of cases.a c Definition varied across data source.
a Denominator for proportion calculation comprises cases with known immunisation status.b 'Controls vaccinated' for the screening method refers to the proportion of the population vaccinated (PPV), which is based on populationbased, provincial survey data.c

Table 3
Comparison of laboratory-confirmed influenza cases from routine surveillance data (iPHIS) with known (n = 7,194) and unknown (n = 7,088) immunisation status, by key characteristics, Ontario, 2007 to 2012 a Pearson's chi-squared test.b 35 cases with unknown age.c 63 cases with other or unknown sex.d 9,554 cases not asked about whether they had a chronic illness or underlying condition.Cases with missing data have been excluded from denominators in proportion calculations.

Table 4
Orenstein et al. haveshown that although poor test sensitivity can underestimate IVE as true cases that tend to be distributed in the non-vaccinated Sensitivity analyses of the potential impact of missing immunisation data on influenza vaccine effectiveness estimates in four different scenarios, Ontario, 2007-2012 (n = 14,282) Scenario 1: all cases with unknown or missing status classified as immunised.Scenario 2: all cases classified as unimmunised.Scenario 3: