12 September 2013
The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials
The test-negative design (TND) is an efficient form of case–control study commonly applied to influenza vaccine effectiveness (VE) estimation. TND validity is predicated on the core assumption that the intervention (vaccine) has no effect on other non-targeted aetiologies resulting in similar illness/disease. Here we verify this core assumption and compare efficacy estimates derived by the TND versus classical per-protocol analysis of four datasets obtained from randomised placebo-controlled clinical trials (RCT) of the live attenuated influenza vaccine (LAIV) in children ≤7 years-old and the elderly ≥60 years-old. We further assess generalisability of the TND approach in two other RCT datasets to evaluate monoclonal antibody in the prevention of respiratory syncytial virus (RSV) hospitalisation. Efficacy estimates and their confidence intervals were virtually identical for per-protocol RCT versus TND analyses of LAIV and also for RSV monoclonal antibody. Neither LAIV nor monoclonal antibodies affected the risk of disease aetiologies that were not specifically targeted by the respective interventions (e.g. other respiratory viruses). This study validates the core assumption of the TND approach for influenza vaccine efficacy estimation and confirms the accuracy and precision of its estimates compared to the gold standard of classic per-protocol RCT analysis of the same data sets. The TND approach is generalisable for other conditions such as RSV for which the core assumption is also met. However, when used in observational studies, the TND, like all designs, still requires assessment for bias and confounding that may exist in the absence of randomised participation and blinded follow-up.
The test-negative design (TND) was developed as an efficient approach to assess influenza vaccine effectiveness (VE) using available sentinel surveillance structures. First publications based on the TND for influenza VE estimation came from Canada in 2005 (for the pilot 2004/05 season)  and 2007 (for the subsequent 2005/06 season) . In Canada this approach has been used within existing surveillance structures annually since [3-8]. Following a publication on the methodological validity of the TND , other investigators in Europe [10-13], the United States [14-16] and Australia [17,18] also began to publish VE findings based on the TND from 2009 onward.
The TND is a type of case–control design whereby vaccine status is compared between influenza test-positive cases versus test-negative controls who present to a clinician, generally with some standardised definition of influenza-like illness (ILI). While classical case–control studies require intense efforts to recruit non-diseased controls, the TND draws controls from the same source population as the cases, namely ill patients who are tested to identify a specific aetiology of interest for their illness or disease. It is therefore a convenient and relatively low-cost design that also has recently demonstrated its usefulness in rapidly gauging vaccine protection early in the influenza season [8,13,16].
The simplicity of the TND approach, however, has understandably raised concerns about its validity, including the misclassification of cases as controls due to imperfect test sensitivity. Theoretical work has previously shown that in fact test specificity rather than sensitivity is the most critical factor influencing VE estimation based on the TND . Given high test specificity and prevalence of the targeted aetiology (i.e. influenza) equal to or lower than that of other aetiologies (i.e. other respiratory viruses) with similar clinical presentation (i.e. acute respiratory illness (ARI) or ILI), the TND performs comparably to the classical case–control or cohort design, even with suboptimal test sensitivity . Most published studies applying this method to influenza VE estimation have used highly specific diagnostic methods such as polymerase chain reaction (PCR) confirmation [1-8,10-18]. Detection of influenza by culture is nearly 100% specific . As culture has historically been considered the gold standard, its sensitivity is very high especially in young children (<5 years-old) although PCR detects 2% to 13% more cases . In elderly patients (>65 years-old), viral culture has a sensitivity between 21% and 51% compared to PCR .
The efficacy of a preventive intervention (e.g. vaccine) reflects proportionate reduction in the frequency of the targeted disease in those receiving the intervention (vaccine) compared to individuals who did not receive the intervention and is ideally assessed in the optimal conditions of suitably powered randomised placebo-controlled clinical trials (RCT). For vaccines, efficacy is calculated by comparing attack rates (ARs) in the vaccinated and unvaccinated through the relative risk (RR) and according to the following equation :
Like efficacy, effectiveness also compares proportionate reduction of risk but this is estimated in field conditions through observational studies of the intervention, without randomisation to address other possible influences. Efficacy and effectiveness are calculated the same way. In RCT and in cohort studies where a census of the source population is available, ARs are calculated by dividing the number of vaccinated and unvaccinated cases of the disease in question by the total number of individuals belonging to their respective categories. In case–control studies where there is no census of the source population, effectiveness can be validly estimated by the odds ratio (OR) assuming that controls are a representative sample of that population and the exposure distribution (e.g. vaccine coverage) is the same as in the source population . For this condition to be met and a control series to be valid, the sampling fraction (θ) must be the same in vaccinated and unvaccinated non-diseased controls. The θs then cancel out when calculating the OR which approximates the RR when the disease is rare (Table 1). For the TND approach, the condition of representative exposure distribution (e.g. vaccine coverage) in controls emerging from the same source population will apply if the intervention has no effect on other aetiologies manifesting similar clinical presentation as the target pathogen. For the TND as applied to influenza vaccine efficacy/effectiveness estimation, internal validity is therefore predicated on the core assumption that influenza vaccine has no effect on non-influenza causes (e.g. other respiratory pathogens such as parainfluenza, respiratory syncytial viruses (RSV)) of ARI or ILI. These non-influenza episodes would then be expected to occur at the same mean frequency (f) per vaccinated or unvaccinated individual (i.e. fvaccinated= funvaccinated). Consequently, among individuals affected by these other aetiologies, the proportion who are vaccinated should be similar to the vaccine coverage in the source population (Table 1).
Table 1. Comparison of the randomised placebo-controlled clinical trial (RCT)/cohort, classical case–control and test-negative design (TND) case–control
Large double-blind RCTs optimise the comparability of vaccinated and unvaccinated individuals with respect to eligibility criteria, follow-up, and disease ascertainment thereby minimising the influence of bias and confounding. As such, the RCT represents the ideal context to assess the validity of the TND core assumption and to verify the accuracy and precision of efficacy estimates derived in that way. Using four datasets from large double-blind RCTs of live attenuated influenza vaccine (LAIV) among children and the elderly we have therefore directly compared original RCT per-protocol efficacy estimates against those instead derived by TND analysis. To test the core assumption hypothesis that influenza vaccine has no effect on other ARI/ILI aetiologies we have also derived efficacy against non-influenza causes of illness. Finally, to illustrate the generalisability of the TND we applied it to two RCT datasets collected for the evaluation of humanised monoclonal antibody (palivizumab) in preventing RSV hospitalisation.
Four datasets from published, double-blind RCTs of LAIV (Flumist, MedImmune) among children and the elderly were used [24-27]. In these studies, after being administered vaccine or placebo, participants were actively followed throughout the winter season by phone calls or home visits and nasal/throat swabs were collected for each episode of ARI (Table 2). Respiratory specimens were tested by viral culture and the primary outcome was culture-confirmed influenza (test-positive) due to any strain regardless of antigenic similarity. Specimens with influenza-negative culture were not further tested to identify other specific causative pathogens. For the RSV studies, we analysed datasets from two published double-blind RCTs of palivizumab administered every 30 days during the RSV season to premature infants (≤35 weeks gestation) or infants with bronchopulmonary dysplasia  and to children with haemodynamically significant congenital heart disease . The primary outcome was RSV-associated hospitalisation where diagnosis of RSV was confirmed by rapid antigen detection test (sensitivity: 82%, specificity: 95% ). No further testing was done to identify the aetiology in patients with RSV negative tests.
Table 2. Characteristics of participants in the live attenuated influenza vaccine (LAIV) and palivizumab randomised placebo-controlled clinical trials (RCT)
Per-protocol randomised cohort estimates of efficacy were calculated using ARs according to the above equation. Three approaches to TND analysis of LAIV protection were conducted. In participant-based analysis without censoring for influenza, controls included participants with any negative swabs without excluding those who may have tested positive for influenza at another time within the study period (Table 1). In participant-based analysis with censoring for influenza, controls included only participants with negative swabs who furthermore never had a positive test for influenza at any other time during the study period (i.e. excludes those test-positive for influenza). In specimen-based analysis, cases were influenza-positive specimens and controls were influenza-negative specimens rather than individuals: this approach accounts for the multiple episodes of respiratory infections not attributable to influenza that an individual can sustain during the study period.
To estimate vaccine effects on other non-influenza ARI aetiologies, the AR of these infections was calculated by dividing the number of participants/specimens testing negative for influenza by the total enrolment in their respective categories of exposure (vaccine or placebo). Per above, this was conducted as participant-based (with/without censoring) and specimen-based analysis.
Each of the above was repeated for the RSV studies, modified for the intervention (palivizumab) and outcome of interest (laboratory-confirmed RSV hospitalisation).
Consistent with the expectation of large, randomised placebo-controlled participation in these RCT data sets, we assumed that risk factors for influenza were similarly distributed in both groups before vaccination and that blinding ensured comparable follow-up and case ascertainment, minimising bias and confounding. Consequently, only crude results are presented without further statistical adjustment. Point estimates of efficacy and their 95% confidence intervals (CI) for TND analyses were computed using Mantel–Haenszel method as (1-(OR or RR) X 100).
Each of these RCTs had received prior Institutional Review Board/Ethics Committee approval. The current analysis involved only additional statistical analyses of de-identified data and thus no additional approvals were required.
Live attenuated influenza vaccine (LAIV) studies
Three RCT datasets among children ≤7 years of age [24-26] and one among adults ≥60 years of age  were analysed. Together these trials included 6,077 participants each actively monitored for outcomes of interest accrued over the course of a single season. The first paediatric study included children who were vaccinated with LAIV and followed for one season. The second study recruited the same paediatric participants whose parents gave consent for revaccination and follow-up for a second season. For all LAIV trials, groups were shown in publication to be balanced with respect to baseline characteristics so that further adjustment for residual confounding was not required.
For all LAIV studies, the point estimates and surrounding 95% CIs for efficacy against ARI due to influenza were virtually identical in the classical per-protocol RCT and TND analyses with little variation using these datasets whether TND analysis was participant-based (with/without censoring) or specimen-based (Table 3). Good concordance between TND and RCT analysis approaches was observed both in paediatric studies with high efficacy and in the elderly study where protection was lower. LAIV had negligible effect on non-influenza aetiologies of ARI as shown by the zero or near-zero efficacy associated with test-negativity (Table 3).
Table 3. Influenza vaccine efficacy against influenza and non-influenza illnesses estimated by per-protocol and various test-negative design (TND) analysis approaches in four randomised placebo-controlled clinical trials (RCT) of live attenuated influenza vaccine (LAIV)
To more closely represent surveillance-based TND approaches as extensively published [1-8,10-18], we also assessed the same parameters for medically-attended ARI. Again the point estimates and surrounding 95% CIs for efficacy were virtually identical in the classical per-protocol RCT and TND analyses (Figure).
Figure. Efficacy (% of disease prevented) of live attenuated influenza vaccine (LAIV) against medically-attended influenza acute respiratory illness estimated by per-protocol and various test-negative design (TND) analysis approaches in four randomised placebo-controlled clinical trials (RCT) and efficacy (% of RSV hospitalisation prevented) by palivizumab in two RCTs
Respiratory syncytial virus studies
For the two RSV trials, groups were shown in the original publications to be balanced on baseline characteristics except household smoking in one trial ; adjusted analysis did not influence efficacy estimates and we did not pursue adjustment here. In these studies, repeated hospitalisation during the study period was rare. Therefore all three TND analysis approaches gave similar results and we present only the participant-based analysis with censoring (Table 4 and Figure). Point estimates and 95% CI for palivizumab protection against RSV hospitalisation were virtually identical by the per-protocol RCT and the TND analysis. Palivizumab provided no protection against non-RSV causes of acute respiratory hospitalisation (RSV test-negative) as again shown by the null efficacy.
Table 4. Efficacy of palivizumab to prevent RSV and non-RSV hospitalisation by per-protocol and test-negative design (TND) participant-based analysis with censoring in two randomised placebo-controlled clinical trials (RCT) conducted in children.
In this analysis, we confirm that estimates of efficacy and their 95% CIs were similar when derived according to the classical per-protocol RCT analysis or various TND approaches and this was observed in children and the elderly at respectively high or reduced efficacy values. When applied to data for an unrelated passive immunising agent targeting RSV, the TND gave similar results, thereby demonstrating its generalisability beyond influenza and to endpoints representing more severe disease outcomes such as hospitalisation.
To obtain valid case–control study results, controls must be representative of the source population . As such, they should belong to the same source population from which cases were identified and should be individuals who theoretically would have been identified as cases had they acquired the targeted aetiology of interest. The actual source population from which clinic or hospital cases emerge is often undefined. In classic case–control studies, controls are usually recruited among patients consulting or admitted to the same facility as cases but the disease for which they consult may be distinct from that which the targeted aetiology typically manifests. In contrast, test-negative controls derived from among patients presenting with similar clinical illness (e.g. ARI or ILI) and tested for diagnosis provide some inherent reassurance that they emerge from the same source population as cases, would have consulted and would have been considered cases had their aetiology been the targeted pathogen rather than otherwise. More than thirty years ago, Broome et al. applied this sort of approach in using patients infected by non-vaccine type invasive pneumococcal infection to serve as controls in their analysis of pneumococcal VE . The TND as now applied to influenza VE is an extension of the same logic. The main advantage of the TND is its ease of access to a series of controls representative of the source population. Here we have shown that this simplicity does not necessarily come at the cost of validity.
We explicitly presented three TND analysis approaches. While in this paper all three approaches performed similarly this may not always be true. Censored/uncensored participant-based and specimen-based approaches are not intrinsically the same and the choice of one approach over another must be consciously expressed. As displayed in Table 1, the uncensored and the specimen-based analyses directly derive the RR of influenza in vaccinated versus unvaccinated individuals. Conversely the analysis censoring for influenza positive participants shifts the effect measure to an OR that is necessarily sensitive to the rare disease assumption. Even if the vaccine does not actually influence the risk of non-influenza aetiologies, unvaccinated individuals should more often be infected by influenza during the study period assuming that vaccine is protective against influenza. As such, censoring participants based on influenza positivity eliminates more participants with non-influenza aetiologies in the unvaccinated than the vaccinated group thereby introducing a bias that skews findings toward suggesting the vaccinated are at increased risk of these other aetiologies. This bias will increase with greater prevalence of influenza compared to other respiratory infections in the study population. These concerns related to censoring will apply not only to TND, but also to other fixed cohort follow-up or case–control analyses, requiring authors to be explicit in the approach taken when interpreting their results.
In our analysis, there was a statistically significant but slight increased risk of non-influenza respiratory episodes during the first year of the Belshe study but this was not observed in the three other datasets for which efficacy against other respiratory viruses even more closely approximated the null. It may be argued that the same may not extend to inactivated or other influenza vaccine formulations. While another small trial from Cowling et al. involving 115 participants followed during one season reported an increased risk of non-influenza viruses among recipients of inactivated influenza vaccine  a much larger study covering six seasons and including more than 3,000 patients found no such association . It could be argued that our findings of null vaccine effects against other respiratory viruses are explained by the fact that live virus vaccine is itself predicated on replicating virus and may thereby also directly contribute to broadly cross-protective innate immune mechanisms that are precluded by effective inactivated vaccine (a theory proposed to explain Cowling’s findings ). Although active follow-up of thousands of participants included in our own datasets resulted in one to 3.5 specimens on average per participant (≤10% positive for influenza), the specific non-influenza cause of ARI was not sought in any of the RCTs we used so that we cannot directly address possible vaccine effects on individual pathogens. However, temporary innate immunity is short-lasting, in the range of several weeks and in each of these trials LAIV was administered well before the winter period. LAIV-induced innate immunity is thus unlikely to have substantially altered the overall risk of other respiratory viruses through the full follow-up period. On that basis we believe that our findings supporting the core assumption of the TND for LAIV can also be extended to inactivated formulations although we encourage direct assessment of that through other similarly available RCT data sets. If vaccine nevertheless truly does increase the likelihood of other non-influenza infections by whatever mechanism, this would generally tend to over-estimate efficacy/effectiveness against influenza suggesting TND findings are optimistic representations of vaccine performance.
There are other issues and limitations worth considering in our analysis. RCT estimates and cohort studies provide absolute measures (attack rates) whereas under the usual conditions of TND application there is no census of the source population and analysis can only provide relative measures. We have validated the TND approach for influenza and RSV but this does not imply universal validity in the evaluation of all infectious diseases or their interventions. Before extending the TND to other vaccines or interventions, it is necessary to confirm earlier specified pre-conditions related to test characteristics and the mix of target/non-target disease aetiologies  as well as the core assumption of no effect on non-targeted aetiologies of diseases with similar symptomatology. In the influenza vaccine studies we used, influenza was the targeted aetiology of ARI and positive viral culture for influenza was the main outcome. The sensitivity of viral culture to detect influenza in specimens collected early after the onset of disease is expected to be high in young children but lower in the elderly . Despite that, the TND performed comparably well to De Villiers’ RCT analysis in the elderly. The current analysis took advantage of existing large clinical trial databases where participants had been randomised and followed in blinded fashion to identify the outcome with stringent approaches applied equally to immunised and unimmunised participants. This most likely minimised differences between the two groups, eliminated confounding and provided efficacy estimates. Effectiveness derived from observational studies, in contrast, is susceptible to these additional methodological concerns. As such, the TND approach is valid but cannot compensate for other methodological flaws. Indiscriminate use of the TND in observational studies can lead to errors of interpretation, particularly if testing was applied differentially and varied with the likelihood of immunisation, exposure and/or test-positivity. TND should be considered as a variation on the case–control design and as for all observational designs, one needs to begin from the premise that bias and confounding may be operating. Intense scrutiny of the methods and data set for signals of bias (e.g. selection, information, confounding bias) remains a requirement.
In conclusion, the TND approach appears valid not only for influenza vaccine efficacy and effectiveness assessment but also for other diseases and interventions provided that the core assumption requiring no effect of the preventive intervention on non-targeted aetiologies is fulfilled and that bias as a potential concern for all observational designs is adequately addressed.
Funding: Costs related to professional time were covered by the institutions/organisations of the individual investigators. No additional financial support for this analysis was provided
Conflict of interest
GDS received research grants from GSK and Sanofi Pasteur and participated to an adhoc GSK advisory board meeting for which travel expenses were reimbursed. CSA and XWW are MedImmune employees.
The study was conceived and led by Gaston De Serres and Danuta M Skowronski. Christopher Ambrose identified and provided access to the appropriate clinical trial data, and Xionghua Wilson Wu conducted the statistical analysis. All authors contributed to the analysis of the data, the writing of the manuscript and its intellectual content.
- Skowronski DM, Gilbert M, Tweed SA, Petric M, Li Y, Mak A, et al. Effectiveness of vaccine against medical consultation due to laboratory-confirmed influenza: results from a sentinel physician pilot project in British Columbia, 2004-2005. Can Commun Dis Rep. 2005;31(18):181-91. Available from:http://www.phac-aspc.gc.ca/publicat/ccdr-rmtc/05vol31/dr3118a-eng.php
- Skowronski DM, Masaro C, Kwindt TL, Mak A, Petric M, Li Y, et al. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005-2006 season of dual A and B vaccine mismatch in Canada. Vaccine. 2007;25(15):2842-51.
- Skowronski DM, De Serres G, Dickinson J, Petric M, Mak A, Fonseca K et al. Component-specific effectiveness of trivalent influenza vaccine as monitored through a sentinel surveillance network in Canada, 2006-2007. J Infect Dis. 2009;199(2):168-79.
- Janjua NZ, Skowronski DM, De Serres G, Dickinson J, Crowcroft NS, Taylor M, et al. Estimates of influenza vaccine effectiveness for 2007-2008 from Canada's sentinel surveillance system: Cross-protection against major and minor variants. J Infect Dis. 2012;205(12):1858-68.
- Skowronski DM, De Serres G, Crowcroft NS, Janjua NZ, Boulianne N, Hottes TS, et al. Association between the 2008-09 seasonal influenza vaccine and pandemic H1N1 illness during Spring-Summer 2009: four observational studies from Canada. PLoS Med. 2010;7(4):e1000258.
- Skowronski DM, Janjua NZ, De Serres G, Hottes TS, Dickinson JA, Crowcroft N, et al. Effectiveness of AS03-adjuvanted pandemic H1N1 vaccine: case-control evaluation based on the sentinel surveillance system in Canada, autumn 2009. BMJ. 2011;342: c7297.
- Skowronski DM, Janjua NZ, De Serres G, Winter A-L, Dickinson JA, Gardy JL, et al. A Sentinel Platform to Evaluate Influenza Vaccine Effectiveness and New Variant Circulation, Canada 2010-2011 Season. Clin Infect Dis. 2012;55(3):332-42.
- Skowronski DM, Janjua NZ, De Serres G, Dickinson JA, Winter AL, Mahmud SM, et al. Interim estimates of influenza vaccine effectiveness in 2012/13 from Canada's sentinel surveillance network, January 2013 Euro Surveill. 2013;18(5):pii=20394. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20394
- Orenstein EW, De Serres G, Haber MJ, Shay D K, Bridges CB, Gargiullo P et al. Methodologic issues regarding the use of three observational study designs to assess influenza vaccine effectiveness. Int J Epidemiol. 2007;36(3): 623-31.
- Kissling E, Valenciano M, Falcao J, Larrauri A, Widgren K, Pitigoi D, et al. I-Move towards monitoring seasonal and pandemic influenza vaccine effectiveness: lessons learnt from a pilot multi-centric case-control study in Europe 2008-9. Euro Surveill. 2009;14(44):pii=19388. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19388
- Kissling E, Valenciano M, Larrauri A, Oroszi B, Cohen JM, Nunes B et al. Low and decreasing vaccine effectiveness against influenza A(H3) in 2011/12 among vaccination target groups in Europe: results from the I-MOVE multicentre case–control study. Euro Surveill. 2013;18(5):pii=20390. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20390
- Pebody RG, Andrews N, McMenamin J, Durnall H, Ellis J, Thompson CI et al. Vaccine effectiveness of 2011/12 trivalent seasonal influenza vaccine in preventing laboratory-confirmed influenza in primary care in the United Kingdom: evidence of waning intra-seasonal protection. Euro Surveill. 2013;18(5):pii=20389. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20389
- McMenamin J, Andrews N, Robertson C, Fleming D, Durnall H, von Wissmann B et al. Effectiveness of seasonal 2012/13 vaccine in preventing laboratory-confirmed influenza infection in primary care in the United Kingdom: mid-season analysis 2012/13. Euro Surveill. 2013;18(5):pii: 20393. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20393
- Belongia EA, Kieke BA, Donahue JG, Greenlee RT, Balish A, Foust A et al. Effectiveness of inactivated influenza vaccines varied substantially with antigenic match from 2004-2005 to the 2006-2007 season. J Infect Dis. 2009;199(2):159-67.
- Treanor JJ, Talbot HK, Ohmit SE, Coleman LA, Thompson MG, Cheng PY et al. Effectiveness of seasonal influenza vaccines in the United States during a season with circulation of all three vaccine strains. Clin Infect Dis. 2012;55(7):951-9.
- Centers for Disease Control and Prevention (CDC). Early estimates of seasonal influenza vaccine effectiveness-United States, January 2013. MMWR Morb Mortal Wkly Rep. 2013;62(2):32-5.
- Kelly H, Carville K, Grant K, Jacoby P, Tran T, Barr I. Estimation of the influenza vaccine effectiveness from routine surveillance data. PLoS One. 2009;4(3):e5079.
- Fielding JE, Grant KA, Tran T, Kelly HA. Moderate influenza vaccine effectiveness in Victoria, Australia, 2011. Euro Surveill. 2012;17(11): pii=20115. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=20115
- Petric M, Comanor L, Petti CA. Role of the laboratory in diagnosis of influenza during seasonal epidemics and potential pandemics. J Infect Dis. 2006;194 Suppl 2:S98-110.
- World Health Organization (WHO).WHO recommendations on the use of rapid testing for influenza diagnosis. Geneva: WHO. [Accessed 16 Aug 2013]. Available from http://www.who.int/influenza/resources/documents/RapidTestInfluenza_WebVersion.pdf
- Talbot HK, Falsey AR. The diagnosis of viral respiratory disease in older adults. Clin Infect Dis. 2010;50(5):747-51.
- Orenstein WA, Bernier RH, Hinman AR. Assessing vaccine efficacy in the field. Further observations. Epidemiol Rev. 1988;10:212-41.
- Rothman KJ, Greenland S, Lash TL, editors. Case-control studies. Modern Epidemiology 3rd Edition. Philadelphia: Lippincott Williams & Wilkins; 2008, p111-128.
- Belshe RB, Mendelman PM, Treanor J, King J, Gruber WC, Piedra P et al. The efficacy of live attenuated, cold-adapted, trivalent, intranasal influenzavirus vaccine in children. N Engl J Med. 1998;338(20):1405-12.
- Belshe RB, Gruber WC, Mendelman PM, Cho I, Reisinger K, Block SL et al. Efficacy of vaccination with live attenuated, cold-adapted, trivalent, intranasal influenza virus vaccine against a variant (A/Sydney) not contained in the vaccine. J Pediatr. 2000;136(2):168-75.
- Lum LC, Borja-Tabora CF, Breiman RF, Vesikari T, Sablan BP, Chay OM et al Influenza vaccine concurrently administered with a combination measles, mumps, and rubella vaccine to young children. Vaccine. 2010;28(6):1566-74.
- De Villiers PJ, Steele AD, Hiemstra LA, Rappaport R, Dunning AJ, Gruber WC et al. Efficacy and safety of a live attenuated influenza vaccine in adults 60 years of age and older Vaccine. 2009;28(1):228-34.
- Impact-RSV study group. Palivizumab, a humanized respiratory syncytial virus monoclonal antibody, reduces hospitalization from respiratory syncytial virus infection in high-risk infants. Pediatrics. 1998;102(3 Pt 1):531-7.
- Feltes TF, Cabalka AK, Meissner HC, Piazza FM, Carlin DA, Top FH Jr et al Palivizumab prophylaxis reduces hospitalization due to respiratory syncytial virus in young children with hemodynamically significant congenital heart disease. J Pediatr 2003;143(4):532-40. http://dx.doi.org/10.1067/S0022-3476(03)00454-2
- Boivin G, Côté S, Déry P, De Serres G, Bergeron MG. Multiplex real-time PCR assay for detection of influenza and human respiratory syncytial viruses. J Clin Microbiol. 2004;42(1):45-51.
- Broome CV, Facklam RR, Fraser DW. Pneumococcal disease after pneumococcal vaccination: an alternative method to estimate the efficacy of pneumococcal vaccine. N Engl J Med. 1980;303(10):549-52.
- Cowling BJ, Fang VJ, Nishiura H, Chan KH, Ng S, Chiu SS et al. Increased Risk of Noninfluenza Respiratory Virus Infections Associated With Receipt of Inactivated Influenza Vaccine. Clin Infect Dis. 2012;54(12):1778-82.
- Sundaram ME, McClure DL, VanWormer JJ, Friedrich TC, Meece JK, Belongia EA. Influenza vaccination is not associated with detection of non-influenza respiratory viruses in seasonal studies of influenza vaccine effectiveness. Clin Infect Dis. 2013 Epub June 6 2013.