26 February 2015
A pilot validation in 10 European Union Member States of a point prevalence survey of healthcare-associated infections and antimicrobial use in acute hospitals in Europe, 2011
We present a pilot validation study performed on 10 European Union (EU) Member States, of a point prevalence survey (PPS) of healthcare-associated infections (HAIs) and antimicrobial use in Europe in 2011 involving 29 EU/European Economic Area (EEA) countries and Croatia. A total of 20 acute hospitals and 1,950 patient records were included in the pilot study, which consisted of validation and inter-rater reliability (IRR) testing using an in-hospital observation approach. In the validation, a sensitivity of 83% (95% confidence interval (CI): 79–87%) and a specificity of 98% (95% CI: 98–99%) were found for HAIs. The level of agreement between the primary PPS and validation results were very good for HAIs overall (Cohen’s κappa (κ): 0.81) and across all the types of HAIs (range: 0.83 for bloodstream infections to 1.00 for lower respiratory tract infections). Antimicrobial use had a sensitivity of 94% (95% CI: 93–95%) and specificity of 97% (95% CI: 96–98%) with a very good level of agreement (κ: 0.91). Agreement on other demographic items ranged from moderate to very good (κ: 0.57–0.95): age (κ: 0.95), sex (κ: 0.93), specialty of physician (κ: 0.87) and McCabe score (κ: 0.57). IRR showed a very good level of agreement (κ: 0.92) for both the presence of HAIs and antimicrobial use. This pilot study suggested valid and reliable reporting of HAIs and antimicrobial use in the PPS dataset. The lower level of sensitivity with respect to reporting of HAIs reinforces the importance of training data collectors and including validation studies as part of a PPS in order for the burden of HAIs to be better estimated.
In 2011, the European Centre for Disease Prevention and Control (ECDC) initiated the first European point prevalence survey (PPS) of HAIs and antimicrobial use in acute care hospitals  involving 29 European Union (EU)/ European Economic Area (EEA) countries and Croatia. The objective was to estimate the total burden (prevalence) of HAIs and antimicrobial use in European acute care hospitals.
A pilot validation study was undertaken in the first phase of this PPS in 2011 with two major objectives: (i) to test the sensitivity and specificity of reporting HAIs and antimicrobial use and the level of agreement between primary and validation data collectors, whereby this constituted the validation component of the study; (ii) to test the inter-rater reliability (IRR) of hospital data collectors across Europe.
This paper focuses on the aggregated results for several EU Member States of this pilot validation study. Ten EU Member States took part in the validation component (Bulgaria, Finland, Germany, Hungary, Italy, Latvia, Lithuania, Poland, Spain, United Kingdom) and eight of these countries in the IRR component (Bulgaria, Finland, Germany, Italy, Lithuania, Poland, Spain, United Kingdom).
The sample size for the pilot study was calculated to produce validation results overall for the European PPS rather than at individual country specific level. A pilot ECDC PPS had indicated a prevalence of 7.1% and an average hospital size of 300 beds . The sample size for the pilot validation study was calculated at approximately 2,000 patients for an estimated sensitivity of reporting HAIs of 80% with a precision of +/- 5% and a prevalence of 7% [3,4]. These 2,000 patients were specified by the ECDC validation pilot protocol as approximately 200 patients per country within the 10 participating countries, sampled in at least two hospitals per country.
Two approaches, including a validation method by an external validation team (method 1) and an on-site assessment of the IRR of different hospital PPS data collectors (method 2), were taken in order to address the objectives. The methods are summarised here and a full description is available in the ECDC pilot validation protocol .
A standard ECDC protocol was used by all countries . Each country collected data on 100 patient records from each of two hospitals. The hospitals and the patient records were chosen by the national coordinators from each country and not randomly allocated at a country level. A number of approaches were taken including retrospective, simultaneous same day, simultaneous same time, blind and unblind data collection. The approach undertaken by each country was purposively selected dependent on timing of the primary PPS and availability of resources. Countries also had an option of oversampling within the protocol, whereby the number of HAIs in the validation sample was increased on purpose to increase the precision of the specificity estimation, by selecting wards with higher prevalence (e.g. intensive care units) in blind validation or by including all HAI cases detected in the primary PPS in unblind validation.
The validation findings were considered the ‘gold standard’ (true positives and true negatives) as the validation team consisted of at least one trained expert from (and/or acting on behalf of) the national/regional PPS coordinating centre (external to the validated hospital), using the ECDC-PPS protocol and codebook  and accompanied by a hospital staff member for the purposes of access and orientation.
Identical data to the primary data collector were collected by the validator using one or more of the approaches outlined above. Patient notes, nursing notes, hospital information systems and clinical ward personal were the data sources used.
From the validation dataset, the positive predictive value (PPV) was calculated as the percentage of patients with true HAIs (or patients receiving antimicrobials as appropriate) among all positive patients in the primary dataset, and the negative predictive value (NPV) as the percentage of true negative cases among all patients identified as negative in the primary sample. The results of the validation were applied to the aggregated primary data by multiplying the number of all positive cases in the primary sample by the PPV to obtain an approximation of the number of true positives to account for potential differences in prevalence due to oversampling. The same procedure was performed for negative cases with the NPV. This allowed determination of the sensitivity and specificity for the primary sample [4,6]. 95% confidence intervals (CI) were calculated using a continuity-corrected version of the Wilson’s score method. They were evaluated as ‘worst case’ instances using a combination method. The effects of omitting these adjustments, in most cases did not result in major differences to the results presented here.
On site assessment of inter-rater reliability
Five HAI-positive and 10 HAI-negative patient records were selected from a single setting, i.e. intensive care unit (ICU), where the prevalence of infection was the highest, or, if access to the ICU was restricted, in a limited number of other wards with expected high HAI prevalence, such as high dependency units.
Between two and five hospital primary PPS data collectors gathered data at an agreed time in the selected ward/setting in turn with the national contact point (validator). A procedure was followed to minimise any potential bias inclusive of the other rater(s) waiting in another room or at a distance where the reproducibility process could not be heard (e.g. use music in the waiting room/area). Data items were collected as detailed in method 1 and agreement between the data collections was analysed using kappa (κ) statistics (0.81–1.00 is very good, 0.61–0.80 is good, 0.41–0.60 is moderate, 0.21–0.40 is fair/marginal, < 0.2 is poor; negative values are possible and also denote ‘poor’ agreement [7-9]. κ statistics were also reported for certain variables of the validation approach (method 1), as it can be argued that the external validation team does not truly represent a gold standard for HAIs and variables such as the McCabe score.
The primary data set that originated from the 20 hospitals in the 10 participating countries comprised 3,958 patient records. Among these, the prevalence of HAIs was 9% (367 patients) and the prevalence of antimicrobial use was 38% (1,504 patients). Validation data were collected from October to December 2011. Of the 3,958 primary patient records, a total of 1,950 were selected for validation in accordance with the calculated study sample size. Of those, 1,912 were matched to the primary dataset, since it was not possible to link all patient records due to errors in data entry or missing data. The reported prevalence of HAIs in the matched validation dataset was 12% (233 patients) and the prevalence of antimicrobial use was 46% (878 patients). Due to oversampling in the validation dataset, the prevalence of HAIs in this dataset was significantly higher than in the ‘primary’ set, Χ2(1) = 7.7, p = 0.005. The prevalence of antimicrobial use was also higher in the validation dataset, Χ2(1) = 27.5, p < 0.001. Four of the ten countries included oversampling.
There was 97% agreement (κ = 0.81) between primary records and the validation records for the presence of an HAI (Table 1). The level of agreement was very good across all the most important types of HAIs, ranging from κ = 0.83 for bloodstream infections to κ =1.00 for lower respiratory tract infections (Table 2). Specificity of reporting HAIs was 98% (95% CI: 98–99%) with sensitivity comparatively lower at 83% (95% CI: 79–87%) (Table 1). The sensitivity by type of HAI ranged from 83% for bloodstream infections to 100% for lower respiratory tract infections, with specificity values higher than 99% for all types of HAI (Table 2).
Table 1. Validation of the point prevalence survey for assessing healthcare-associated infections, 10 European Union Member States, 2011
Table 2. Validation of the point prevalence survey for healthcare-associated infections (HAIs), by type of HAI, 10 European Union Member States, 2011 (n=1,912 patient records)a
Very good results (96% agreement, k=0.91) were achieved with respect to the recording of overall antimicrobial use (Table 3). Sensitivity and specificity were both very high at 94% (95% CI: 92.9–95.3%) and 97% (95% CI: 96.1–97.5%) respectively. Validation of the route of antimicrobial administration oral, parenteral showed that oral antimicrobials were frequently reported as parenteral, resulting in a lower specificity for the parental route and a lower sensitivity for the oral route (Table 4).
Table 3. Validation of the point prevalence survey for assessing antimicrobial use, 10 European Union Member States, 2011
Table 4. Validation of the point prevalence survey for antimicrobial use, by administration route, 10 European Union Member States, 2011 (n=1,912 patient records)a
At the individual variable level, some variation was noted. Agreement on basic demographic variables was very good: age (κ=0.95), sex (κ=0.93), specialty of physician (κ=0.87). A high level of agreement was also found with respect to the presence of invasive devices, although specificity for the presence of peripheral vascular catheters (93%; 95% CI: 91–95%) was noted to be significantly lower than that of central venous catheters (99%; 95% CI: 98–99%). Variables which required more interpretation such as McCabe score had a moderate score (κ = 0.57).
Eight of ten countries participated in the IRR component of the pilot study with a total of 44 raters across all the participating hospitals, rating 195 patient records. An analysis of IRR by selected variables was undertaken on the dataset. Variables were selected on the basis of their importance and the frequency of reporting in the dataset. Analysis of IRR overall showed a very good level of agreement (κ = 0.92) for both the presence of HAIs (96%) and antimicrobial use (97%) (Table 5). There was very good IRR (κ > 0.8) for most of the PPS variables (with the exception of HAI origin) (κ =0.31).
Table 5. Inter-rater reliability results by selected variables, pilot validation study of a point prevalence survey on healthcare-associated infections and antimicrobial use, eight European Union Member States, 2011 (n=44 raters and 195 patient records)
Studies on validation of national HAI surveillance are rarely published, and when they are, a variety of approaches are described, according to a recent review in the United States (US) . In that review, of those that included either a validation or an IRR study, the results were varied, underscoring the need for PPS to include validation studies to add confidence to interpretation of the data. This study is the first multi-country validation study undertaken on the first ECDC PPS dataset. Based on the findings, a revised protocol for data validation of the PPS of HAIs and antimicrobial use in European acute care hospitals was made available in 2012. This protocol might be helpful to other countries considering similar studies in the future .
The validation component identified an overall sensitivity of 83% (95% CI: 79–87%) and specificity of 98% (95% CI: 98–99%) for the presence of HAI. The level of agreement between the primary analysis and the validation assessment was very good across all the types of HAI. Previous studies indicated some variation at the level of individual types of HAI. In these studies respiratory tract infections had lower sensitivity, specificity and inter-rater reliability than other types of HAI [11-13]. However, the results of this pilot study indicated a high level of specificity and a high level of agreement for these types of HAIs. It is likely that the training given to support ECDC PPS has had an impact on the good validity results in our pilot study, however, this is difficult to assess and, to our knowledge, no study has been published to date assessing the effect of training on data validity. Moreover, the relatively good sensitivity and specificity results found in our pilot study may have been influenced by the ‘experimental’ conditions (e.g. selection of two hospitals per country willing to participate), which may have resulted in higher sensitivity and specificity than would have been found in validation across a non-selected group of hospitals. Indeed, in four national validation surveys carried out in 2012 during the second phase of the ECDC PPS, the average sensitivity of reporting HAIs was 71.9%, considerably lower in our pilot study . The sensitivity (83%) in our study indicates potential underreporting of HAIs in the ECDC PPS. This underreporting of HAIs may have resulted from difficulties with application of definitions or availability of patient record information at the time of data collection. To the authors’ knowledge, this was the first study which formally validated the reporting of antimicrobial use within a PPS study. Antimicrobial use had a high sensitivity of 94% (95% CI: 93–95%) and specificity of 97% (95% CI: 96–98%) with a very good level of agreement. Validation of the route of antimicrobial showed that oral antimicrobials were frequently reported as parental antimicrobials.
Other variables within the validation dataset were well recorded. A complex patient records review study by Yawn and Wollan (2005)  found that demographic data that required copying explicit information (e.g. sex, birth date), ‘free-text’ data that required identifying and copying (e.g. chief complaints and diagnoses), and data that required abstractor judgment in determining what to record (e.g. whether heart disease was considered) differed in terms of rates of agreement. In our study, agreement between the validation and the primary data collectors on more basic demographic variables ranged from moderate for the McCabe score to very good for sex and age. This finding was in line with the scarce literature published to date [11-12,15-19], wherein basic demographic variables such as age and sex tend to have very good levels of agreement compared to those variables where interpretation is required, such as the McCabe score or other markers of co-morbidity. The variables requiring abstractor judgment in this pilot validation study usually involved verification with a clinician present on the ward, which may account for the higher than expected validity.
The IRR component showed that the in-hospital IRR reliability was very good, for HAIs and antimicrobial use. This level of agreement was also found for other variables with the notable exception of HAI origin, which had a fair/marginal κappa. No studies that have looked formally at IRR in more than one country were identified in the literature. One study  did examine the difference between teams of data collectors within an Indonesian PPS and indicated that inter-observer variation differed significantly between the teams of data collectors in terms of completeness of data, and most importantly in the number of detected HAIs. Differences of note in this previous study were with respect to surgical site infection, urinary tract infection and septicaemia (p = 0.01) and the reported agreement (κ) did not exceed 0.59 for any type of HAI . Their evaluation indicated that ascertainment was affected by underreporting in patient records, and the retrospective nature of data collection for validation purposes.
While the issue of record keeping with respect to device use, infection criteria and antimicrobial use, had been identified as a potential limitation at the outset of this study, the outcomes in the validation aspects of this pilot study were better than expected in this regard. The IRR results overall in this pilot study were better than those published previously in the literature with respect to presence of HAIs and types of HAI.
As with all observation studies of this nature there are a number of potential biases which are acknowledged herein. The first of these is the potential for selection bias as participating countries chose the hospitals and patient records; these were not randomly allocated at a country level. Observer bias potential is also acknowledged as not all the validators were blinded to the primary results although the high levels of IRR indicate minimal risk of this.
In summary, this pilot study suggested that the ECDC PPS dataset of HAIs and antimicrobial use was valid and reliable. Basic demographic data and antimicrobial use data had very good levels of validity and reliability and may not need to be routinely collected in future validation studies. The high specificity and IRR are an indication that the training on the case definitions organised during preparation of the ECDC PPS was effective. The lower sensitivity findings show the potential for underreporting of HAIs in the ECDC PPS and highlight the importance of validation studies for future surveillance activities in order for the burden of HAI to be better estimated.
National Participants in the ECDC pilot validation study
Bulgaria: Rossitza Vatcheva-Dobrevska, Ivan Ivanov; Finland: Tommi Kärki; Germany: Petra Gastmeier; Hungary: Karolina Böröcz, Ágnes Hajdu; Italy: Silvio Brusaferro, Maria Luisa Moro, Luca Arnoldo; Latvia: Elina Dimina, Uga Dumpis; Lithuania: Jolanta Ašembergienė, Rolanda Valintėlienė; Poland: Aleksander Deptula, Waleria Hryniewicz; Spain: Jose Angel Rodrigo Pendas, Josep Rafart Vaqué.
Conflict of interest
JR, LP, JG, SC, SHo, BCoo, WM, GH, OL, BCog, SHa, and CS all contributed to the design of the study and reviewed and commented on the manuscript. In addition JR led the study, LP managed and coordinated the study, JG analysed the data and JR and LP interpreted the results and wrote the manuscript. The following people conducted the study in their respective countries RVD & II (Bulgaria); TK & OL (Finland); PG & SHa (Germany); KB & ÁH (Hungary); SB, MLM & LA (Italy); ED & UD (Latvia); JA & RV (Lithuania); AD & WH (Poland); JARP & JRV (Spain); SHo (United Kingdom).
- European Centre for Disease Prevention and Control (ECDC). Point prevalence survey of healthcare-associated infections and antimicrobial use in European acute care hospitals, 2011–2012. Stockholm: ECDC; 2013. Available from: http://www.ecdc.europa.eu/en/publications/Publications/healthcare-associated-infections-antimicrobial-use-PPS.pdf
- Zarb P, Coignard B, Griskeviciene J, Muller A, Vankerckhoven V, Weist K, et al. The European Centre for Disease Prevention and Control (ECDC) pilot point prevalence survey of healthcare-associated infections and antimicrobial use. Euro Surveill. 2012;17(46):pii=2031. PMID:23171822
- European Centre for Disease Prevention and Control (ECDC). Pilot validation study of the ECDC Point Prevalence Survey of healthcare-associated Infections and Antimicrobial Use in European Acute Care Hospitals. OG/23/06/2011-PROC/2011/060. Stockholm: ECDC; 2011. Available from: http://www.ecdc.europa.eu/en/aboutus/calls/_layouts/forms/Call_DispForm.aspx?List=02511b7b-3a16-4c4b-9304-54cfc08a1647&ID=586
- European Centre for Disease Prevention and Control (ECDC). PPS Validation Protocol version 2.1 Stockholm: ECDC; 2014. Available from: http://www.ecdc.europa.eu/en/publications/Publications/HAI-PPS-Validation-protocol-Sept-2014.pdf
- European Centre for Disease Prevention and Control (ECDC). Point prevalence survey of healthcare-associated Infections and antimicrobial use in European acute care hospitals: protocol version 4.3. Stockholm: ECDC, 2012 Available from: http://www.ecdc.europa.eu/en/publications/publications/0512-ted-pps-hai-antimicrobial-use-protocol.pdf
- Huotari K, Agthe N, Lyytikäinen O. Validation of surgical site infection surveillance in orthopedic procedures. Am J Infect Control. 2007;35(4):216-21. http://dx.doi.org/10.1016/j.ajic.2006.01.009 PMID:17482992
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20(1)37-46. http://dx.doi.org/10.1177/001316446002000104
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74. http://dx.doi.org/10.2307/2529310 PMID:843571
- Altman DG. Practical statistics for medical research. London: Chapman and Hall;1991.
- Llata E, Gaynes RP, Fridkin S. Measuring the scope and magnitude of hospital-associated infection in the United States: the value of prevalence surveys. Clin Infect Dis. 2009;48(10):1434-40. http://dx.doi.org/10.1086/598328 PMID:19351269
- Gastmeier P, Kampf G, Wischnewski N, Hauer T, Schulgen G, Schumacher M, et al. Prevalence of nosocomial infections in representative German hospitals. J Hosp Infect. 1998;38(1):37-49. http://dx.doi.org/10.1016/S0195-6701(98)90173-6 PMID:9513067
- Smyth E, McIlvenny G. Hospital-acquired infection surveillance: removing the manual data-entry bottleneck. Br J Healthc Comput Inf Manage. 1997;14(10):31-4.
- Gastmeier P, Kampf G, Hauer T, Schlingmann J, Schumacher M, Daschner F,et al. Experience with two validation methods in a prevalence survey on nosocomial infections. Infect Control Hosp Epidemiol. 1998;19(9):668-73. http://dx.doi.org/10.2307/30141532 PMID:9778166
- Yawn BP, Wollan P. Interrater reliability: completing the methods description in medical records review studies. Am J Epidemiol. 2005;161(10):974-7. http://dx.doi.org/10.1093/aje/kwi122 PMID:15870162
- Zuschneid I, Geffers C, Sohr D, Kohlhase C, Schumacher M, Ruden H, et al. Validation of surveillance in the intensive care unit component of the German nosocomial infections surveillance system. Infect Control Hosp Epidemiol. 2007;28(4):496-9. http://dx.doi.org/10.1086/512631 PMID:17385161
- Fabry J, Morales I, Metzger MH, Russell I, Gastmeier P. Quality of information: a European challenge. J Hosp Infect. 2007;65 Suppl 2:155-8. http://dx.doi.org/10.1016/S0195-6701(07)60035-8 PMID:17540262
- Duerink DO, Roeshadi D, Wahjono H, Lestari ES, Hadi U, Wille JC, et al. Surveillance of healthcare-associated infections in Indonesian hospitals. J Hosp Infect. 2006;62(2):219-29. http://dx.doi.org/10.1016/j.jhin.2005.08.004 PMID:16307823
- Stewart S, Coubrough S, Reilly J, Allardice G. P17.36 Assessing data quality: Lessons learned from the Scottish National HAI Prevalence Survey. J Hosp Infect. 2006;64 Suppl 1:S92. http://dx.doi.org/10.1016/S0195-6701(06)60307-1
- Hajdu A, Eriksen HM, Sorknes NK, Hauge SH, Loewer HL, Iversen BG, et al. Evaluation of the national surveillance system for point-prevalence of healthcare-associated infections in hospitals and in long-term care facilities for elderly in Norway, 2002-2008. BMC Public Health. 2011;11(1):923. http://dx.doi.org/10.1186/1471-2458-11-923 PMID:22165849