Electronically assisted surveillance systems of healthcare-associated infections: a systematic review

Background Surveillance of healthcare-associated infections (HAI) is the basis of each infection control programme and, in case of acute care hospitals, should ideally include all hospital wards, medical specialties as well as all types of HAI. Traditional surveillance is labour intensive and electronically assisted surveillance systems (EASS) hold the promise to increase efficiency. Objectives To give insight in the performance characteristics of different approaches to EASS and the quality of the studies designed to evaluate them. Methods In this systematic review, online databases were searched and studies that compared an EASS with a traditional surveillance method were included. Two different indicators were extracted from each study, one regarding the quality of design (including reporting efficiency) and one based on the performance (e.g. specificity and sensitivity) of the EASS presented. Results A total of 78 studies were included. The majority of EASS (n = 72) consisted of an algorithm-based selection step followed by confirmatory assessment. The algorithms used different sets of variables. Only a minority (n = 7) of EASS were hospital-wide and designed to detect all types of HAI. Sensitivity of EASS was generally high (> 0.8), but specificity varied (0.37–1). Less than 20% (n = 14) of the studies presented data on the efficiency gains achieved. Conclusions Electronically assisted surveillance of HAI has yet to reach a mature stage and to be used routinely in healthcare settings. We recommend that future studies on the development and implementation of EASS of HAI focus on thorough validation, reproducibility, standardised datasets and detailed information on efficiency.


Introduction
Surveillance of healthcare-associated infections (HAI) entails the systematic collection of data on the presence of HAI, analysis and transformation of the data into information and sharing this information with those who can take action to prevent HAI. Surveillance with feedback is a key/core component for effective infection prevention and control in the strategies of both the World Health Organization (WHO) and the European Centre for Disease Prevention and Control (ECDC) [1,2]. Surveillance and epidemiology are the first criteria included in the minimum standard of practice of an infection preventionist (Association for Professionals in Infection Control and Epidemiology (APIC) 2016 professional and practice standards). They are also a core task according to the Dutch society of Hygiene and Prevention in Healthcare (VHIG 2014, professional profile). The Health and Social Care Act 2008 code of practice of the prevention and control of infections and related guidance (2015, United Kingdom) likewise states that there should be evidence of local surveillance and of the use of comparative data, where available, to monitor infection rates in healthcare settings. In 2005, legislation on mandatory reporting of HAI had already been enacted in 35 states of the United States. In 2008, the Centers for Medicare and Medicaid Services stopped additional payment to hospitals to cover for the costs of treating mediastinitis after coronary artery bypass graft surgery, because this HAI was deemed to be preventable by HAI surveillance and control. Likewise, surgical site infection (SSI) after specific orthopaedic procedures and bariatric surgery were not covered anymore. This change reflected the importance of HAI surveillance and prevention. Hospital accrediting authorities, including Joint Commission International, also recognise the importance of surveillance activities.
Surveillance in its traditional format, in which every patient's record is assessed on the presence of HAI, is labour intensive and can take up to 756 manhours for a point prevalence survey or an estimated 1.5 fulltime equivalent (FTE) per 10,000 admissions [3,4]. Surveillance of HAI should ideally include all hospital wards, medical specialties as well as all types of HAI, such that fully informed, evidence-based decisions can be made to prioritise and structurally address the relevant infection issues of the particular healthcare setting. However, in the face of limited resources, available labour intensive manual surveillance systems have driven most healthcare centres to apply so-called targeted forms of surveillance that include only high risk wards and/or few types of medical procedures and/or only a few types of HAI.
Alternatively, efforts have been made to significantly improve the efficiency (i.e. reducing the time spent on surveillance while obtaining the same results) of HAI surveillance by applying information technologies to query data routinely available in hospital electronic databases. With the ongoing application of new information technologies in healthcare, the types and sizes of these electronic databases have been increasing over the last decades. It is, however, unclear what level of efficiency and accuracy (i.e. proportion of individuals truly positive and negative for an HAI) has been attained so far by applying these newer electronic information sources and technologies. Increasing the efficiency of surveillance can be achieved by introducing a computerised algorithm-based selection of highrisk patients followed by confirmatory assessment of the selected cases by the infection control practitioner (ICP) (semi-automated electronically assisted surveillance system (EASS)). Alternatively, surveillance can be based on the outcome of a computerised algorithm alone, without confirmation of individual cases by the ICP (fully automated EASS).
With this systematic review, we aim to give insight in the current status of EASS by assessing their performance (sensitivity and specificity) compared with traditional surveillance, the variables used in the electronic algorithms and the quality of the studies (including efficiency) that presented and evaluated them.

Inclusion of articles
In this systematic review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed [5]. The following online databases were searched on 10 January 2018: Embase, Medline Ovid, Cochrane, Web of Science, Scopus, CINAHL (EBSCOhost) and Google Scholar. The Embase search was the basis for the search method and then translated for use in other databases. All publications, except grey literature, in the past up to 10 January 2018 were included. The full description of the queries per database is provided in the supplementary material. The process of identification, screening and inclusion of articles for full text synthesis is consistent with the method described by Bramer et al. [6][7][8]. In brief, this method consists of a query in online databases, review of all the abstracts by two reviewers independently to screen for articles that probably meet the inclusion criteria (eligible) followed by full article review to decide whether an article is included based on the inclusion and exclusion criteria. Included were studies which systematically evaluated the performance of an electronically assisted or fully automated surveillance method against the gold standard (expert opinion of the ICPs based on the United States Centers for Disease Control and Prevention-based criteria for HAI [9]), and reported at least one performance metric (e.g. sensitivity, specificity). Exclusion criteria were potential incorporation bias, non-English language studies and poster or conference abstracts.

Data collection process
Using a standardised format, we abstracted the following reported information from each study: (i) year of publication, country and setting; (ii) the study type, population characteristics and sample size; (iii) the type of HAI (bloodstream infections (BSIs), lower respiratory tract infections (LRTIs), SSIs or urinary tract infections (UTIs)) targeted for surveillance, likewise the medical specialties or wards that were participating in the surveillance effort; (iv) the sensitivity, specificity, positive and negative predictive value, accuracy of the EASS; and (v) all variables that were used in their algorithms. Missing information was not checked with the authors.

Algorithm categories
For each study, we assigned the respective algorithm used to one of the categories based on the variables included as presented in Table 1.

Overall quality and performance score
Two different scores were calculated based on the data extracted from each study, one regarding the reported design of the study and one regarding the performance of the EASS presented. First, to assess the quality of the design we scored six indicators that represent different aspects of the design of the study. As an example, for indicator 1, the rationale was as follows: an EASS should be evaluated in a set of patients that is independent of the set used to develop the automated method, as recommended by Govindan et al. in 2010 [10]. Thus when the study population consisted of a validation cohort and a separate development cohort this was considered better (5 points) than when the study population consisted of a development cohort only (1 point). Subsequently we calculated an overall quality score by adding up the points assigned for each of the six quality indicators presented in Table 2.
Secondly, for each study we calculated an overall performance score of the EASS presented in the study by multiplying its published sensitivity and specificity. This overall performance score is primarily based on the sensitivity and specificity to detect all types of HAI.
If not all types of HAI were included we calculated an overall performance score based on the means of the sensitivities and specificities specified for the types of HAI that were included in the study.

Ethical statement
Ethical approval was not applicable for this review of published literature.

Results
The online databases search identified 2,410 records after de-duplication, of which 78 studies [3,4, were included in this systematic review ( Figure 1).

Timeline and geographical distribution
Of the 78 identified studies, 30 were from Europe, six from Asia, one from South America and the remaining 41 from North America. The first articles about EASS of HAI originated in North America and were published in the 1980s and 1990s. Thereafter, their number increased gradually, involving European centres since the turn of the century, and, more recently centres in Table 2 Quality of study designs: indicators used to evaluate the level of quality of study designs investigating an electronically assisted surveillance system

Indicator 1 -validation and test cohort
The study population consisted of a validation cohort and a separate development cohort 5 The study population consisted of a validation cohort 3 The study population consisted of a development cohort only 1 Indicator 2 -reported context of study The observed prevalence or incidence of HAI in the healthcare institute was presented in the article 5 Prevalence or incidence of HAI was not presented in the article 0 other parts of the world; a peak in the annual number of published articles was reached in 2014 ( Figure 2).
In the earliest identified study, in 1986, Evans et al. [30] presented the results of HAI surveillance through their Health Evaluation through Logical Processing (HELP) system at the Latter Day Saints hospital, Salt Lake City, Utah. These pioneers showed that more HAI were found with this EASS, while its execution required only 35% of the time. In follow-up studies HELP was used to evaluate algorithms to predict infections even before onset of symptoms [29]. It was not until the year 2003 that the first European study was published describing the performance of an EASS; this EASS was implemented in 1998 in a university affiliated teaching hospital in France [11].

Quality
As for indicator 1 of the overall quality score ( Table  2), 18 of the 78 (23%) articles included in this review used a development and separate validation cohort. The majority of articles (53 of 78) clearly presented the point prevalence or incidence of the HAI in the population they studied (quality indicator 2, Table 2). The EASS was hospital-wide (maximum points for quality indicator 3, Table 2) and included all types of HAI (maximum points for quality indicator 4, Table 2 with at least the 'big four' types of HAI, i.e. BSIs, LRTIs, UTIs and SSIs) in nine of the 78 studies. Just under 40% (31/78) of the studies presented performance characteristics other than only sensitivity and specificity (maximum points for indicator 5). Less than 20% (14/78) of the studies presented the actual time reduction that was achieved by introducing an EASS for HAI (maximum points for  Articles excluded after two independent reviewers screened all the abstracts and discrepancies were resolved by consensus (n = 2,147) PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
indicator 6, Table 2). None of the articles attained the maximum score (all six indicators 'green or maximum points appointed') and, interestingly, the overall quality score did not seem to improve over the years (supplementary material Table 1).

Performance of electronically assisted surveillance systems per type of healthcareassociated infection
EASS were most extensively developed and applied in intensive care unit (ICU) settings (44 of the 78 articles included in this review); 20 articles describe an EASS that was developed exclusively for the ICU and targeted the ICU's two most common HAI (ventilator associated pneumonia (VAP) and central line associated BSIs (CLABSI)) [

Algorithms and parameters used in electronically assisted surveillance system
In their review 'Surveillance and use of computers in hospital infection control' Wenzel and Streed [87] presented a timeline of the evolution of surveillance of infectious diseases including healthcare-associated ones, from 1532 to 1989, pointing out key figures and events in its history. As they elaborated on the different elements of surveillance, the authors eventually addressed how the application of computers could help infection control with their surveillance activities and add to their accuracy and probably to efficiency as well. In 2002 Peterson and Brossette [88] addressed the importance of clinical microbiological laboratories and the clinical microbiologist in HAI control. They stated that laboratory-based surveillance has the advantage of measuring hospital-wide occurrences from a single, central data collection point.  [90] performed a systematic review in which they categorised the EASS in multisource (including microbiological data) (n = 37), multisource (excluding microbiological data) (n = 4) and single source systems (n = 3). Of these 44 articles included for review, 21 were validation studies. There was no difference in the performance of multisource systems excluding or including microbiology. Of note, the single source validation studies, used natural language processing techniques to extract information from the radiology reports and electronic health record to detect LRTI and BSI respectively [33,91]. Based on the results of their review in 2014 de Bruin et al. [92] concluded that driven by the increased availability of electronic patient data, EASS tend to use more data sources. This is making systems more sensitive yet less specific, but also allows systems to be tailored to the needs of healthcare institutes' surveillance programmes. The findings of Cato's review [93], in 2015, suggest that the majority of EASS for HAI surveillance are using standard definitions of HAI, but the lack of standardised use of data formats, denominator, and external validation in these systems reduces the reliability of their findings.

Figure 2
Published studies on electronically assisted surveillance of healthcare-associated infections by year of publication and by region of the world (n = 78)

Microbiological data
The most prevalent variable included in EASS is the microbiological examination (present in at least one of the algorithms presented in 61 of the 78 articles). It is a crucial variable in algorithms for surveillance of BSI. Good sensitivities have been achieved using the results of microbiological examinations as the sole variable ( Figure 3). In other types of HAI (UTI, SSI and LRTI) the sensitivity of algorithms that use microbiological examination alone is lower. As a consequence of the definitions used as 'gold standard', the microbiological examination variable is often combined with clinical characteristics (Figure 3). Although the microbiology variable seems straightforward, it can mean several things, i.e. the presence of bacteriological examinations, which use either culture, serological or molecular techniques or all of these. The presence and quality of the microbiological examination highly depends on the availability of the service -often lacking in low income settings -, and on the adherence to diagnostic protocols. Finally, a negative microbiological examination does not unequivocally exclude the presence an infectious disease.

Chemistry
Clinical chemistry evaluation of biomarkers for infections are routinely performed in most clinical settings.

Antibiotics
Following microbiological data, antibiotic prescriptions data are the second most used variable in EASS algorithms (38 of 78 studies) and, when not combined with microbiological data, most often found in combination with ICD coding [12,16,36,56].

International Classification of Diseases
The sensitivity of ICD-coding (9th or 10th edition) is generally low, in five of the seven studies that use this as the sole variable of the algorithm [11,39,54,70,77,78,83]. The two studies with sensitivities of 0.86 and 0.97 focused on SSI after coronary artery bypass graft surgery and total joint replacement [39,54]. ICD-coding has not been studied for all types of HAI surveillance nor for LRTI surveillance (Figure 2). Electronically assisted surveillance system per type of healthcare-associated infection

Bloodstream infections (including central line associated)
Twenty-three publications ( Table 3) described the performance of an EASS to detect BSI. Sensitivity ranged from 0.32 to 1.0, specificity from 0.37 to 1.
In 2017, Gubbels et al. presented the results of an automated nationwide survey, using the Danish Microbiology Database and the Danish National Patient Registry. National trends showed an increase in hospital-acquired bacteraemia (HAIBA) cases between 2010 and 2014. Although comparison of their HAIBA algorithm with the results of point prevalence surveys showed a sensitivity of 0.36 they concluded that given the many advantages of automated surveillance, the HAIBA algorithm allows monitoring of HAIBA across the healthcare system, supports prioritising preventive measures, and holds promise for evaluating interventions [80]. Another approach to automated surveillance of BSI is the reuse of the ICD-coding system. Unfortunately the performance to detect BSI in the paediatric ICU was shown to be very poor (sensitivity of 0.32), as Bond et al. showed [78]. In contrast, one adult ICU-tailored algorithm showed excellent performance characteristics (sensitivity and specificity of 1) [28].

Urinary tract infections (including catheter-associated urinary tract infections)
UTI is one of the 'big four' and is third, behind SSI and LRTI, in HAI prevalence in the Netherlands (2017) [94].
Internationally 18 publications (   sensitivity [79]. ICD-10 codes for UTI surveillance were shown to perform poorly (sensitivity 0.02) and were only studied in one study [83]. On the other hand a urine culture as the only parameter proved to perform well [38,50].

Surgical site infections
Not surprisingly, the majority of publications, 29 in total, presented the performance of an EASS to detect SSI (Table 6). Different fields of surgery are targeted, from neurosurgery to total hip replacements and different algorithms with sensitivities ranging from 0.02 to 1.0 and specificities ranging from 0.59 to 1.0 have been explored. Administrative data (ICD-9 admission and discharge coding and claims data) generally performed poorly when used as the only variable for case finding [37,46,54,70,77].
Interestingly a Michigan study by Baker et al. showed a 0.89 sensitivity for ICD-9 codes when these were solely used to determine post-caesarean section SSIs. Combining ICD-9 with specific post-operative antibiotics increased the specificity of the case finding algorithm to 0.95 and resulted in a ca 90% workload reduction [12]. Fifteen years later in Denmark, comparable performance characteristics were found for in-hospital and post-discharge surveillance of postcaesarean section SSIs using ICD-10 discharge codes in combination with relevant antibiotics and culture data [49].
Adding microbiology and radiographic data (culture, magnetic resonance imaging ordered) as variables of case finding algorithms resulted in a positive predictive value of 0.97, although at the cost of sensitivity (0.48) [19]. A text mining algorithm as presented by Michelson et al. found all of the 22 SSIs detected by traditional hospital-based surveillance, along with an additional 37 SSIs not detected by traditional surveillance [53].

Quality and performance
The quality of the 78 studies included in this review varied from an overall quality score of 5 to 26. Likewise, the overall performance score of the EASS developed and used for HAI surveillance varied widely among the studies, with overall performance scores ranging from a low 0.2 to the maximum of 1.0. When quality of study design was plotted against the level of performance of the EASS there seemed to be a, albeit weak, negative correlation, suggesting that the best designed studies tended to yield lower performance scores or vice versa, that the studies reporting very high performance scores of their EASS were generally less well designed ( Figure 4).
Based on this view of the overall quality and performance score we selected the 10 best studies (labelled in the figure) that showed the highest overall quality scores and also yielded a high overall performance score ≥ 0.85 ( Figure 4 upper right quadrant). They, together, constitute a reference or can be considered a benchmark, for EASS development. Interestingly all these studies were reported only recently (2013-2016), indicating that some progress has been made in designing and evaluating EASS (Table 7).
Interestingly, the majority of the 10 best EASS presented in Table 7 used a two-step procedure, first a selection step based on an automated algorithm, followed by step in which a confirmatory assessment of the selected cases by the ICP is performed. The monitoring of nosocomial infections (MONI) software on the ICU, the MONI-ICU system [27], however, achieved a high specificity of the EASS for all types of HAI based on an automated selection step only. This was achieved in a university hospital ICU by building a system consisting of fuzzy logic sets, which used a rich dataset including clinical, laboratory and nursing data from the patient data management systems in operation at the ICU wards on a daily basis. Du et al. [3] showed that based on a dataset including microbiology, clinical chemistry, antibiotics prescriptions and clinical characteristics, it is possible to perform continuous hospital-wide all type HAI surveillance with an EASS. Of note, they stated that their hospital has the most advanced computer information system in China, including an integrated hospital information system (IHIS). When documented in the electronic patient record clinical characteristics, like body temperature or  the presence of invasive devices, improve sensitivity and specificity of EASS. The data in Table 7 suggests that richer datasets lead to better performing EASS.
Remarkably only five articles presented the time reduction achieved by using an EASS.

Conclusions
Ideally, surveillance of HAIs comprises all departments, medical specialties and includes all types of HAI, because only then fully informed decisions can be made to prioritise and structurally address the relevant infection issues of the particular healthcare setting. In this review of the literature we found that very few EASS are 'all types, hospital-wide', although performance characteristics of these systems are generally good. The absence of traditional hospital-wide surveillance of HAI by the infection preventionist, which can be used as gold standard for validation can be one cause. On the other hand, the fact that most studies have been performed in an ICU setting implicates that hospital-wide electronic health records or data warehouses are still not ready to be used for decision support and surveillance.
Because of limited resources and the labour intensity of manual surveillance systems, efforts have been made to improve the efficiency of HAI surveillance by applying information technologies to query electronic datasets. However, we found that in literature, less than 20% of the studies present the actual time reduction that was achieved by introducing an EASS for HAI. Oher quality indicators, most importantly using different populations for development, testing and validation, are also lacking. Although sensitivity of EASS is generally high the specificity is variable, needing a confirmatory computer-assisted assessment by the infection preventionist and decision support. The parameters used in the algorithms studied vary considerably, do not have a standardised format nor are readily available in all hospital settings around the globe.
Although PRISMA guidelines were followed, this review has some limitations. Only English literature and no grey literature were considered for inclusion. The scoring system that was developed for and used in this study to assess the quality of the included studies' design was not prior validated.
In summary, thus far computer-assisted surveillance of HAI has not reached a mature stage, it is yet to be used routinely in most healthcare settings; we are learning, but we have not yet mastered the art. Although progress is being made towards a digital infrastructure for the learning health system [95], it is, in our opinion, not likely that EASS of HAI will be implemented globally within the next decade. A data-driven and decisionsupported healthcare system, including infection control surveillance, requires next generation electronic health records systems [96], clinical ownership and a good and close working relationship between infection control professionals and medical information specialists [97].
We recommend that future studies on the development and implementation of EASS of HAI focus on thorough validation, reproducibility in different hospital-settings of the algorithm and standardised datasets and present detailed information on efficiency. This information, together with the continuous focus on the importance of surveillance of HAI, is needed to convince healthcare providers, professionals and boards of directors, to invest in EASS, in the future of infection control.

Figure 4
Distribution of overall quality score of studies on electronically assisted surveillance systems vs the overall performance score of the electronically assisted surveillance system reported in these studies (n = 78) Overall quality score Overall performance score (sens x spec) Sens: sensitivity; spec: specificity.
In the upper right quadrant the 10 best studies are ranked from 1 to 10 (see Table 7 for more detail).

Table 7
Summary of the 10 best studies on electronically assisted surveillance systems included in the systematic review, ordered based on their overall quality score  The cells in this table are green when a finding is present and red when absent.