Announcements
Eurosurveillance remains in the updated list of the Directory of Open Access Journals (DOAJ). It was first added to the DOAJ on 9 September 2004. Eurosurveillance is also listed in the Securing a Hybrid Environment for Research Preservation and Access / Rights MEtadata for Open archiving (SHERPA/RoMEO) [2], a database which uses a colour‐coding scheme to classify publishers according to their self‐archiving policy and to show the copyright and open access self-archiving policies of academic journals. Eurosurveillance is listed there as a ‘green’ journal, which means that authors can archive pre-print (i.e. pre-refereeing), post-print (i.e. final draft post-refereeing) and archive the publisher's version/PDF.

ESCAIDE participants are invited to the fifth Eurosurveillance scientific seminar on 30 November 2016

Follow Eurosurveillance on Twitter: @Eurosurveillanc

Read our articles on Zika virus infection

Read our articles on mcr-1-mediated colistin resistance

Epidemiological investigation of MERS-CoV spread in a single hospital in South Korea, May to June 2015. Available from: http://www.eurosurveillance.org / ViewArticle.aspx?ArticleId=21169


In this issue


Home Eurosurveillance Edition  2016: Volume 21/ Issue 8 Article 3
Back to Table of Contents
Previous Download (pdf)
Next

Eurosurveillance, Volume 21, Issue 8, 25 February 2016
Review
Haller, Deindl, Cassini, Suetens, Zingg, Abu Sin, Velasco, Weiss, Ducomble, Sixtensson, Eckmanns, and Harder: Neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants: Umbrella review and evidence-based outcome tree

+ Author affiliations


Citation style for this article: Haller S, Deindl P, Cassini A, Suetens C, Zingg W, Abu Sin M, Velasco E, Weiss B, Ducomble T, Sixtensson M, Eckmanns T, Harder T. Neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants: Umbrella review and evidence-based outcome tree. Euro Surveill. 2016;21(8):pii=30143. DOI: http://dx.doi.org/10.2807/1560-7917.ES.2016.21.8.30143

Received:06 February 2015; Accepted:30 September 2015


Introduction

Sepsis is a major cause of death in neonates [1]. The majority of sepsis episodes (> 80%) occurs in preterm neonates [2]. Among very low birth weight infants (VLBW; < 1,500g), rates of sepsis range between 11% and 46% [3]. Sepsis in this high-risk population is mostly acquired during hospital stay with a late onset beyond 48–72 hours of life. Early onset sepsis, which becomes apparent within the first 48–72 hours of life is ‘nosocomial’ in the sense that it occurs in the hospital but should not be considered healthcare-associated because its origin is linked to childbirth and/or maternal-fetal transmission of pathogens [4,5].

Neonatal sepsis and systemic inflammatory response syndrome (SIRS) are associated with brain damage that results in disability, particularly among preterm and VLBW infants [6-9]. However, adverse neurological outcomes frequently occur in VLBW infants for reasons other than sepsis [10]. Therefore, the impact of healthcare-associated neonatal sepsis on adverse outcome is difficult to establish.

A European consortium, as part of a project initiated and funded by the European Centre for Disease Prevention and Control (ECDC), recently developed an incidence- and pathogen-based approach for estimating the burden of communicable diseases expressed in Disability Adjusted Life Years (DALYs) [11]. Relevant health outcomes of communicable diseases are represented by outcome trees which map the weighted progressions of diseases over time by ordering the conditional probabilities of associated health outcomes [11]. Outcome trees take into account probabilities and duration of health outcomes. The burden of healthcare-associated infections (HAIs) was not yet addressed by ECDC for two reasons: (i) Patients with HAIs differ from the general population in terms of comorbidities [7] and may be different regarding other factors as for example certain risk behaviour and social determinants; (ii) HAIs cannot be allocated to a specific pathogen, and thus the pathogen-based approach is not applicable.

Statements about the burden of communicable diseases must be based on evidence-based medicine principles as in other fields of medicine. Systematic reviews have become the gold standard of assessing the evidence in medicine but they are time consuming and expensive. Systematic reviews of systematic reviews (so-called ‘umbrella reviews’) offer a time-saving alternative to identify and exploit the current state of evidence in a field [12,13]. The aim of the study was to identify the relevant sources for the construction of an evidence-based outcome tree for neurological sequelae due to healthcare-associated neonatal sepsis in VLBW infants by systematically identifying and analysing existing systematic reviews that addressed neurodevelopmental impairment during infancy. From a clinical perspective, we aimed at investigating the extent by which sepsis in VLBW infants causes neurological impairments. Ultimately, the resulting outcome tree will constitute the basis for the estimation of the burden of hospital-acquired neonatal sepsis expressed in disability-adjusted life years (DALYs), within the general framework of the ECDC Burden of Communicable Diseases in Europe project (BCoDE).

Methods

We followed the approach described by Whitlock et al. [14] and Robinson et al. [15] to identify and exploit existing systematic reviews by re-analysing their data. Our study was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA) [16].

Identification of studies

In a first step, we performed a systematic review of systematic reviews (i.e. an umbrella review) on the association between neonatal sepsis and neurodevelopment in later life. To identify relevant systematic reviews we performed a systematic literature search using MEDLINE, EMBASE and Cochrane Central Register of Controlled Trials (CENTRAL), without language restrictions (Box 1). All systematic reviews published from 1 January 2000 until 25 September 2013 were eligible if meeting predefined inclusion criteria (see below).

Box 1

Search strategy for umbrella review on the association between neonatal sepsis and neurodevelopment in later life, date of search 25 September 2013


We used the following search strategy:

#1 “outcome”

#2 “follow-up”

#3 “sequel*”

#4 “consequence”

#5 “death”

#6 “cerebral palsy”

#7 “retinopathy”

#8 “necrotizing enterocolitis”

#9 “bronchopulmonary dysplasia”

#10 “neurodevelopmental impairment”

#11 “periventricular leukomalacia”

#12 “intraventricular haemorrhage”

#13 neonat*”

#14 “newborn”

#15 “sepsis”

#16 (#1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12) AND (#13 OR #14) AND #15

Filters: Publication date from 2000/01/01; Humans; “Medline” OR “systematic review” OR “meta-analysis” OR “intervention”

A further search for planned, ongoing and published systematic reviews was performed in the Prospective International Register of Systematic Reviews (PROSPERO).

In a second step, we performed an umbrella review on the positive predictive value of neurodevelopmental impairment for later cognitive function. To identify appropriate systematic reviews, we performed a systematic literature search (date of last search: 2 July 2014) using MEDLINE, EMBASE and the Cochrane Central Register of Controlled Trials (CENTRAL) (Box 2).

Box 2

Search strategy for umbrella review on the positive predictive value of neurodevelopmental impairment for later cognitive function, date of search 2 July 2014


The search strategy was the following:

#1 “predictive value”

#2 “Bayley scale*”

#3 “systematic review”

#4 “meta-analysis”

#5 #3 OR #4

#1 AND #2 AND #5 (no filters)

Electronic search was complemented by manually checking the reference lists of identified reviews and studies.

Study eligibility

In a first step, we searched for systematic reviews on the association between neonatal sepsis and neurodevelopment in later life. These systematic reviews had to capture primary studies which fulfilled the following inclusion criteria: (i) study population had to be neonates; (ii) the exposure had to be sepsis acquired in a healthcare setting; (iii) the comparator (or control) had to be participants without sepsis; (iv) the studies included had to be cohort studies or clinical trials; (v) the studies had to investigate at least one neurodevelopmental outcome during follow-up, and (vi) the studies had to be conducted in a healthcare setting within an upper-middle- or high-income country [17]. An expert panel discussed and agreed on the addressed outcomes to be relevant (for names and affiliations of the members of the expert panel, see Acknowledgements).

In a second step, the systematic reviews were searched for positive predictive values of neurodevelopmental impairment for later cognitive function. To be eligible, a systematic review had to fulfil the following inclusion criteria: (i) participants of the included studies had to be infants (aged 1–6 years) at first examination; (ii) neurodevelopmental impairment had to be measured by the Bayley Scales of infant development [18] at first examination; (iii) cognitive function (intelligence quotient) had to be measured by a standardised test (e.g. Wechsler Intelligence Scale [19]) at second (follow-up) examination; (iv) a positive predictive value or data allowing its calculation had to be reported in the review or in the studies analysed in the review.

Data extraction

Two independent reviewers (SH and TH) screened the systematic reviews, located the primary studies analysed in the reviews and extracted the following data: citation, study period, study design, demographics, sex, ethnicity, definition of sepsis, definition of outcome, length of follow-up, number of exposed and non-exposed with outcome, test used for first and second examination, positive predictive value and prevalence of condition. Discrepancies between the reviewers were solved by discussion until a consensus was reached.

Risk of bias assessment

The assessment of multiple systematic reviews (AMSTAR) tool was used to determine the methodological quality of the systematic reviews included [20]. Risk of bias in the included cohort studies was assessed using the Newcastle Ottawa Scale [21]. Following the suggestions of the Cochrane Collaboration, we assessed risk of bias separately for each outcome in each study [22]. For the studies reporting positive predictive values, the Scottish Intercollegiate Guidelines Network (SIGN) checklist for diagnostic accuracy studies was used to assess risk of bias [23]. The results of the risk of bias assessments were expressed in terms of ‘high risk of bias’, ‘low risk of bias’ and ‘unclear risk of bias’. All risk of bias assessments were conducted by two independent investigators (SH and TH).

Assessment of the quality of the body of evidence

We adapted the methodology of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group to assess the quality of the body of evidence [24,25]. The GRADE methodology was initially developed to assess intervention studies. According to GRADE, the quality of evidence indicates the extent to which one can be confident that the estimate of effect is correct. Taking into account the entire body of evidence on one outcome, four levels of evidence quality are applied: + , very low; +  + , low; +  +  + , moderate; and +  +  +  + , high. Adapting the original GRADE approach and considering the proposal by Huguet et al. [26], all bodies of evidence were initially graded as high quality of evidence. Considering the following criteria led to decreasing evidence quality: (i) risk of bias, (ii) inconsistency, (iii) indirectness, (iv) imprecision and (v) publication bias (for details on the criteria, see [24].

Data synthesis

Extracted study characteristics and data were summarised in tables, together with risk of bias assessments. For data synthesis, the following two effect measures were used:

  • Risk differences were applied to calculate attributable risk to sepsis exposure as follows: Using data of the individual studies we subtracted the absolute risk of developing sequelae in controls from the risk of developing sequelae in cases (infected minus uninfected and corresponding 95% confidence intervals).

  • Positive predictive values were used to estimate the probability that an individual with an adverse neurodevelopmental outcome during infancy continues to suffer from impairment during later life. Positive predictive values were either taken directly from the publications or calculated using the reported data as follows: number of true positives divided by the sum of true positives and false positives.

Risk differences were pooled across the studies, using the ‘metan command’ in Stata (Stata 12, Stata Corp, TX, US). In the presence of heterogeneity, a random-effects model was used. Otherwise, study data were combined using a fixed-effects model. I2, a direct measure of inconsistency of study results in a meta-analysis, was used to quantify the extent of heterogeneity. I2 ranges from 0% to 100%, with 0% indicating no inconsistency. Because the numbers of studies were too small (< 10), publication bias was not investigated. Positive predictive values were not pooled but rather presented as a range of values to account for heterogeneity [27].

Development of the outcome tree

The results of the systematic reviews were used to construct an outcome tree, based on the methodology described by Kretzschmar et al. [11]. An outcome tree maps the weighted progression of a disease over time by ordering the conditional probabilities of associated health outcomes. Blocks indicate health outcomes. Arrows indicate transition between outcomes (e.g. the transition from neurodevelopmental impairment to permanent cognitive impairment). Attributable risks (i.e. risk differences) that derived from the systematic review were attached to the respective blocks and arrows.

Results

Neurodevelopmental sequelae of neonatal sepsis

Our search identified a total of 207 titles (Figure 1A). After eliminating duplicates and screening of titles and abstracts four publications were left for full text evaluation. Of these, only one systematic review fulfilled the inclusion criteria and thus, was eligible for further analysis [3], whereas the remaining three publications were not eligible [28-30]. This systematic review was of acceptable methodological quality (AMSTAR summary score: 7/11). The review included 17 original studies reporting data on neurodevelopmental sequelae of neonatal sepsis in VLBW infants. We screened the abstracts and full texts of these studies and identified nine of them to be eligible (see Figure 1B), whereas seven did not fulfil the inclusion criteria [6,31-36] and one citation could not be located in data banks or libraries.

Figure 1

Selection process for (panel A) systematic review of systematic reviews and (panel B) primary studies on neurological sequelae of neonatal sepsis, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 25 September 2013

/images/dynamic/articles/21389/15-00078-f1

DIMDI: Deutsches Institut für Medizinische Dokumentation und Information.

All nine included studies [7,8,37-43] were cohort studies. Details are shown in Table 1. The studies included a total of 5,620 neonates born between 1983 and 2007 and were conducted in six upper-middle and high-income countries. Three studies provided data on infants with extremely low birth weight (ELBW; < 1,000g) [7,40,44]. A further three studies reported on neonates with VLBW [37,38,41], whereas the remaining three studies based their inclusion criteria on gestational age [8,42,43]. Eight studies provided a definition of neonatal sepsis that was based on clinical and/or laboratory parameters. One study did not provide a definition [41]. One study reported on invasive Candida spp. infections only [40]. Duration of follow-up varied between 12 and 52 months.

Table 1

Characteristics of included studies on neurodevelopment after neonatal sepsis, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 25 September 2013


Reference Location Birth year(s) Population Definition of sepsis Duration of follow-up na Risk of biasb
Addison et al. (2009) [37] US 1999–2001 VLBW Positive blood culture; at least 7 days of antibiotics 12–18 months 65 High
Chen et al. (2008) [38] Taiwan 1998–2005 VLBW Culture-proven sepsis and unstable vital signs 18–39 months 122 Low
Friedman et al. (2000) [40] Canada 1988–1996 ELBW Positive-culture Candida spp. or supportive brain autopsy 24 months 299 Low
Göcer et al. (2011) [41] Turkey 2002 VLBW Not defined 33–45 months 117 High
Hack et al. (2000) [44] US 1992–1995 ELBW Positive blood culture and clinical signs 20 months 221 Low
Msall et al. (1994) [42] US 1983–1986 23–28 weeks GA Positive blood culture and 14–21 days antibiotics 52 months 149 Low
Schlapbach et al. (2011) [65] Switzerland 2000–2007 24–28 weeks GA Positive blood culture and clinical signs and antibiotics for ≥ 5 days 18–24 months 372 Low
Shah et al. (2008) [8] Australia 2001–2003 < 30 weeks GA Positive blood culture and biomarker and antibiotics ≥ 5 days 24 months 192 High
Stoll et al. (2004) [7] US 1993–2001 ELBW Positive blood culture and antibiotics ≥  5 days 18–22 months 4083 Low

ELBW: extremely low birth weight; GA: gestational age; US: United States; VLBW: very low birth weight.

a Study size

b According to Newcastle-Ottawa scale [21]

From the reported outcomes we considered the following five outcomes as clinically relevant: neurodevelopmental impairment, cerebral palsy, vision impairment, hearing impairment and death. Neurodevelopmental impairment was defined as having a Mental Developmental Index (MDI) < 70 [18]. For vision and hearing impairment, varying definitions were used in the studies. According to the Newcastle-Ottawa Scale [21], three studies had high risk of bias, while the remaining six showed a low risk of bias (Table 1).

Figure 2A shows the results of the meta-analysis of risk differences for neurodevelopmental impairment in infants with neonatal sepsis, as compared with those without sepsis. Eight studies reported risk estimates. We calculated a statistically significant pooled risk difference of 13% (95% CI: 5–20), with a large and significant between-study heterogeneity. This heterogeneity was mainly due to the study by Hack et al. [44]. Excluding this study lowered heterogeneity (I2 = 32.6%; p = 0.18) but had only a small impact on the pooled risk difference (15%; 95%CI: 9–20%). Since the study by Friedman et al. [40] was the only study in which the exposure was non-bacterial (Candida spp.), further sensitivity analysis was performed with six studies [7,37,38,41-44]. The pooled risk difference (14%; 95%CI: 9̶̶–19%) did not differ largely from the estimate of the complete dataset, but showed lower heterogeneity (I2 = 26.1%; p = 0.24).

Figure 2

Forest plot of risk differences of (panel A) impaired neurodevelopment (mental development index < 70) and (panel B) cerebral palsy in neonates with sepsis compared with those without sepsis, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 25 September 2013

/images/dynamic/articles/21389/15-00078-f2

CI: confidence interval; MDI: Mental Developmental Index.

Studies are ordered alphabetically by first author. The pooled risk differences (overall; diamonds) were calculated by means of a random-effects model. Ninety-five percent CIs are shown in parentheses and as horizontal bars.

Neurodevelopmental impairment was defined as having a MDI < 70 [18].

Figure 2B displays the single study estimates and the pooled risk difference for the outcome cerebral palsy, which was reported by four studies. Infants who experienced neonatal sepsis had an 8% (95% CI: 6–10) higher risk of developing cerebral palsy than those who did not. Study results were highly homogenous (I2 = 0%).

Only two studies reported data on hearing impairment (Figure 3A) and vision impairment (Figure 3B) following neonatal sepsis. While there was a significant effect on vision impairment (9%; 95% CI: 7–11), the risk difference for hearing impairment was smaller and not significant (4%; 95% CI: -2 to 10).

Figure 3

Forest plot of risk differences of (panel A) hearing impairment and (panel B) vision impairment in neonates with sepsis, compared with neonates without sepsis, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 25 September 2013]

/images/dynamic/articles/21389/15-00078-f3

CIs: confidence intervals.

Studies are ordered alphabetically by first author. The pooled risk differences (overall; diamonds) were calculated by means of a random-effects model. Ninety-five percent CIs are shown in parentheses and as horizontal bars.

Two studies analysed mortality in association with neonatal sepsis. From Friedman et al. [40] we could calculate a risk difference of 2% for invasive Candida spp. infection (95% CI: -13 to 17), while Msall et al. [42] provided data to calculate a risk difference of 14% for bacterial sepsis (95% CI: -5̶ to 33). Meta-analysis was not conducted because exposure and results were too heterogeneous.

Neither grouping of primary studies by birth weight, nor by publication date had an influence on risk differences of neurodevelopmental impairment or cerebral palsy. The number of studies was too small to allow stratified meta-analysis.

To systematically assess the quality of evidence for each outcome we applied the GRADE methodology. For neurodevelopmental impairment, the quality of the evidence had to be graded down by three levels: (i) for serious risk of bias, (ii) for serious inconsistency due to widely differing point estimates of the single studies, and (iii) for serious imprecision due to a wide CI around the pooled estimate. Therefore, evidence quality was only ‘very low’ for this outcome. Regarding cerebral palsy, very serious risk of bias (grading down two levels) led to an evidence quality of ‘low’ for this outcome. For the outcome vision impairment, evidence quality was graded down to ‘moderate’ due to serious risk of bias. Accounting for serious risk of bias and serious imprecision, evidence quality was graded down to ‘low’ for the outcome hearing impairment.

Predictive value of early neurodevelopmental impairment for later cognitive function

Our search identified three potentially eligible reviews. After title and abstract screening, only one publication remained for full text screening [45]. This systematic review fulfilled our inclusion criteria and was therefore used as a database for further analysis.

The review was of acceptable methodological quality (AMSTAR summary score: 7/11). It contained a total of 18 publications that reported data on the relation between MDI scores during the first three years of life and cognitive function measured later in life in VLBW infants. After abstract and full text screening, four studies were eligible for further analysis [39,46-48], whereas the remaining 14 did not meet the inclusion criteria [49-62].

All included studies were cohort studies. Details are shown in Table 2. Studies accumulated a total of 472 infants of either VLBW (n = 2 studies) or ELBW (n = 2 studies) who were born between 1977 and 2004 in three different high-income countries. All four studies used the Bayley Scale of infant development to assess the proportion of infants with neurodevelopmental impairment (i.e. MDI < 70) at 12 to 24 months, and re-evaluated the study sample at 3.4 to 8.6 years of age, using three different test batteries. According to the SIGN50 checklist [23], all four studies had a low risk of bias.

Table 2

Characteristics of included studies on predictive value of early neurodevelopmental impairment for later cognitive function, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 2 July 2014


Reference Location Birth year(s) Population Test at first examination Test at second examination Age
at first test
Age
at second test
na PPV
(95% CI)
Prevalence of MDI < 70 Risk of biasb
Munck et al. (2012) [47] Finland 2001–2004 VLBW Bayley Scale vers. Two (MDI < 70) Wechsler Scale (FSIQ < 70) 2 years (corrected) 5 years 124 83%
(36–100)
4% Low
Hack et al. (2005) [39] US 1992–1995 ELBW Bayley Scale vers. Two (MDI < 70) Kaufman Assessment Battery for Children, Mental Processing Composite (MPC < 70) 20 months 8.6 years 200 37%
(27–49)
15% Low
Kitchen et al. (1987) [46] Australia 1977–1980 ELBW Bayley Scale (MDI < 70) Wechsler Preschool and Primary Scale (FSIQ < 71) 2 years (corrected) 5.5 years (corrected) 54 67%
(22–96)
7% Low
Ross et al. (1985) [48] US 1978–1979 VLBW Bayley Scale (MDI < 70) Stanford-Binet Intelligence Scale (IQ < 70) 12 months 3.4 years 94 75%
(35–97)
6% Low

CI: confidence interval; ELBW: extremely low birth weight; FSIQ: full-scale intelligence quotient; IQ: intelligence quotient; MDI: mental development index; MPC: mental processing composite; PPV: positive predictive value; US: United States; VLBW: very low birth weight.

a Study size.

b According to SIGN50 checklist [23].

The positive predictive value, i.e. the probability of having a positive test result at the second examination when the first test result was positive, varied between 37% (95% CI: 27–49) and 83% (95% CI: 36–100). Heterogeneity between estimates was mainly due to the study by Hack et al. [39], which was the only study that did not use an externally validated test battery to assess IQ at follow-up. Excluding their estimate from the study pool resulted in positive predictive values between 67% (95% CI: 22–96) and 83% (95% CI: 36­–100).

Outcome tree for neurological sequelae of neonatal sepsis

The results of the systematic review were used to develop the outcome tree (Figure 4). Risk differences obtained from meta-analyses were used to estimate the transitional probabilities for acquiring neurodevelopmental impairment, cerebral palsy, vision impairment, hearing impairment and death after having experienced sepsis during neonatal life. Furthermore, we used the positive predictive values identified in the second systematic review to estimate the probability of having a permanently impaired cognitive function after early neurodevelopmental impairment.

Figure 4

Evidence-based outcome tree for neurological sequelae of neonatal sepsis, umbrella review on neurological sequelae of healthcare-associated sepsis in very-low-birthweight infants, date of search 2 July 2014

/images/dynamic/articles/21389/15-00078-f4

Outcomes (e.g. neurodevelopmental impairment) are shown in blocks. Arrows represent transitions between outcomes. Percentages (%) attached to arrows correspond to transitional probabilities between outcomes.

Discussion

We developed an outcome tree for neurological sequelae of neonatal sepsis in VLBW infants using the methods of evidence-based medicine. Our study shows that 4–14% of neurological sequelae in ELBW and VLBW are attributable to neonatal sepsis. Although this may be lower than anticipated in this high-risk group, about three-quarters of infants with early neurodevelopmental impairment suffer from persistent cognitive impairment later in life. Evidence quality was low to very low, mainly due to high risk of bias in the single studies as well as imprecision of estimates.

Due to the inclusion criteria of the systematic review, primary studies which used different definitions of neonatal sepsis were analysed together. While some authors defined sepsis as culture-proven sepsis plus clinical signs [38,44], others applied a definition that included antibiotic treatment in addition to positive blood culture [37,42,43]. Moreover, one study did not provide any definition of neonatal sepsis [41]. Definition issues also applied for outcomes. Vision impairment was defined differently in the primary studies: while both studies defined vision impairment as uni- or bilateral blindness, one study also allowed the need for corrective lenses [44]. To estimate the impact of definition of exposure and/or outcome on the associations of interest, a larger number of carefully conducted prospective studies with subgroup analyses of sufficient power would be needed.

To assess attributable risk, we used the original data of the studies included in the systematic reviews to calculate risk differences. This approach may not adjust for potential confounders, which might bias the relation between exposure and outcome. In nearly all analysed studies the infants with neonatal sepsis differed in a number of important prognostic variables from controls such as gestational age, birth weight and co-morbidities. While the original studies did adjust for such variables by applying multivariate analysis, we could not do the same because (i) potential confounders were not uniform, (ii) some studies adjusted for variables which are not confounders for our purpose, and (iii) we have not had access to the original database with the individual data to do so. We therefore did not follow this approach any further, but considered the problem of confounding in the risk of bias assessments. The study by Hack et al. [44] on ELBW infants born between 1992 and 1996 illustrates that confounding may be outcome specific and leads to surprising results. In this particular study, neonatal exposure to sepsis was associated with hearing impairment, but also with a lower likelihood of neurodevelopmental impairment. This may best be explained by confounding due to postnatal use of corticosteroids administered in that time period in neonates without symptoms of infection to prevent chronic lung disease. Among others, Yeh et al. were able to show the strong side effects of this therapeutic strategy on neurodevelopment when they evaluated long-term neurodevelopmental outcomes of children who had participated in a randomised controlled trial on the effects of dexamethasone therapy [63]. As for the other studies, it was surprising to find that risk differences for neurodevelopmental impairment and cerebral palsy following neonatal sepsis were similar across different settings. Grouping of studies by birth weight or year of publication did not reveal trends for risk differences. More studies would be needed to analyse whether they are independent of birth weight, gestational age, setting and time. It may be concluded, however, that the attributable risks can be used as endpoints for studies evaluating the effectiveness of specific sepsis therapy. Further, it may be hypothesised that sepsis therapy has not, over the years, improved to a similar extent as overall neonatal intensive care.

For meta-analysis, we pooled the risk differences from the individual studies to arrive at a single measure of attributable risk for each outcome. Statistical pooling of risk differences has been reported to cause problems with consistency, with relative risk estimates (including odds ratios) being more consistent than risk differences [64]. For comparison, we pooled the estimates of the calculated relative risks (data available upon request from the authors). Since this analysis did not detect less inconsistency we concluded pooling of risk differences to be an adequate approach.

Our study has several strengths. We based our analyses on a comprehensive systematic review of systematic reviews. By using an outcome-focused approach, we were able to perform a detailed assessment of risk of bias and evidence quality, thereby emphasising the limitations of the current evidence base.

Limitations of our study mainly arise from the limitations in the systematic reviews and primary studies included. In particular, risk of bias and imprecision of the reported estimates might limit the scientific and clinical value of the data summarised here. The search by Alshaikh et al. [3] was last performed June 2012. We did not conduct a more recent search for primary studies. Thus we may not exclude the possibility that more recent studies could have influenced our findings. Our results may be further improved by using the primary datasets of the included cohort studies and then adjusting for confounders compiled throughout all studies, such as sex, birth weight and gestational age.

In conclusion, this systematic review of systematic reviews shows that VLBW infants with sepsis during neonatal life have an increased risk of developing permanent neurological impairment during later life. The magnitude of this effect varies by outcome, while evidence quality was low to very low. To improve the evidence base, carefully planned and conducted prospective studies are needed.


Acknowledgements

Funding source: This study was conducted within the project ‘Evidence-based research to develop outcome trees for healthcare-associated infections’ initiated and funded by the European Centre for Disease Prevention and Control (ECDC) through a specific service contract (ECDC.4226) to the Robert Koch Institute, Public Federal Institute under jurisdiction of the German Health Ministry, Berlin, Germany. The authors would like to thank Benedetta Allegranzi (World Health Organization), Maria Deja (Charité University Medicine Berlin), and Andreas Jansen, Diamantis Plachouras and Edoardo Colzani (all ECDC), for discussion and comments.

Conflict of Interest

None declared

Authors’ contributions

Sebastian Haller, Thomas Harder conceptualised the study, performed systematic reviews, extracted the data, performed the statistical analysis and drafted the manuscript. Philipp Deindl gave scientific advice, reviewed the data and revised the manuscript. Alessandro Cassini conceptualised the study, reviewed the data and revised the manuscript. Carl Suetens gave scientific advice, reviewed the data and revised the manuscript. Walter Zingg gave scientific advice, reviewed the data and revised the manuscript. Muna Abu Sin reviewed the data, contributed to analysis and interpretation of the data and revised the manuscript. Edward Velasco reviewed the data and contributed to analysis and interpretation of the data. Bettina Weiss reviewed the data, contributed to analysis and interpretation of the data and revised the manuscript. Tanja Ducomble reviewed the data and contributed to analysis and interpretation of the data. Madlen Sixtensson reviewed the data and contributed to analysis and interpretation of the data. Tim Eckmanns conceptualised the study, coordinated the study, reviewed the data and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.


References

  1. Lawn JE, Cousens S, Zupan J, Lancet Neonatal Survival Steering Team. 4 million neonatal deaths: when? Where? Why?Lancet. 2005;365(9462):891-900. DOI: 10.1016/S0140-6736(05)71048-5 PMID: 15752534

  2. Vergnano S, Menson E, Kennea N, Embleton N, Russell AB, Watts T,  et al.  Neonatal infections in England: the NeonIN surveillance network. Arch Dis Child Fetal Neonatal Ed. 2011;96(1):F9-14. DOI: 10.1136/adc.2009.178798 PMID: 20876594

  3. Alshaikh B, Yusuf K, Sauve R. Neurodevelopmental outcomes of very low birth weight infants with neonatal sepsis: systematic review and meta-analysis.J Perinatol. 2013;33(7):558-64. DOI: 10.1038/jp.2012.167 PMID: 23328927

  4. Chan GJ, Lee AC, Baqui AH, Tan J, Black RE. Risk of early-onset neonatal infection with maternal infection or colonization: a global systematic review and meta-analysis.PLoS Med. 2013;10(8):e1001502. DOI: 10.1371/journal.pmed.1001502 PMID: 23976885

  5. Klinger G, Levy I, Sirota L, Boyko V, Reichman B, Lerner-Geva L, Israel Neonatal N. Epidemiology and risk factors for early onset sepsis among very-low-birthweight infants. Am J Obstet Gynecol 2009; 201(1):38 e1-6.

  6. Murphy DJ, Hope PL, Johnson A. Neonatal risk factors for cerebral palsy in very preterm babies: case-control study.BMJ. 1997;314(7078):404-8. DOI: 10.1136/bmj.314.7078.404 PMID: 9040385

  7. Stoll BJ, Hansen NI, Adams-Chapman I, Fanaroff AA, Hintz SR, Vohr B,  et al. , National Institute of Child Health and Human Development Neonatal Research Network. Neurodevelopmental and growth impairment among extremely low-birth-weight infants with neonatal infection.JAMA. 2004;292(19):2357-65. DOI: 10.1001/jama.292.19.2357 PMID: 15547163

  8. Shah DK, Doyle LW, Anderson PJ, Bear M, Daley AJ, Hunt RW, Inder TE. Adverse neurodevelopment in preterm infants with postnatal sepsis or necrotizing enterocolitis is mediated by white matter abnormalities on magnetic resonance imaging at term. J Pediatr 2008; 153(2):170-5, 175.e171.

  9. Nelson KB, Dambrosia JM, Grether JK, Phillips TM. Neonatal cytokines and coagulation factors in children with cerebral palsy.Ann Neurol. 1998;44(4):665-75. DOI: 10.1002/ana.410440413 PMID: 9778266

  10. Kidokoro H, Anderson PJ, Doyle LW, Woodward LJ, Neil JJ, Inder TE. Brain injury and altered brain growth in preterm infants: predictors and prognosis.Pediatrics. 2014;134(2):e444-53. DOI: 10.1542/peds.2013-2336 PMID: 25070300

  11. Kretzschmar M, Mangen MJ, Pinheiro P, Jahn B, Fèvre EM, Longhi S,  et al. , BCoDE consortium. New methodology for estimating the burden of infectious diseases in Europe.PLoS Med. 2012;9(4):e1001205. DOI: 10.1371/journal.pmed.1001205 PMID: 22529750

  12. Cislak A, Safron M, Pratt M, Gaspar T, Luszczynska A. Family-related predictors of body weight and weight-related behaviours among children and adolescents: a systematic umbrella review.Child Care Health Dev. 2012;38(3):321-31. DOI: 10.1111/j.1365-2214.2011.01285.x PMID: 21752064

  13. Ioannidis JP, Zhou Y, Chang CQ, Schully SD, Khoury MJ, Freedman AN. Potential increased risk of cancer from commonly used medications: an umbrella review of meta-analyses.Ann Oncol. 2014;25(1):16-23. DOI: 10.1093/annonc/mdt372 PMID: 24310915

  14. Whitlock EP, Lin JS, Chou R, Shekelle P, Robinson KA. Using existing systematic reviews in complex systematic reviews.Ann Intern Med. 2008;148(10):776-82. DOI: 10.7326/0003-4819-148-10-200805200-00010 PMID: 18490690

  15. Robinson KA, Whitlock EP, Oneil ME, Anderson JK, Hartling L, Dryden DM,  et al.  Integration of existing systematic reviews into new reviews: identification of guidance needs. Syst Rev. 2014;3(1):60. DOI: 10.1186/2046-4053-3-60 PMID: 24956937

  16. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151(4):264-9, W264.

  17. The World Bank. Country and Lending Groups. Washington, DC: The World Bank. [Accessed: 6 February 2015] Available from: http://data.worldbank.org/about/country-and-lending-groups

  18. Bayley N. Bayley Scales of infant development. 2nd ed. San Antonio, TX: Psychological Corporation; 1993.

  19. Wechsler D. Wechsler Intelligence Scale for children. New York: Psychological Corp.; 1949.

  20. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C,  et al.  Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10. DOI: 10.1186/1471-2288-7-10 PMID: 17302989

  21. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A,  et al.  Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66(9):982-93. DOI: 10.1016/j.jclinepi.2013.03.003 PMID: 23683848

  22. Higgins JPTAD. Sterne JAC, editors. Assessing risk of bias in included studies. In: Higgins JPT, Green S,editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from: www.cochrane-handbook.org

  23. Scottish Intercollegiate Guidelines Network (SIGN). SIGN 50: A guideline developer's handbook. Edinburgh: SIGN; October 2014. Available from: http://www.sign.ac.uk/guidelines/fulltext/50/

  24. Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.J Clin Epidemiol. 2011;64(4):380-2. DOI: 10.1016/j.jclinepi.2010.09.011 PMID: 21185693

  25. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P,  et al. , GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008;336(7650):924-6. DOI: 10.1136/bmj.39489.470347.AD PMID: 18436948

  26. Huguet A, Hayden JA, Stinson J, McGrath PJ, Chambers CT, Tougas ME,  et al.  Judging the quality of evidence in reviews of prognostic factor research: adapting the GRADE framework. Syst Rev. 2013;2(1):71. DOI: 10.1186/2046-4053-2-71 PMID: 24007720

  27. Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and Presenting Results. In: Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0. The Cochrane Collaboration, 2010. Available from: http://srdta.cochrane.org/

  28. Ergaz Z, Ornoy A. Perinatal and early postnatal factors underlying developmental delay and disabilities.Dev Disabil Res Rev. 2011;17(2):59-70. DOI: 10.1002/ddrr.1101 PMID: 23362026

  29. Swinburn K, McIntyre S, Hines M, Novak I. Research for infants at high risk of cerebral palsy: Is it time for an agreed set of outcomes and measures? 6th Biennial Conference of the Australasian Academy of Cerebral Palsy & Developmental Medicine, United Kingdom: Wiley-Blackwell Publishing Ltd; 2012.

  30. Teune MJ, Bakhuizen S, Gyamfi Bannerman C, Opmeer BC, van Kaam AH, van Wassenaer AG, et al. A systematic review of severe morbidity in infants born late preterm. Am J Obstet Gynecol. 2011; 205(4):374. e1-9.

  31. Grether JK, Nelson KB, Emery ES, Cummins SK. Prenatal and perinatal factors and cerebral palsy in very low birth weight infants.J Pediatr. 1996;128(3):407-14. DOI: 10.1016/S0022-3476(96)70292-5 PMID: 8774515

  32. Hoekstra RE, Ferrara TB, Couser RJ, Payne NR, Connett JE. Survival and long-term neurodevelopmental outcome of extremely premature infants born at 23-26 weeks’ gestational age at a tertiary center.Pediatrics. 2004;113(1 Pt 1):e1-6. DOI: 10.1542/peds.113.1.e1 PMID: 14702487

  33. Jang DH, Sung IY, Jeon JY, Moon HJ, Kim KS, Kim EA,  et al.  Neurodevelopmental outcomes in very low-birth-weight infants in Korea: 1998-2007 vs 1989-1997. J Child Neurol. 2011;26(11):1405-10. DOI: 10.1177/0883073811408606 PMID: 21693650

  34. Kono Y, Mishina J, Yonemoto N, Kusuda S, Fujimura M, NICU Network, Japan. Neonatal correlates of adverse outcomes in very low-birthweight infants in the NICU Network.Pediatr Int. 2011;53(6):930-5. DOI: 10.1111/j.1442-200X.2011.03424.x PMID: 21752150

  35. Lee BE, Cheung PY, Robinson JL, Evanochko C, Robertson CM. Comparative study of mortality and morbidity in premature infants (birth weight, < 1,250 g) with candidemia or candidal meningitis.Clin Infect Dis. 1998;27(3):559-65. DOI: 10.1086/514712 PMID: 9770157

  36. Tapia Collados C, Feret Siguile MA, Serrano Martínez JL, Sánchez Payá J, Palazón Azorín I, Alonso Barrena AV,  et al.  [Clinical course and prognostic factors in newborns with very low birth weight]. An Esp Pediatr. 1997;47(4):398-404.PMID: 9499310

  37. Addison K, Griffin MP, Moorman JR, Lake DE, O’Shea TM. Heart rate characteristics and neurodevelopmental outcome in very low birth weight infants.J Perinatol. 2009;29(11):750-6. DOI: 10.1038/jp.2009.81 PMID: 19554011

  38. Chen YY, Chen HC, Wang TM, Hsui CC. Developmental outcomes among very low birth weight infants with normal cranial ultrasound images. Clinical Neonatology. 2008; 15(1):5-9.

  39. Hack M, Taylor HG, Drotar D, Schluchter M, Cartar L, Wilson-Costello D,  et al.  Poor predictive validity of the Bayley Scales of Infant Development for cognitive function of extremely low birth weight children at school age. Pediatrics. 2005;116(2):333-41. DOI: 10.1542/peds.2005-0173 PMID: 16061586

  40. Friedman S, Richardson SE, Jacobs SE, O’Brien K. Systemic Candida infection in extremely low birth weight infants: short term morbidity and long term neurodevelopmental outcome.Pediatr Infect Dis J. 2000;19(6):499-504. DOI: 10.1097/00006454-200006000-00002 PMID: 10877162

  41. Göçer C Kavuncuoğlu S, Arslan G, Ertem I, Özbek S, Öztüregen E, et al. Neurodevelopmental problems and factors affecting neurological morbidity of very low birth weight premature infants. Turkish Archives of Pediatrics. 2011; 46:199-206. Available from: http://www.turkpediatriarsivi.com/sayilar/205/buyuk/207-214ing.pdf

  42. Msall ME, Buck GM, Rogers BT, Merke DP, Wan CC, Catanzaro NL,  et al.  Multivariate risks among extremely premature infants. J Perinatol. 1994;14(1):41-7.PMID: 8169677

  43. Schlapbach LJ, Aebischer M, Adams M, Natalucci G, Bonhoeffer J, Latzin P,  et al. , Swiss Neonatal Network and Follow-Up Group. Impact of sepsis on neurodevelopmental outcome in a Swiss National Cohort of extremely premature infants.Pediatrics. 2011;128(2):e348-57. DOI: 10.1542/peds.2010-3338 PMID: 21768312

  44. Hack M, Wilson-Costello D, Friedman H, Taylor GH, Schluchter M, Fanaroff AA. Neurodevelopment and predictors of outcomes of children with birth weights of less than 1000 g: 1992-1995.Arch Pediatr Adolesc Med. 2000;154(7):725-31. DOI: 10.1001/archpedi.154.7.725 PMID: 10891026

  45. Luttikhuizen dos Santos ES, de Kieviet JF, Königs M, van Elburg RM, Oosterlaan J. Predictive value of the Bayley scales of infant development on development of very preterm/very low birth weight children: a meta-analysis.Early Hum Dev. 2013;89(7):487-96. DOI: 10.1016/j.earlhumdev.2013.03.008 PMID: 23597678

  46. Kitchen WH, Ford GW, Rickards AL, Lissenden JV, Ryan MM. Children of birth weight less than 1000 g: changing outcome between ages 2 and 5 years.J Pediatr. 1987;110(2):283-8. DOI: 10.1016/S0022-3476(87)80174-9 PMID: 2433422

  47. Munck P, Niemi P, Lapinleimu H, Lehtonen L, Haataja L, PIPARI Study Group. Stability of cognitive outcome from 2 to 5 years of age in very low birth weight children.Pediatrics. 2012;129(3):503-8. DOI: 10.1542/peds.2011-1566 PMID: 22371467

  48. Ross G, Lipper EG, Auld PA. Consistency and change in the development of premature infants weighing less than 1,501 grams at birth.Pediatrics. 1985;76(6):885-91.PMID: 4069857

  49. Victorian Infant Collaborative Study Group. Eight-year outcome in infants with birth weight of 500 to 999 grams: continuing regional study of 1979 and 1980 births.J Pediatr. 1991;118(5):761-7. DOI: 10.1016/S0022-3476(05)80044-7 PMID: 2019933

  50. Dezoete JA, MacArthur BA, Tuck B. Prediction of Bayley and Stanford-Binet scores with a group of very low birthweight children.Child Care Health Dev. 2003;29(5):367-72. DOI: 10.1046/j.1365-2214.2003.00349.x PMID: 12904244

  51. Doyle LW, Casalaz D, Victorian Infant Collaborative Study Group. Outcome at 14 years of extremely low birthweight infants: a regional study.Arch Dis Child Fetal Neonatal Ed. 2001;85(3):F159-64. DOI: 10.1136/fn.85.3.F159 PMID: 11668155

  52. Evensen KA, Skranes J, Brubakk AM, Vik T. Predictive value of early motor evaluation in preterm very low birth weight and term small for gestational age children.Early Hum Dev. 2009;85(8):511-8. DOI: 10.1016/j.earlhumdev.2009.04.007 PMID: 19450939

  53. Janssen AJ, Nijhuis-van der Sanden MW, Akkermans RP, Tissingh J, Oostendorp RA, Kollée LA. A model to predict motor performance in preterm infants at 5 years.Early Hum Dev. 2009;85(9):599-604. DOI: 10.1016/j.earlhumdev.2009.07.001 PMID: 19643556

  54. Potharst ES, Houtzager BA, van Sonderen L, Tamminga P, Kok JH, Last BF,  et al.  Prediction of cognitive abilities at the age of 5 years using developmental follow-up assessments at the age of 2 and 3 years in very preterm children. Dev Med Child Neurol. 2012;54(3):240-6. DOI: 10.1111/j.1469-8749.2011.04181.x PMID: 22188215

  55. Rose SA, Feldman JF, Wallace IF. Infant information processing in relation to six-year cognitive outcomes.Child Dev. 1992;63(5):1126-41. DOI: 10.2307/1131522 PMID: 1446544

  56. Sajaniemi N, Hakamies-Blomqvist L, Katainen S, von Wendt L. Early cognitive and behavioral predictors of later performance: a follow-up study of ELBW children from ages 2 to 4.Early Child Res Q. 2001;16(3):343-61. DOI: 10.1016/S0885-2006(01)00107-7

  57. Siegel LS. Correction for prematurity and its consequences for the assessment of the very low birth weight infant.Child Dev. 1983;54(5):1176-88. DOI: 10.2307/1129673 PMID: 6194943

  58. Skranes J, Vik T, Nilsen G, Smevik O, Andersson HW, Brubakk AM. Can cerebral MRI at age 1 year predict motor and intellectual outcomes in very-low-birthweight children?Dev Med Child Neurol. 1998;40(4):256-62. DOI: 10.1111/j.1469-8749.1998.tb15458.x PMID: 9593497

  59. Spittle AJ, Spencer-Smith MM, Eeles AL, Lee KJ, Lorefice LE, Anderson PJ,  et al.  Does the Bayley-III Motor Scale at 2 years predict motor outcome at 4 years in very preterm children? Dev Med Child Neurol. 2013;55(5):448-52. DOI: 10.1111/dmcn.12049 PMID: 23216518

  60. van Vliet EO, de Kieviet JF, van der Voorn JP, Been JV, Oosterlaan J, van Elburg RM. Placental pathology and long-term neurodevelopment of very preterm infants. Am J Obstet Gynecol. 2012; 206(6):489. e1-7.

  61. Weisglas-Kuperus N, Baerts W, Smrkovsky M, Sauer PJ. Effects of biological and social factors on the cognitive development of very low birth weight children.Pediatrics. 1993;92(5):658-65.PMID: 7692381

  62. Williams ML, Lewandowski LJ, Coplan J, D’Eugenio DB. Neurodevelopmental outcome of preschool children born preterm with and without intracranial hemorrhage.Dev Med Child Neurol. 1987;29(2):243-9. DOI: 10.1111/j.1469-8749.1987.tb02142.x PMID: 3582794

  63. Yeh TF, Lin YJ, Lin HC, Huang CC, Hsieh WS, Lin CH,  et al.  Outcomes at school age after postnatal dexamethasone therapy for lung disease of prematurity. N Engl J Med. 2004;350(13):1304-13. DOI: 10.1056/NEJMoa032089 PMID: 15044641

  64. Deeks JJ. Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.Stat Med. 2002;21(11):1575-600. DOI: 10.1002/sim.1188 PMID: 12111921

  65. Schlapbach LJ, Aebischer M, Adams M, Natalucci G, Bonhoeffer J, Latzin P,  et al. , Swiss Neonatal Network and Follow-Up Group. Impact of sepsis on neurodevelopmental outcome in a Swiss National Cohort of extremely premature infants.Pediatrics. 2011;128(2):e348-57. DOI: 10.1542/peds.2010-3338 PMID: 21768312



Back to Table of Contents
Previous Download (pdf)
Next

The publisher’s policy on data collection and use of cookies.

Disclaimer: The opinions expressed by authors contributing to Eurosurveillance do not necessarily reflect the opinions of the European Centre for Disease Prevention and Control (ECDC) or the editorial team or the institutions with which the authors are affiliated. Neither ECDC nor any person acting on behalf of ECDC is responsible for the use that might be made of the information in this journal. The information provided on the Eurosurveillance site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her physician. Our website does not host any form of commercial advertisement. Except where otherwise stated, all manuscripts published after 1 January 2016 will be published under the Creative Commons Attribution (CC BY) licence. You are free to share and adapt the material, but you must give appropriate credit, provide a link to the licence, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

Eurosurveillance [ISSN 1560-7917] - ©2007-2016. All rights reserved.
 

This website is certified by Health On the Net Foundation. Click to verify. This site complies with the HONcode standard for trustworthy health information:
verify here.