Prioritisation of infectious diseases in public health: feedback on the prioritisation

Tularaemia, though rare, has recently been increasingly reported in Germany. Most cases are indigenous infections. This report describes two epidemiologically independent infections with Francisella tularensis subspecies holarctica detected in Berlin in February 2011 that were acquired in central Anatolia, Turkey. In Turkey, there have been repeated tularaemia outbreaks since 2000 and the disease should therefore be considered as a differential diagnosis in travellers returning from that country.

Both patients had similar general symptoms including fever, pharyngitis, otitis and cervical lymphadenopathy, but showed different locations of the ulcerations specific for tularaemia.Both infections were characterised by slow and subacute clinical progression.Patient 1 stayed in Turkey between 25 July 2010 and 29 January 2011.Onset of symptoms was on 15 December 2010.Patient 2 had been in Turkey from 24 December 2010 to 8 January 2011 and fell ill on 10 January 2011.
The patients were diagnosed in mid-February 2011, after their return to Berlin.Patient 1 was diagnosed with oropharyngeal tularaemia, Patient 2 with the ulceroglandular form.The latter form is the most common expression of tularaemia.Typical symptoms are ulcerations next to the inoculation site linked with regional, often purulent inflammation of the lymph nodes.In advanced stages with extended lymphadenitis colli and suppurative ulcerations of multiple lymph nodes, the definite identification of the primary inoculation site is difficult to achieve.The exact description of primary clinical symptoms, however, is very important to elucidate the transmission routes and further epidemiological links [2,3].
Further interviews with Patient 2 revealed additional epidemiological information: the patient and one of his siblings both fell ill on 10 January 2011, and a further sibling two days later.However, these two patients remained in Turkey.

Laboratory confirmation
The detection of the pathogen by bacterial culture is difficult, special media are needed and growth is generally slow.More sensitive laboratory methods like PCR are only available in a small number of specialised laboratories [2,3].Laboratory confirmation for the two cases was available on 4 March.The German national reference laboratory for tularaemia in Munich could detect F. tularensis subspecies holarctica (Jellison type B) via PCR in both cases.Specific DNA sequences were detected in the purulent puncture material of affected lymph nodes.
A serological diagnosis done previously in the hospital in Berlin for Patient 2 had shown IgG and IgM antibodies against F. tularensis lipopolysacharides.The infection in Patient 1 was not proved serologically in the hospital, but could be confirmed through specific antibodies in the national reference laboratory.

Public health implications
After the diagnoses were confirmed, information was immediately reported according to the World Health Organization's WHO International Health Regulations (IHR) to the Robert Koch-Institute (RKI).At that time, no recent data were accessible about the tularaemia situation in Turkey.Data about tularaemia infections related to recent travel to Turkey or to neighbouring countries were not found.
Turkish citizens are one of the larger populations with migration background in Germany.At the end of 2010 nearly 2% of the German population were Turkish (1,629,480 inhabitants) [4].In Berlin at the end of 2009, 3% of the population were Turkish citizens (108,000 inhabitants) [5].In addition, an unknown number of German citizens of Turkish origin still have intensive contacts with their families in Turkey and frequently travel there.
Therefore, the health authorities in all 16 German federal states were informed on 8 March during the weekly epidemiological telephone conference (EpiLag), since more imported infections in international travellers could not be excluded.To get further data about the situation in Turkey, but also to alert other countries, the RKI informed the Turkish IHR focal point and the WHO Regional Office for Europe about the infections.Furthermore, information was sent to the Early Warning and Response System (EWRS) of the European Union.On the national level the decision was made to involve the German National Centre for Biological Security at the RKI because the pathogen F. tularensis is classified as a potential biological risk agent.Overall, the risk of further transmission and the threat to public health in Germany was estimated as low.

Epidemiological considerations
In the past years tularaemia outbreaks in Europe were documented in Norway [6,7], Sweden [8], Spain [9] and the UN Administered Province of Kosovo in accordance with Security Council Resolution 1244 of 1999 [10].Parts of Turkey have been strongly affected by the reemergence of tularaemia and a number of outbreaks have been published since 2000 [11][12][13][14][15][16].In Germany, tularaemia cases are rare, however, increasing numbers have been reported since 2007.Some travel-associated tularaemia cases have been reported in Germany (10 of 74 cases between 2001 and 2009), but only one case dating back to 2003 originated from Turkey [17,18].
The tularaemia cases reported in Germany since 2001 and the countries where the infections were acquired are shown in the Figure .Through information exchange via the national German and Turkish IHR Focal Points with the General Directorate of Primary Health Care in the Turkish Ministry of Health, further details were provided about the current situation in Turkey and possible sources of infection (personal communication: Dr. Tamer Sami Pelitli, 18 March 2011).More than a hundred tularaemia cases were reported to the national reference laboratories from central parts of Turkey, especially in the Yozgat province in 2010.The cases had been confirmed serologically and through PCR in two reference laboratories in Ankara and Bursa, Turkey.Based on this information the Ministry of Health of Turkey has implemented an action plan to fight the spread of tularaemia in 2010.This action plan is focussed on the rehabilitation of water systems.As a promising result the number of reported tularaemia cases decreased in 2011 compared with the previous years.After being informed by the German IHR Focal Point about the two cases in Berlin, the Turkish Ministry of Health started active surveillance work in the Yozgat province, but has not yet detected new transmission risks for tularaemia.
In both cases from Berlin the source of infection could not be identified conclusively.However, based on the available epidemiological information, the most probable cause of the two infections is consumption of contaminated water during the stay in a region of Turkey endemic for tularaemia.Transmission has often been associated with decentralised drinking water supply like cisterns and insufficiently treated surface water [15].The clinical presentation at least in one of the patients diagnosed in Berlin supports this suggestion.Oropharyngeal tularaemia is presumably related to oral ingestion of the pathogen.

Clinical considerations
Due to the relatively unspecific general symptoms of tularaemia and the variety of the primary disease patterns (depending on the route of infection) clinical diagnosis is not easy.Therefore, early suspicion of tularaemia depends on a precise medical history and epidemiological data regarding in particular travel history, animal contacts, occupation, and insect bites.

Methods and results
The objective of our study was to detect possible infection of wild birds with WNV during the outbreak in Greece, and to molecularly characterise and define the WNV strain geographical origin in positive samples.An -RT-PCR specific for Japanese encephalitis virus complex was performed for all extracts resulting in a 1,084-bp amplification product covering part of the nonstructural protein 5 (NS5) gene, as described earlier [6].A band of expected size was obtained from one PCR product derived from a magpie harvested near the village of Trilofos (40°28'25.57"N,22°58'28.62"E) in September 2010 (Figure 1).A serum sample from the magpie in question was tested for the presence of WNV IgG antibodies by indirect immunofluorescence test using a commercial kit (EUROIMMUN) [7]; the serum sample was positive at a dilution of 1/30.

Our first focus was on members of the
The positive PCR product was purified using the PureLink PCR Purification Kit (Invitrogen) and was bidirectionally sequenced using the fluorescent BigDye Terminator Cycle sequencing kit v3.1 (Applied Biosystems), followed by fragment separation with a 3,730xl DNA Analyzer (Applied Biosystems).
Phylogenetic analysis was conducted using MEGA 3.1 [9].Nucleotide sequences from other WNV strains were retrieved from Genbank (NCBI).Phylogenetic analysis of 797 nucleotide-long partial NS5 sequences was performed.A neighbour-joining phylogenetic tree using Kimura-2 parameter distance matrix was inferred from 26 WNV strain sequences (including that derived from the magpie in our study) and two sequences of the Japanese Encephalitis virus complex as outgroups (Figure 2).Node support was assessed with 1,000 bootstrap pseudo-replicates.
The WNV sequence derived from the Greek magpie clustered with WNV L2 strain sequences and presented highest (99.9%) sequence similarity to L2 strain sequences derived from birds of prey in Austria obtained in 2008 and 2009 [2].A 99.6% similarity was also observed with the corresponding region of an L2 strain derived from a dead goshawk in Hungary in 2004 [1].No amino acid changes were observed in the genomic region of the magpie derived WNV strain compared to Austrian and Hungarian strains.According to our analysis, all these strains as well as two strains from South Africa belong to the same sub-cluster.A lower sequence similarity (96.8%) was observed with a WNV L2 strain isolated during an outbreak in Russia in 2007.The Russian strain sequence groups with other African strains (including other South African strains) in a separate sub-cluster, suggesting a different reintroduction of WNV L2 in Europe [3].The sequence from the Greek magpie isolate was deposited in GenBank under accession no.JF719073.

Discussion
From early July through October 2010, a WNV outbreak in humans occurred in northern Greece, as confirmed by serologic evidence.To date, no WNV genomic sequences are available from the human cases during this outbreak.A WNV strain sequence derived from a magpie hunted during the outbreak of the human disease was found in this study.The sequence has highest sequence similarity to L2 strain sequences from birds of prey in Austria obtained in 2008 and 2009.WNV RNA fragments, though limited in size, (146 nt NS5 genomic region) with 100% sequence similarity to Hungarian and Austrian L2 strains, were also detected in two pools of mosquitoes caught during the time of the Greek outbreak and in the same area [8].
The mosquito WNV sequence was not included in our analysis because it did not overlap with the magpie WNV sequence.However, the similarity of both to the Austrian L2 strain sequences suggests that the same WNV strain is implicated in the magpie and mosquito infections and associated with the human outbreak.
The evidence may implicate this corvid species in local virus maintenance and generates concerns about possible overwintering and expansion of the virus in neighbouring areas.To test this hypothesis, research must be extended in non-epidemic periods, by performing molecular and serologic surveillance in wild birds and focusing efforts on the isolation of infectious WNV from avian samples.
Phylogenetic analysis of our strain revealed a high sequence similarity with Austrian and Hungarian WNV strains detected in previous years in birds of prey (2004)(2005)(2006)(2007)(2008)(2009).According to these findings, it can be hypothesised that the virus expanded from northern Europe southwards.The area of the recent outbreak is a well-known resting and breeding ground for migratory birds passing on the way from nesting grounds in Europe to wintering areas in Africa.Re-introduction of the virus in the future by birds migrating along the south-eastern migration route that leads from Europe and western Asia to Africa should also be considered possible and needs further investigation.

Figure 2
Phylogenetic tree of West Nile Virus strains based on nt sequences of the NS5 genomic region The sequence from the present study is shown in bold.
The sequences used to derive the phylogenetic tree were 797 nt long.At the Swedish Institute for Communicable Disease Control, statistical models based on queries submitted to a Swedish medical website are used as a complement to the regular influenza surveillance.The models have previously been shown to perform well for seasonal influenza.The purpose of the present study was to evaluate the performance of the statistical models in the context of the influenza A(H1N1)2009 pandemic, a period when many factors, for example the media, could have influenced people's search behaviour on the Internet and consequently the performance of the models.Our evaluation indicates consistent good reliability for the statistical models also during the pandemic.When compared to Google Flu Trends for

Introduction
For this paper, we evaluated the performance of statistical estimates of influenza impact based on queries made on a national medical website.The statistical models were trained on data collected during almost four influenza seasons and were applied to web query data collected during the influenza A(H1N1) pandemic period, since April 2009.Our evaluation concerned both the estimates produced by the web query-based system and their usefulness.
Monitoring of an influenza pandemic relies on a number of surveillance sources.Traditionally, the two main variables collected are the number of laboratoryconfirmed cases and the percentage of patients with influenza-like illness among total visits to appointed sentinel general practitioners.These are two standardised influenza surveillance measures recommended by the World Health Organization and the European Centre for Disease Prevention and Control [1,2].In addition, other sources are used, both formal and informal.In recent years, surveillance based on search behaviour on the Internet has appeared as a potential complement to the traditional sources [3][4][5][6][7][8][9][10]. As the conclusions drawn about the spread and the impact of a pandemic influenza will (or at least should) affect policy makers, it is crucial to evaluate the performance of such additional surveillance methods.
We have previously described a syndromic surveillance system [7] for seasonal influenza which is built on anonymous queries submitted to the search engine of a Swedish medical website: http://www.vardguiden.se.The Vårdguiden website had about 1.2 million visits in January 2010, of which approximately 800,000 were unique.The site is operated by Stockholm county council and around half of the visitors in 2010 originated from the Stockholm region [11] which covers about one fifth of the 9.3 million inhabitants in Sweden.The number of Internet users in Sweden is high: 88% of the population aged 16 to 74 years used the Internet on at least a weekly basis in 2010 [12].During the first quarter of 2009, 36% of the users in Sweden looked for health-related information on the Internet [13].
Our statistical models estimate the influenza burden in Sweden [14] and are trained to approximate the number of laboratory-confirmed cases of influenza and the proportion of patients with influenza-like illness reported by sentinel general practitioners.These estimates are based solely on the number of queries about influenza and influenza symptoms (in total 20 types of queries [7]) submitted to the Vårdguiden search engine.The statistical method behind the models has been described in Hulth et al. [7].The system, which generates a final output in the form of graphs, is fully automatic, including daily transfer of query logs from the medical website to the Swedish Institute for Communicable Disease Control (SMI), statistical calculations, and weekly emails presenting the output of the models that are sent to those in charge of the influenza surveillance at the institute.The email contains two graphs showing the estimated number of laboratoryconfirmed cases and the percentage of patients with influenza-like illness from week 16 in 2009 up to the week before the email is sent.An example of what data are contained in the output for the sentinel model is shown in Figure 1.Panel A shows the estimates for the percentage of patients with influenza-like illness calculated from the web queries.Panel B shows the number of media articles in Sweden on influenza, aggregated by week.Because of a reporting delay in the sentinel data, the automatic email can as soon as a week has ended give an estimate of what the traditional system will show only several days later.
In addition to the automatic emails, the graphs are published every week (starting week 36 in 2009) on a publicly available web page [15].The graphs were also discussed, together with information from a number of other sources, at weekly influenza meetings held at SMI during the most intense phase of pandemic surveillance in 2009/10.By comparing and contrasting the results from the different systems, the epidemiologists got a more complete picture of the spread and the extent of influenza activity in the population.
We have previously shown that the statistical models are able to estimate seasonal influenza [7,16].The purpose of the presented study was to evaluate the performance of the statistical models in the context of the influenza A(H1N1)2009 pandemic.This was a period during which many factors -for example the mediacould have influenced people's Internet search behaviour and consequently the models' performance.

Methods
In order to evaluate the performance of the web querybased influenza surveillance system, we performed one qualitative evaluation and two quantitative analyses.
The qualitative evaluation consisted of a structured interview with key persons who received the output of the statistical models for use as one source of information on the spread of the influenza pandemic in the country.In the quantitative evaluation, we focused on the performance of the sentinel model, as the traditional laboratory reporting indicated exceptionally high influenza levels, far higher than any of the other surveillance systems and did probably not correctly reflect the influenza impact in Sweden [17].Here we compared the output from the web query model to the reference data produced by the traditional surveillance, focusing on the potential advantage of the model output with respect to reporting delays in the sentinel data.In a second analysis, we compared our estimates to those made by Google Flu Trends for Sweden [18].

Evaluating the usefulness of the output
An email was sent to five persons at SMI who were deeply involved in the surveillance and the analysis of the spread and the impact of the influenza A(H1N1)2009 virus on the national level.The email contained five questions on the usefulness of the web query-based influenza surveillance, concerning the information conveyed in the graphs as well as the means through which it was distributed.We also asked the users to suggest improvements that could be made to the system.We obtained replies to this email from four persons.

Reporting delays
The sentinel reporting system suffers from reporting delays, since it relies heavily on manual reporting.The reporting delay for the sentinel data for seasonal influenza is up to three weeks during the influenza season, and can be up to five weeks in the beginning of a season [19].
Our  1. the root mean squared error of prediction (RMSEP).This is one of the standard measures in model evaluation [20], calculated by 2. the mean absolute deviation (MAD).This value is calculated by The advantage of the latter measure is that it is slightly more intuitive than RMSEP, since it tells us how far off the predictions were on average.
We also calculated the R-squared measure, as well as the correlation coefficient.

Comparison with Google Flu Trends
Google Flu Trends was launched for Sweden in October 2009 [18].In this analysis, we compared the estimates done by Google Flu Trends, which were based on queries submitted from Sweden to the general-purpose search engine Google, to those made by our system based on queries submitted to the national Vårdguiden web site.More specifically, since Google Flu Trends was developed on sentinel data, we used the web query-based sentinel data for the comparison.Both sources aggregate data by week, although Google Flu Trends starts the week with a Sunday, whereas our statistical model starts the week with a Monday.

Usefulness
According to the users of the output produced by the web query-based system, the largest contribution of the graphs was as an additional source and a complement to the traditional surveillance.It was stated that one surveillance system is not enough for getting a true picture, and the more sources point in the same direction, the more reliable is the interpretation of the influenza surveillance data.The automatic dispatch was much appreciated and the emails, sent three and a half days before the time when the traditional surveillance was compiled, was valuable as an early signal of what to expect from the traditional surveillance, although it was the trend rather than the height of the curve that was deemed more important.
As part of the graphs produced by the web query-based system, the crude numbers of articles on influenza in online media (obtained from http://www.eniro.se/nyhetssok/) were plotted.The users appreciated that some indication of the media activity was shown in the graphs.It was, however, evident from the answers that we obtained that some of the users believed that this information was corrected for in the statistical estimates.
Two improvements were suggested to the models: that they should be corrected for the impact of media reports on search behaviour; and that they should be divided into the various regions of the country.This latter wish is, however, impossible to fulfil with this particular data source, as no geographical information is stored in the anonymous query logs.One user requested a better explanation of the model's statistics.

Summary evaluation statistics
In Table 1 we summarise the comparison of our model and Google Flu Trends with the actual sentinel reports.The Swedish sentinel model based on web queries predicted the sentinel numbers better when delayed reporting was taken into account, no matter what performance indicator was used.This makes sense because we trained the models on complete sentinel data.In other words we have, by training the models on data including late reports, obtained a system which better mimics the values we will get after a while, once the data have been back-populated.
The MAD value of 0.15 can be compared with the change from 1.11 percentage points to 1.36 percentage points between week 45 and 46 in 2009 [21,22], during the height of the pandemic.Thus, the average deviation paralleled the weekly change during the most intense pandemic period.Although the difference is small, the correlation coefficient indicates that our model performed better than Google Flu Trends for Sweden, with a correlation coefficient of 0.90 versus 0.87.Since Google Flu Trends only provides relative intensity indicators and not absolute estimates of reported influenza, R-squared, RMSEP and MAD could not be calculated for their data.

In-depth comparison with Google Flu Trends for Sweden
Figure 2 shows reported sentinel data (incomplete and complete), our web query-based estimations for sentinel data, and Google Flu Trends for Sweden from week 16 in 2009 to week 6 in 2010.When comparing the output of the sentinel model to the traditional surveillance that the model is supposed to mimic (Figures 1 and 2), we can see that the shape of the curves is very similar.The Google data, which are based on more data than the Vårdguiden data, form a smoother curve compared with the output from our statistical model.It underestimates, however, the height of the peak in November 2009.

Comparison with other published results
The performance of the web query-based sentinel model during the pandemic season in terms of correlation estimates was in line with the performances of various other reported attempts of web-based influenza surveillance (Table 2).The correlation values that have been published are in the region of 0.72-0.94,with one bottom outlier at 0.55 using blog posts [10], and Google's exceptional outlier at 0.96 [6].The latter is especially surprising given that this value was for correlation with validation data.We found two publications that reported R-square estimates: Eysenbach reported an R-squared value of 0.83 [4], and Polgreen et al. reported an R-squared value of 0.38 [5], but it has to be noted that these were the values obtained when comparing the model to the data used for the fitting process.The R-squared value in our sentinel model denotes the performance relative to previously unknown data, during an exceptional influenza season.In light of this, the estimate of 0.75 is high.

Discussion
Overall, the performance of the statistical models based on queries submitted to the Swedish Vårdguiden web site exceeded our expectations during the pandemic, especially because the models were trained on seasonal influenza.The curve produced by the web query-based sentinel model was very similar to the one obtained from the traditional surveillance the model is supposed to mimic.
We have shown that an independently developed and controlled system such as ours can be comparable in reliability to Google Flu Trends, a model that is trained on much larger data volumes.One downside is that our model has a higher variance, which becomes manifest in numerous small fluctuations of the model estimates in Figure 2, trend shifts that are not reflected in the reported sentinel data.Such false signals can be a cause for concern if the model is to be used to guide public health action, and means in practice that observed trend shifts cannot be trusted unless sustained for two weeks or more.
While others have indicated that the under-estimation of the influenza peak in Sweden of Google Flu Trends could be due to a limitation in the Swedish sentinel system [23], the fact that our model (in addition to other surveillance methods) shows the same pattern as the sentinel reports [17], rather indicates that it is Google Flu Trends that is lacking in the quantitative estimation.
The quantitative evaluation statistics also indicate good reliability.It is debatable, however, whether they are suitable for evaluating surveillance systems for communicable diseases.Such measurements tend to investigate the performance in estimating absolute levels of activity, and give equal weight to the entire period of investigation, including periods of low activity.In future work, it might be more important to look  at how a surveillance system captures the dynamics of the disease, such as rapid increases in activity levels or the timing of peaks.
We have also described the results of a qualitative evaluation in which we interviewed four colleagues who were receiving the output from the statistical models.In summary, it was valuable for those working with the surveillance to have an additional source of information, as this increased their confidence in their estimates and predictions of the spread and the impact of the influenza A(H1N1)2009 virus.
One unknown factor here is the media impact on search behaviour.The interviewees explicitly asked for media activity to be incorporated in the statistical model.Such a model should intuitively perform better than a model without this information.We have performed some early experiments on including media activity in our web query-based statistical models.However, we have not yet found a satisfactory model to correct for the assumed impact of media reporting on peoples' search behaviour.

Conclusions
In this paper, we have described an evaluation of a syndromic surveillance system based on queries submitted to the search engine on a Swedish medical website and regularly used during the pandemic influenza period.From our experience, we can say that there are a number of advantages of using web queries as a source for surveillance during a pandemic: • The system is fully automatic; • The estimates are produced earlier than the traditional sources that it is supposed to mimic; • They do not require people to see a doctor; • There is no reporting delay in the system; • The system is cheap to maintain; • A system based on web queries can easily be adapted to different symptoms or diagnoses.
In addition, the presented analyses demonstrated that the system is reliable, stable and performs well when compared with conventional surveillance systems.
When comparing the output from our sentinel model to Google Flu Trends for Sweden, we can conclude that although our models had been trained on a substantially smaller set of data, they were at least equivalent to Google Flu Trends in terms of performance, and in terms of peak estimation even seemed to be more precise.
No current method can, however, give us the true spread and impact of an infectious disease in society.Until such a method is invented, the best we can do is to use multiple sources for surveillance, be it an influenza pandemic or another infectious disease.Syndromic surveillance based on web search behaviour clearly has a role to play as such a source.In 2004, the German public health institute, the Robert Koch Institute (RKI), prioritised pathogens by public health criteria and presented the methodology and findings.In order to further improve the methodology, the RKI invited experts to give feedback on this via a structured web-based questionnaire.The survey was completed by 72 participants during 15 July 2008 to 15 January 2009.Prioritisation of pathogens was considered as useful for public health purposes by 68 participants and for both surveillance and epidemiological research by 64 participants.Additional pathogens were suggested, including some that are resistant to antimicrobials.The criteria incidence, severity, outbreak potential, emerging potential and preventability were each considered as useful or very useful for the prioritisation (by more than 65 participants for each criterion).Weighting of the criteria was judged as relevant or very relevant by 67 of participants, but needs more explanation.It was also suggested that the group carrying out the prioritisation be composed of a median of 15 experts (range: 5-1,000).The feedback obtained in the survey has been taken into account in the modification of the methodology for the next round of prioritisation, which started in December 2010.

Background
Strengthening communicable disease surveillance and response at national level requires a substantial and long-term commitment of human, financial and material resources.The usefulness of prioritisation as part of this process, irrespective of the methodology used, has been demonstrated by several research groups [1][2][3][4][5][6][7].This investment begins ideally with a systematic review of the national priorities for surveillance [8,9].In 2004, the Department for Infectious Disease Epidemiology of the Robert Koch Institute (RKI), the German national public health institute in the portfolio of the federal Ministry of Health, initiated an exercise on prioritising various pathogens to guide the research and surveillance strategies of the department.After a literature review, we developed a methodology, including a scoring system for 12 criteria for selected pathogens.For each criterion, a three-tiered score (-1, 0 and +1) was used.Independently, each criterion was weighted: a group of experts ranked the 12 criteria in terms of perceived importance.A mean value was calculated for each criterion (its weight), by which the score of the criteria was to be multiplied.The total weighted scores led to a ranked list of 85 pathogens.Initial findings were presented at three international scientific conferences in 2006 and 2007 [10][11][12] and were covered in a national non-scientific magazine [13]: this generated public interest and feedback from scientists and patient advocacy groups.
A review of previous prioritisations strategies used by others and details of the methodological approach we used (Figure ) were subsequently published in 2008 [14][15][16].A review and possible revision of our approach is part of the methodology.This current process is described in this paper.
To refine the prioritisation methodology further, develop it into a standard tool and ensure that it is fully understandable, an open call was issued, inviting respondents to complete an online structured questionnaire on the prioritisation methodology and relevance of the prioritisation tool.In addition, we targeted representatives of the scientific community as well as health policy stakeholders.
This paper presents the findings of the survey and discusses their potential implications for the planned modification of the methodology of prioritising pathogens.

Survey approach
When we published extensive descriptions of the prioritisation methodology [14][15][16], we invited readers to give feedback and comments through an online questionnaire.Additionally, we contacted by email all German regional epidemiologists (n=60), all members and alternates of the scientific Advisory Forum (n=64) of the European Centre for Disease Prevention and Control (ECDC), all heads of the German national reference laboratories (n=66) and all members of the Committee for Epidemiology of Infections (n=12) and four relevant German epidemiological societies and associations, asking them to take part in the online survey.
The online survey contained the list of the 85 selected pathogens and the 12 criteria used in the prioritisation, with questions on the usefulness and appropriateness of these criteria.In order to compare the participants' feedback on the criteria, we gave a numerical value to each possible answer and calculated the mean value for each criterion.To assess the usefulness of the criteria, the possible answers were: very useful (with a value of 3), useful (value of 2) and dispensable (value of 1).
The survey contained additional questions on the number and profession of experts that participants considered should take part in a prioritisation process and also questions about the participants themselves.
The questionnaire was internally pretested and then posted in both English and German on the RKI home page from 15 July 2008 to 15 January 2009.The data were analysed using Epi Info software.

Survey findings Participants
In total, 72 participants completed the survey.Most (n=35) found out about the it from the national epidemiological bulletin, 18 received the email request sent by the RKI, 11 read about it in Eurosurveillance [15] and eight found it coincidentally on the Internet.
Of the 72 experts, 54 were working in Germany, nine in other European Union (EU) countries and six in non-EU countries.For three respondents, no information on the country in which they worked was available.
The participants had a variety of professions and institutional affiliations, with the majority being medical doctors by training (Table 1).
Almost all participants (n=68) provided information on the length of their work experience: the median duration was 18 years (range: 3-40 years).

Feedback and comments
Prioritising pathogens was considered useful for public health purposes by 68 participants, for both surveillance and epidemiological research by 64, and for clinical research by 57.Most respondents considered prioritisation to be beneficial for public health services, at the national (n=58) and international (n=49) level.Additionally, 33 participants believed that the prioritisation will also be useful for regional public health services, universities and ministries of health, to guide surveillance and research agendas.A total of 29 participants considered that it would be beneficial to local public health services.
Most participants (n=40) considered that the list of 85 pathogens [15] was comprehensive and appropriate, Adapted from [15].while16 proposed changes to the list; 16 answered that they did not know.The following additional pathogens and topics were suggested:

Figure
• all Brucella spp.• all Campylobacter spp.Seven participants suggested including pathogens resistant to antimicrobials as a separate group (e.g.bacteria producing extended-spectrum beta-lactamase, vancomycin-resistant enterococci and oxacillinresistant S. aureus).

Prioritisation criteria
Definitions of the scores for each criterion are described elsewhere [15].Table 2 describes the respondents' rating of the usefulness of the prioritisation criteria, by their profession or institutional affiliation.

Incidence
Incidence was judged by 68 participants as a very useful or useful criterion for prioritisation.The comments received mainly reflected the difficulty in getting adequate data on incidence, especially for diseases that are not notifiable.One suggestion was to include 'unknown' in the highest score of the criterion, to indicate that the level of attention should be high if information is lacking.

Severity
This criterion was considered to be useful or very useful by 68 participants.Comments referred to the difficulty of incorporating different issues such as hospitalisation, work-time lost due to sick leave and persisting disabilities into one single criterion.Furthermore, the issue was raised of how work-time lost due to sick leave can be judged if children, unemployed and retired people are concerned.It was also suggested that cost of medical care be included as an additional aspect.

Mortality
A total of 62 participants thought this a useful or very useful criterion.One respondent suggested that lifeyears lost be used instead of mortality for diseases that affect children more than adults.The scarcity of reliable data sources to score this criterion was a concern expressed by three of the participants.
As replacing the mortality criterion with case fatality rate had already been suggested in the prioritisation process in 2004 -as mortality is influenced by incidence (a separate criterion) -we asked in our survey whether case fatality rate should be used instead.A total of 33 participants recommended the replacement, 17 preferred mortality, while nine could not see a difference and 13 did not have an opinion.Five participants were in favour of including both criteria.

Outbreak potential
This criterion was considered by 69 participants as useful or very useful.Two participants suggested using the basic reproductive rate (R 0 ) of a pathogen, rather than the frequency of outbreaks, to judge outbreak potential.A fixed threshold of five or more cases per outbreak for all pathogens was questioned by three participants.

Trend
A total of 57 participants considered this as a useful or very useful criterion.However, for 12 respondents the definitions used for each score were not clear enough.They suggested that a timescale for the trend should be determined.Questions were also raised on how to score diseases with unclear trends.

Emerging potential
This criterion was judged by 65 of the participants as useful or very useful.Five considered that endemicity and a low probability of the disease being introduced should not be included in the same score.Additionally, inclusion of the emergence of pathogen strains resistant to antimicrobials as a separate aspect of the definition of the highest score was proposed.It was also suggested that this criterion should be combined with the trend criterion.

Evidence for risk factors/groups
A total of 62 respondents judged this criterion as useful or very useful.A clear definition of the kind and quality of 'scientific evidence' was requested by some participants.It was also suggested that this criterion be combined with the evidence for pathogenesis criterion, to cover transmission routes and pathogenesis.
Two respondents questioned whether existing scientific evidence should be part of the prioritisation approach, as it leads to conflation of the relevance of a disease for public health and knowledge of the disease.These two aspects are important, but should be judged independently.

Validity of epidemiologic information
This criterion was judged by 62 participants as useful or very useful.Here the definition of the score 0 ('epidemiologic information exists but is scientifically not very valid') was considered imprecise.The applicability of this criterion and the lack of reliable data that are needed to score it were raised as concerns.

International duties and public attention
A total of 52 participants thought this a useful or very useful criterion.However, the definitions were not clear and as several aspects are included in each definition, some participants indicated that it is problematic to assign a single score in situations when separate aspects should be scored differently.They also thought it hard for the scoring to take into account rapidly occurring changes in public or political attention.

Evidence for pathogenesis
This criterion was considered by 57 of the participants as useful or very useful.The problem of assessing different aspects of the criterion using a single score was raised again.Combination of this criterion with the evidence on risk factors/groups criterion was suggested.

Preventability
In total, 67 respondents judged this as a very useful or useful criterion.The task of scoring the availability of prevention measures and need for further research in a single criterion was criticised.It was also suggested that availability of an effective vaccine be included as a separate criterion.

Treatability
This criterion was deemed by 61 respondents as useful or very useful.The distinction between the definitions of the three scores was not clear to some participants and might need clarification.The issue of incorporating drug resistance into the prioritisation was raised again.One participant suggested merging preventability, treatability and severity into a single criterion.

Suggestions for additional criteria
Participants suggested that the prioritisation tool include assessment of the economic impact of a disease or its control measures, the concept of life-years saved or lost, emergence of antimicrobial resistance and monitoring of vaccination effects, for example, on incidence or pathogenicity.

Scoring system
A total of 54 participants found the three-tiered scoring system to be adequate; six would have preferred a twotiered and four a five-tiered system.Five suggested introducing a more continuous scoring (e.g. from low to high, on a scale from 1 to 10), whenever possible.

Weighting process
The weighting process was judged by 49 participants as very relevant and by 18 as relevant.Two thought it irrelevant and three did not know.The weighting method was considered plausible but initially difficult to understand by 31 participants, 19 understood the weighting method immediately and for 13 it remained unclear.Some respondents supported the separation of the weighting from the actual prioritisation.
One participant pointed out that basing the numerical value of the weighting on the ranking of the criteria may result in bias, as it assumes that the difference in importance between each criterion in the ranked list is always equal.We therefore suggest that values between 1 and 10 be used instead for the weighting, without any ranking.
The need for a better description of the weighting process was highlighted by two participants.

Size and composition of an expert group for prioritisation
The participants proposed that the median size of an expert group needed to conduct the prioritisation exercise of surveillance and research activities of a national public health agency was 15 (range: 5-1,000).They suggested that experts representing the following professions or institutions should take part in future exercise rounds: national public health service (suggested by 65 participants), university faculty of infectious diseases (by 59), microbiologist (by 57), hospital epidemiologist or hygienist (by 51), international public health organisations (by 48), regional public health service (by 47), hospital physician (by 45) and local public health service (by 37).Two respondents suggested that health economists be involved.

Conclusions
Setting priorities in research can serve as a catalyst for public debate and create networks of stakeholders [4,17].The opinion of the user of the prioritisation is very important, as exchanging experiences and discussing the topics with the various stakeholders is highly relevant [3,18].Indeed, Lomas et al. stated, when describing prioritisation efforts, 'The process is more important than the science' [19].Our survey was one step in involving various stakeholders and proved very useful in helping to develop our prioritisation methodology further, even if the set up of the survey was neither able nor intended to be representative.As the survey was announced in an open call and as some email requests were sent to generic email addresses, we have no information about the denominator and are therefore unable to calculate the response rate.Given the survey design, it is also impossible to tell whether the opinions of those who responded were representative.It is possible that those who chose to take part in the survey were those who were relatively positive about the prioritisation process.However, even if that were the case, they provided constructive criticism and comments, which have helped us to improve our methodology.
Overall, the participants commented positively on the prioritisation methodology: although there was variation between the responses of participants with different professions and institutional affiliations, the proposed criteria were mostly considered useful.However, it became clear that the definitions of some criteria were unclear for scoring purposes.We will therefore try to clarify the problematic definitions.

Which pathogens should be included?
The suggested list of 85 pathogens was seen as fairly comprehensive by most participants.However, given the recommendations, we realised that some additional pathogens could be included in future, as their importance has changed since the list was drawn up in 2004.

How should the prioritisation take into account antimicrobial resistance and emerging diseases?
Interestingly, antimicrobial resistance was mentioned at various points in the survey as an essential issue that should be addressed.We believe it can be sufficiently accounted for if it is an integral part of the criterion of treatability and we therefore propose that it be included in its definition.
Participants also questioned how an endemic disease could be scored in the same way as a disease that is unlikely to emerge.We believe this to be justified, as an endemic disease has generally already led to an established infrastructure for prevention, surveillance, diagnosis and treatment.Similarly, diseases that are not endemic and are very unlikely to emerge in a country in near future should probably not be considered a priority when resources are limited.A disease with potential to emerge generates new challenges and thus deserves special attention, at least for prevention and surveillance.

How should disease severity be assessed?
One of the issues raised in various ways throughout the survey was the challenge of adequately accounting for the severity of an illness resulting from an infectious disease.Participants suggested that the prioritisation should take into account other aspects of disease severity, such as the economic impact of an illness, life-years lost, the effect of work-time lost due to sick leave if children, unemployed and retired people are concerned, and the cost of care.However, including the requirement for such detailed information might increase the problem of lack of relevant data, resulting in difficulty in scoring this criterion, as discussed above.Our original approach intentionally attempted to keep the score definitions within each criterion as simple as possible.We will, however, take those issues into consideration when redefining these definitions.
Detailed instructions concerning the process of assigning a single score to a multicategory criterion will be developed and provided during the next prioritisation.

How should the prioritisation take into account variability of incidence trends and outbreak potential?
Some participants drew attention to the fact that a time frame would be needed for the scoring of some criteria (e.g.trend or emerging potential).We consider that it would depend on how frequent the prioritisation exercise is planned to be repeated and what its main objective is.For example, a disease with a highly variable incidence from one year to another should probably have a high score for outbreak potential, while the scoring for incidence should probably be based on some sort of average yearly figure for the previous five or 10 years.Furthermore, if recent observations indicate that despite observed fluctuations yearly incidence tends to increase, it should be appropriately accounted for in the trend criterion.
The fixed threshold of five cases or more per outbreak for all pathogens was questioned by some participants.The underlying rational for the threshold was that in Germany, only a few households have five or more members, suggesting that most outbreaks of less than five cases are likely to be limited to one household.Such outbreaks have fewer implications for public health services, as opposed to larger outbreaks.Obviously this distinction may be more appropriate for common gastrointestinal pathogens, which are responsible for the vast majority of all outbreaks.However, for practical reasons we decided to use this threshold for all diseases.
Country of infection not yet published on the national level

Figure 1
Figure 1 Bird sampling area for West Nile Virus, Central Macedonia, Northern Greece, 2009-2011

Figure 1
Figure 1Week-by-week web query-based estimates of the percentage of patients with influenza-like illness among all patients seen, Sweden, week16, 2009-week 19, 2010

Table 1
Evaluation statistics for models predicting influenza burden based on Internet queries, Sweden, 2009/10 NA: not available.

Table 2
Reported performances of different web-based influenza surveillance systems

Table 1
Employment information and education level of prioritisation survey participants, Robert Koch Institute, 15 July 2008 to 15 January 2009 (n=72) a More than one answer allowed.

Table 2
Usefulness scores of the prioritisation criteria by survey participants' profession or institutional affiliation, Robert Koch Institute, 15 July 2008 to 15 January 2009 (n=72)