07 July 2011
Letter to the editor: Prioritisation of infectious diseases in public health: feedback on the prioritisation methodology, 15 July 2008 to 15 January 2009
To the editor: We read with great interest the results presented by Gilsdorf and Krause (2011)  of a survey to experts on the methodology used by Krause et al. (2008)  to prioritise 85 pathogens of public health importance. Their work deals with a very relevant subject, given current pressure on health budgets: the allocation of finite disease surveillance and control resources among competing alternatives, infectious diseases in this case. The authors correctly identify the evaluation as being multi-dimensional and compensatory. Unfortunately, they appear to have overlooked findings and principles of well-established methodologies for assessing the impact of multiple effects on non-tradable goods, such as multi-criteria decision analysis (MCDA) . It is within this perspective that we make our comments.
First, we noted that some of the criteria considered by the authors do not exhibit certain essential properties of evaluation criteria, particularly with regards to preferential independence. For example, it seems difficult to assess the “treatability” of a pathogen without considering simultaneously the “evidence for pathogenesis”. We observed that lack of preferential independence may also exist between other criteria within the groups “Information needed” and “Health gain opportunity”. When criteria are not preferentially independent, the use of linear additive models of weighted pathogen-specific scores, as in Krause et al. (2008), should not be used, as the overall impact cannot be assessed by simply adding up partial impacts. The criteria set should be redefined to make sure the essential properties hold and a simple weighted sum may be employed .
Second, the score’s scale, -1, 0 or 1, lacks granularity and discretises continuous variables unnecessarily. Under the current model, a disease with an incidence of 20/100,000 would score “0” whereas as a disease with incidence of 20.1/100,000 would score ”1”. As suggested by some of the respondents to the survey, a continuous score is better suited. This issue is easily dealt with in MCDA, with the assessment of value functions, which map out and normalise different levels of impact into 0-100 scales.
The score “0” holds a double label: “average importance” and “lack of knowledge”. The ambiguity of criteria labels is something to avoid in all prioritisation exercises. In this particular application, two diseases, one well known but “average” and the other suffering from lack of evidence, could score similarly. This would not help in the ranking of diseases and the subsequent distribution of resources that would probably allocate greater relevance to the unknown than to the average known. The survey respondents were rightly concerned about the need to incorporate uncertainty in their assessment against a number of criteria. To this request, the authors argued that the complexity of such addition to their model may outweigh the benefits. We would just like to add that simple approaches to handling uncertainty in decision frameworks similar to this are already available and widely used and that uncertainty about impacts should not increase the ambiguity in assessments.
Third, when allocating the weights to the criteria, the authors failed to recognise that weights are scaling constants, which aggregate partial impacts into overall impacts, and not direct measurements of importance. Indeed, a survey respondent correctly identified the limitations associated with this approach to weighting “that the difference in importance between each criterion is always equal”. This could lead to misleading conclusions. Weights should reflect explicitly public value trade-offs of the group involved in model building and their assessment has to follow careful elicitation procedures to avoid well-known biases .
Fourth, a number of survey respondents raised the need for a time frame for some of the criteria. This is a very genuine concern that should be expanded to include all the characteristics that define the context (e.g. geographical location). In our experience, prioritisation of diseases is a valid exercise that allows systematic comparisons to support strategic resource allocation. Like any other general strategy, it will fail to capture all possible presentations and heterogeneities that will surely be present depending on the risk pathways involved. Alternative methods to reactively measure the impact of such variability are required to feed into the regular strategic prioritisation of diseases. MCDA has been successfully used in these contexts and, in our view, provides a robust methodological framework for such evaluations .
We would like to finish our note with a comment on the composition of the expert group for prioritisation and congratulate the authors for engaging with a wide group of technical experts. If we may, we would like to suggest that the authors consider the incorporation of experts on MCDA to this group. This would follow common practice in other scientific fields such as nuclear waste management and drugs risk-benefit assessments.
- Gilsdorf A, Krause G. Prioritisation of infectious diseases in public health: feedback on the prioritisation methodology, 15 July 2008 to 15 January 2009. Euro Surveill. 2011; 16(18):pii=19861. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19861
- Krause G; Working group on prioritisation at the Robert Koch Institute. Prioritisation of infectious disease in public health – call for comments. Euro Surveill. 2008; 13(40):pii=18996. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=18996
- Montibeller G, Franco LA, Del Rio Vilas VJ. Prioritising emerging threats with multi-criteria decision analysis. Proceedings of the 21st International Conference on Multi-Criteria Decision Making; 2011 Jun 13-17, Jyvaskyla, Finland. Forthcoming.
- Keeney RL. Common mistakes in making value trade-offs. Operations Research 2002;50(6):935-45.
- Keeney RL, Raiffa H. Decisions with multiple objectives: preferences and value tradeoffs, 2nd ed. Cambridge: Cambridge University Press; 1993.