Comparing and ranking outcomes

310

The comparative nature of assessments means that, in most cases, results from a number of scenarios need to be compared, in order to determine their relative tolerability or pereferability, so that policy choices can be made.  It is not the role of the assessors to make these choices, but at the very least the results of the assessment need to be made available in a form that enables comparison.  In some cases, assessors will also need to contribute to, or carry out, the actual comparison either with or on behalf of the users.  An understanding of how the comparison will be done, and what forms of information are needed for this purpose, is therefore crucial. 

This becomes all the more important when the issues being considered are complex, and when a large range of impacts are involved, for in these circumstances the process of comparison may be difficult.   Three specific challenges are likely to occur:

  1. How to assess the significance (in statistical terms) of apparent differences between the impacts of different scenarios, in the light of the inherent uncertainties;
  2. How to balance and make trades-off between different outcomes (or areas or population groups), which may show contradictory impacts;
  3. How to enable the comparisons to be done in a way that involves, and gives fair consideration to, the different stakeholders concerned

Formulaic answers to these difficulties are unlikely to be universally applicable.  For example, the significance of the apparent impacts can in some cases be tested using standard statistical techniques, but these depend on quantitative understanding of the uncertainties involved and are only valid where the impacts themselves have been measured in quantitative terms.  Likewise, the issue of trades-off can (to some extent) be avoided by generating and using aggregate measures of impact (e.g. DALYs or monetary value), but in reality these merely internalise some of the value judgements that need to be made (e.g. the weights attached to different health outcomes), and still leave open the question of who wins and who loses (in relative if not absolute terms), and how such distributional issues should be dealt with.  In principle, also, stakeholders can freely contribute to the processes of comparison and ranking, but differences in expertise and practical constraints of time and finance often mean that this is an imperfect process.   

Different approaches to comparing and ranking the outcomes may therefore need to be applied, depending on the nature of the issue and the needs of users (and the underlying value systems).   For example:

  1. Cost-benefit analysis (comparing the monetary value of total costs and benefits) can be used, where it is agreed that all relevant aspects of the impacts can be converted to monetray values, and where an essentially utilitarian approach has been adopted;
  2. Cost-effectiveness analysis (minimising the cost to achieve a specified health gain, or avoid a specified health loss) can be used where costs can be monetised but where benefits cannot be, or where a clear health objective exists;
  3. Mutli-criteria assessments can be carried out, where costs and benefits cannot be monetised, where monetisation is considered by be incomplete (or unacceptable), and where several different outcomes have to be considered. 

Information on each of these methods is gven via the links in the panel to the left.