Introduction

Environmental health assessments are endeavors of producing science-based support to societal decision making upon issues related to environment and health. They produce information for specific needs thus making the information objects produced in assessments intentional artifacts; means to ends. There are two common views to assessment performance. The uncertainty approach focuses on the intrinsic properties of the information produced in assessment. The quality assurance/quality control approach focuses on the process of producing that information. However, in order to capture all factors that contribute to the overall assessment performance, it is necessary to consider not only both the production process and the information produced, but also the process of using that information. The properties of good assessment consider assessment performance as a function of (i) the quality of information content (ii) applicability of information, and (iii) the efficiency of the assessment process. These property categories are further broken down into eight individual properties that jointly contribute to overall assessment performance: informativeness, calibration, relevance, pertinence, availability, usability, acceptability, intra-assessment efficiency, and inter-assessment efficiency. As such, the properties of good assessment simultaneously address features of: (a) production of information, (b) the information content, (c) the information product, (d) information in use, and (e) characteristics of use context. The properties of good assessment can be used in ex-post evaluation of assessments, but it is most useful in ex ante evaluation during the design and execution phases of assessments.

Performance of assessment

The fundamental purpose of assessment is to improve societal decision making. Assessment should thus provide relevant information about the situation that the decision making is about, most often preferably in a quantitative form if possible. In general the information is useful to be formulated as predictions on the impacts of possible decision options on certain outcomes that are of societal importance. All assessments should always be done according to a specified information need in a decision making situation. When the purpose is identified and kept clear in mind and preferably publicly explicated, it helps to guide the assessment process in producing a desired kind of assessment product that helps in making good decisions. Explicit definition of the assessment purpose, preferably early on in the process, is an essential and integral part of assessments.

The overall purpose of risk assessment can be considered as composing of two different aspects. The general purpose is to describe reality, i.e. to explicate real-world phenomena in a comprehensible and usable form. But mere describing reality without any specific, identified need for the description would probably not make much sense as such. There must also be a certain specific purpose to undertake such a task, an instrumental purpose, or use purpose, for the output of the assessment. For each particular assessment, the specific situation, and thereby the instrumental purpose, is naturally different and thus the outputs of each assessment are and should always be case-specific and in accordance with the contextual setting of the particular case.

The performance, or the goodness, of assessment means how well it fulfills its purpose, or actually its both purposes. Assessments produce pieces of information, artifacts, that are intended for use in a certain setting. The performance of assessment is a property of a piece of information in its use, and derived from its making. As practical means to practical ends, the performance of assessments can be considered according to their instrumental functions ^[1]:

• functional goal
• use plan
• contexts of use
• artifact type

In the context of environmental health assessment, all of these instrumental functions can be determined to some extent. he functional goal is to improve societal decision making, the artifact type is an assessment report, i.e. a description of a piece of reality, the context of use and use plan are often such that policy makers read the report and decide upon something according to the understanding they developed by internalizing the assessment information. I every single case, the situation is however somewhat special, and accordingly the instrumental functions to meet the needs are somewhat different. It can also be questioned whether the normal situations and conditions are such that they optimally promote fulfilling the purposes of assessment.

Approaches to assessment performance

There are two commonly applied approaches to assessment performance that can be referred to as 1) the quality assurance/quality control (QA/QC) approach, and 2) the uncertainty approach. This is a rough, but practical, simplification, and both of these typifications consist of several varying sub-approaches.

The QA/QC approach focuses on the procedure of making the assessment. To put it very briefly, it uses a well-defined procedure as a proxy for good quality of the outcome of the process. Another example of somewhat similar thinking is present in the ranking of different study types and the weight of evidence they produce, and usually concluding that randomized clinical trials produce the best evidence without actually considering what actually is the evidence they produce.

The uncertainty approach focuses on the properties of the assessment product, the piece of information produced in the assessment. The range of different specific approaches to uncertainty is very broad, but they tend to build on the objective properties of the information that are independent from both the making and the use of the information. The basis of the uncertainty approaches is in statistics, but many specific approaches also extend to considering qualitative aspects of uncertainty.

Properties of good assessment

Based on the above considerations, it is possible to define the general properties of good assessment. This approach considers assessments as compounds of sets of questions and hypotheses answers to them. It also addresses not only the assessment product, but also the process of producing it as well as the process of using it. The properties also have their implications that should be taken account of in regard to assessment methodologies. The general properties of good assessments are illustrated as a tree structure in the diagram below. The general goal, good assessment, is the node on the left of the diagram and properties needed to constitute this goal are broken down while moving to right in the diagram.

The properties of good assessment consist of two categories:

• Effectiveness of assessment product, consisting of
  • Quality of information content produced in assessment
  • Applicability of assessment product
• Efficiency of assessment process

The first two categories, quality of content and applicability of output together form the effectiveness of assessment product. The effectiveness means the potential of assessment to have the intended influence on the decision making processes that the assessment addresses. Effectiveness thus indicates the potential of advancing towards the primary purpose of assessment of improved societal decisions.

Quality of content refers to the goodness of the information content that is produced in the assessment. It consists of three properties: informativeness, calibration and relevance. The point of reference in informativeness and relevance is the reality that is described by the information. The point of reference in relevance is the question that the information is intended to address.

Informativeness can be considered as the tightness of spread in a distribution (All results estimates of variables should be considered as distribution estimates of some form rather than point estimates). The tighter the spread, the smaller the variance and the better the informativeness. Another way to describe informativeness, in particular for qualitative information, is how many different worlds does the information rule out.

Calibration means the correctness or exactness of information, i.e. how close is it to the real phenomenon it describes. Evaluation of calibration can often be complicated in many situations, but it is an important property complementing informativeness. In particular in qualitative terms, informativeness and calibration can be considered as constituting truthlikeness.

Relevance can be described as the coherence of the assessment, i.e. does the assessment address all necessary, and only necessary, aspects of the phenomena being assessed. Relevance is considered in terms of the assessment contents in relation to assessment scope.

Applicability refers to the potential of conveying the information content in the assessment product to its intended use(s). Applicability also consists of three properties: usability, availability and acceptability. Usability and availability are properties of the assessment product that are realized in the process of its use. Acceptability is fundamentally a property of the assessment process, also realized in the process of using the assessment product.

Pertinence means the relevance of the assessment results in relation to the use that they are intended for. It is considered in terms of the assessment scope in relation to the practical need for information.

Usability refers to issues that affect how the user manages to create understanding about the content, such as e.g. clarity of presentation, language used etc. Usability is strongly influenced also by the capabilities and other properties of the users and is often not fully controllable by those who produce the information. Usability can be addresses by explicit definition of the use purpose of the assessment and identifying the intended users and uses.

Availability refers to the openness of access by the intended users to use the assessment product when and where needed. Availability is affected by issues such as e.g. chosen media of representation, and openness of the assessment process.

Acceptability is particularly strongly influenced by the intrinsic properties of the acceptor. Fundamentally it is a question of accepting, or not accepting, the assessment process, how the information in the assessment was come up with.

Whereas the properties constituting effectiveness are primarily related to the assessment product, efficiency is clearly a property of the assessment process. Basically efficiency can be described as the amount of effectiveness (a function of quality of content and applicability) given the effort spent in producing that effectiveness. Efficiency can be considered in terms of a particular assessment or a whole endeavor of assessment practice producing a series of assessments.

Intra-assessment efficiency means the efficiency within a certain assessment, i.e effectiveness of a single assessment over the expenditure of efforts in carrying out that assessment.

Inter-assessment efficiency refers to the reduction rate of the marginal efforts needed for each new assessment with the same quality of output when making a series of assessments. This means in practice the ability to avoid doing the same work again if it has been already done in a previous assessment.

In can be considered that the properties related to quality of content are the most crucial ones. Assuring the goodness of questions and hypotheses as answers to them should thus be the first priority in assessment. Anyhow, the quality of content has most very little significance if the applicability of the assessment remains low. It is therefore also important to explicitly consider all aspects of effectiveness when designing and executing assessments. The importance of efficiency mainly comes from practical limitations and inevitably scarce resources for making assessments. It is necessary to strive for best possible outcomes with the available resources.

The properties of good assessment can be used as a framework for evaluating the performance of assessment. It can be applied in evaluation that takes place as a separate process aside the assessment process, or it can be applied as principles that guide the design and execution of the assessment. The properties of good assessment address the assessment product, but also the process of making it as well as using it. They also address the information produced in the assessment from different perspectives, goodness of questions, answers, rationale, and representation.