5. Specifying and measuring the usability of products

The framework describes the components of usability and the relationship between them.

In order to specify or measure usability it is necessary to decompose effectiveness, efficiency and satisfaction and the components of the context of use into sub-components with measurable and verifiable attributes. The components and the relationships between them are illustrated in Figure 1.

When specifying or measuring usability, the following information is needed:

• A description of the components of the context of use including users, equipment, environments, and tasks. This may be a description of an existing context, or a specification of intended contexts. The relevant aspects of the context and the level of detail required will depend on the scope of the issues being addressed. The description of the context needs to be sufficiently detailed so that those aspects of the context which may have a significant influence on usability could be reproduced.
• Usability measures consisting of target or actual values of effectiveness, efficiency, and satisfaction for the intended contexts.

Figure 1 - Usability framework

Relevant characteristics of the users need to be described. These may include knowledge, skill, experience, education, training, physical attributes, and motor and sensory capabilities. It may be necessary to define the characteristics of different types of user, for example with different levels of experience or performing different roles.

Relevant characteristics of the equipment need to be described. The description of the hardware, software and materials may be in terms of a set of products, one or more of which may be the focus of usability specification or evaluation, or it may be in terms of a set of attributes or performance characteristics of the hardware, software and other materials.

Relevant characteristics of the physical and social environment need to be described. Aspects which may need to be described include attributes of the wider technical environment (e.g. the local area network), the physical environment (e.g. workplace, furniture), the ambient environment (e.g. temperature, humidity) and the social and cultural environment (e.g. work practices, organisational structure and attitudes).

The goals of use of a product should be described. Goals may be decomposed into sub-goals which specify components of an overall goal and the criteria which would satisfy that goal. For example, a telephone sales clerk may have the goal to "Maintain customer orders". This overall goal might then be decomposed into sub-goals such as:

• "Make accurate record of all orders placed by customers"
• "Provide information rapidly in response to customer inquiries about orders placed".

The level at which the overall goal is set is a function of the boundary of the work system which is under consideration and which provides the context of use. In the example above the work system under consideration consists of clerks taking telephone orders.

The specification or measurement of the usability of a particular product should identify the overall goal, specific sub-goals, the relevant context of use, including the tasks and resources involved, and the measures of effectiveness, efficiency and satisfaction which are chosen as being relevant to the goals which have been identified.

Tasks are the activities undertaken to achieve a goal. Characteristics of tasks which may influence usability should be described as part of the context of use, e.g. the frequency and the duration of performance.

Detailed descriptions of the activities and processes may be required if the description of the context is to be used as a basis for the design or evaluation of details of interaction with the product. This may include description of the allocation of activities and steps between the human and technological resources. Tasks should not be described solely in terms of the functions or features provided by a product or system. Any description of the activities and steps involved in performing the task should be related to the goals which are to be achieved.

For the purposes of evaluating usability, a set of key tasks will typically be selected to represent the significant aspects of the overall task.

NOTE - User tasks and sub-tasks can be identified by task analysis (for more information see the bibliography in Annex E).

Annex A gives examples of how the components of the context of use can be described in terms of characteristics which may be relevant to usability.

A description of usability measures consists of target or actual values of effectiveness, efficiency, and satisfaction for the required contexts. It is normally necessary to provide at least one measure for each of effectiveness, efficiency and satisfaction. Because the relative importance of components of usability depends on the context of use and the purposes for which usability is being described, there is no general rule for how measures should be chosen or combined.

The choice of measures and the level of detail of each measure, is dependent on the objectives of the parties involved in the measurement. The importance each measure has relative to the goals should be considered. For example where usage is infrequent, high importance may be given to measures of learning and relearning.

If it is not possible to obtain objective measures of effectiveness and efficiency, subjective measures based on the user's perception can provide an indication of effectiveness and efficiency.

Measures of effectiveness relate the goals or sub-goals of the user to the accuracy and completeness with which these goals can be achieved.

For example if the desired goal is to accurately reproduce a 2-page document in a specified format, then accuracy could be specified or measured by the number of spelling mistakes and the number of deviations from the specified format, and completeness by the number of words of the document transcribed divided by the number of words in the source document.

Measures of efficiency relate the level of effectiveness achieved to the expenditure of resources. Relevant resources may include mental or physical effort, time, materials or financial cost. For example human efficiency could be measured as effectiveness divided by human effort, temporal efficiency as effectiveness divided by time, and economic efficiency as effectiveness divided by cost.

If the desired goal is to print copies of a report, then efficiency could be specified or measured by the number of usable copies of the report printed, divided by the resources spent on the task such as labour hours, process expense and materials consumed.

Measures of satisfaction describe the comfort and acceptability of the use.

Satisfaction can be specified and measured by attitude rating scales or measures such as the ratio of positive to negative comments during use. Additional information may be obtained from longer term measures such as rate of absenteeism, health problem reports, or the frequency with which users request transfer to another job.

Measures of satisfaction may assess attitudes to use of the product, or assess the user's perception of aspects such as efficiency, helpfulness or learnability.

Further examples of measures that can be used for assessing usability are included in Annexes B and C.

Care should be taken in generalising the results of any measurement of usability to another context which may have significantly different types of users, tasks or environments. If measures of usability are obtained over short periods of time the values may not take account of infrequent events which could have a significant impact on usability, for example intermittent system errors.

For a general-purpose product it will generally be necessary to specify or measure usability in several different representative contexts, which will be a subset of the possible contexts and of the tasks which can be performed. There may be differences between usability in these contexts.