Usability Testing

Usability Testing

Introduction to Medical Device Usability Testing

1. Basic Concepts

Usability: As defined in IEC 62366-1, the characteristic of the user interface that facilitates use and thereby establishes effectiveness, efficiency and user satisfaction.

Usability Engineering (UE) / Human Factors Engineering (HFE): a systematic approach to applying usability principles to medical device design, aiming to reduce the risks caused by use errors by optimizing user–device interaction.

Core purpose: to ensure that the medical device is safe and effective when operated by intended users in the intended use environment, with acceptable use error risk.

2. Key Regulations and Standards

Standard/guidance Content
IEC 62366-1:2015 + A1:2020 Application of usability engineering to medical devices (international)
IEC/TR 62366-2:2016 Guidance on the application of usability engineering (technical report)
FDA HFE Guidance (2016) Applying Human Factors and Usability Engineering to Medical Devices
FDA Draft Guidance (2022) Draft guidance on the content and format of human factors information in 510(k) submissions
ANSI/AAMI HE75 Detailed guidance on human factors engineering in medical device design
YY/T 1474 Chinese equivalent standard (identical adoption of IEC 62366-1)
MDR (EU) 2017/745 Usability requirements of the EU Medical Device Regulation

3. Key Terms

Term Meaning
Intended user Clinicians, nurses, patients, caregivers, etc.
Use environment Operating rooms, ICUs, homes, emergency scenes, etc.
Use scenario Description of the specific situation in which the user performs a task
Critical task A task that, if performed incorrectly, could result in serious harm
Use error A user action resulting in an outcome different from that intended
Use-related risk Risk associated with user interaction
User interface (UI) Includes hardware, software, labeling, IFU, training materials, etc.

4. Usability Engineering Process (IEC 62366-1)

1. Prepare the use specification
 ↓
2. Identify hazards and hazardous situations related to the user interface
 ↓
3. Identify critical tasks
 ↓
4. Establish the user interface specification
 ↓
5. Establish the user interface evaluation plan
 ↓
6. User interface design, implementation and formative evaluation
 ↓
7. Summative evaluation
 ↓
8. Compile the usability engineering file (UEF)

5. Two Core Types of Evaluation

1. Formative evaluation

  • Timing: early and middle stages of product development; multiple rounds possible

  • Purpose: find design flaws and improve iteratively

  • Sample size: 5–8 participants per user group (finds 80%+ of usability problems)

  • Methods: cognitive walkthrough, heuristic evaluation, usability testing, expert review

  • Use of results: input to the next design iteration; not used for final validation

2. Summative evaluation

  • Timing: after design freeze and before market launch

  • Purpose: validate that the user interface design adequately controls use-related risks

  • Sample size: at least 15 participants per user group (FDA requirement)

  • Method: simulated use testing

  • Requirements:

    • Conducted in a representative use environment

    • Using the final or production-equivalent product

    • Must cover all critical tasks

    • Followed by root-cause interviews (to analyze the causes of use errors)

6. User Groups

Different user groups must be tested separately. Common groupings:

Grouping dimension Examples
Profession Physicians, nurses, technicians, patients, family members
Experience Novices, experienced users
Use environment Professional hospital use, home use
Physical characteristics Users with limited vision, hearing or dexterity

For example, a home blood glucose meter may involve two groups – patients + caregivers – each requiring ≥15 participants.

7. Identifying Critical Tasks

Critical tasks are the core of usability testing and are identified based on:

  1. Risk analysis (ISO 14971): tasks where use errors could cause serious harm

  2. Historical data: adverse events, recalls and complaints for similar products

  3. Task analysis: breaking down the user’s operating workflow

Typical critical task examples:

  • Infusion pump: setting the correct dose and rate

  • Defibrillator: selecting the correct energy and delivering the shock

  • Insulin pen: loading the cartridge, setting the dose, injecting

  • Blood glucose meter: sampling blood, reading results, recognizing abnormal values

8. Test Methods and Data Collection

Common methods

Method Applicable stage Description
Simulated use testing Formative/summative Completing tasks in a simulated environment
Cognitive walkthrough Formative Experts analyze the user’s reasoning step by step
Heuristic evaluation Formative Reviewing the interface against usability principles
Interviews/questionnaires Throughout Collecting subjective feedback
Observation Throughout Recording actual user behavior

Key data to record

  • Task completion (success/failure/partial success)

  • Completion time

  • Use errors, close calls and operational difficulties

  • Subjective user feedback (SUS scale, NASA-TLX, etc.)

  • Video/audio recordings

  • Post-test interviews (root-cause analysis of errors)

9. Usability Engineering File (UEF)

Key documents to retain:

  1. Use specification

  2. User interface specification

  3. Known problems analysis (similar marketed products)

  4. List of hazard-related use scenarios

  5. List of critical tasks

  6. Formative evaluation reports

  7. Summative evaluation report

  8. User interface evaluation plan and results

  9. Training materials, IFU, labeling

10. Differences Between FDA and IEC 62366

Dimension IEC 62366-1 FDA HFE Guidance
Focus Safety risks caused by use errors Safety and effectiveness of critical tasks
Known problems analysis Not mandatory Mandatory (analysis of similar products)
Summative test sample size Not specified ≥15 participants per user group
Test report format Relatively flexible Detailed template requirements

In practice, we recommend following the FDA requirements, which satisfies both.

11. Common Challenges and Recommendations

  1. Recruiting users is hard: recruiting professional users (e.g. surgeons) takes time and must be planned early

  2. Environment simulation: recreate real use conditions as closely as possible (lighting, noise, stress)

  3. Defining use errors: “success” and “failure” criteria must be clearly defined

  4. Root-cause analysis: don’t just record “user error” – dig into the causes (knowledge, perception, cognition, action)

  5. Design changes: significant changes after summative evaluation may require retesting

  6. IFU and training: usability problems cannot be solved by relying on the IFU; the design itself must be safe

Scroll to Top