Human Factors Testing Sample Size Requirements: Is It Time to Reevaluate?
Event Type
Poster Presentation
TimeThursday, April 152:00pm - 3:00pm EDT
LocationMedical and Drug Delivery Devices

According to the 2016 FDA Human Factors Guidance, Human Factors Validation Testing should include a minimum of 15 test participants from each intended user group. This sample size has been debated, as some experts suggest that fewer than 15 test participants may be needed to achieve similar results. However, the current FDA requirement of 15 participants per user group proposed by Laura Faulker has not been modified by the Human Factors community since the draft FDA guidance for Human Factors was initially published in 2011.

To investigate the 15 participant sample size requirement as it relates to User Manual Comprehension tasks, UserWise collected data from two usability studies where the user documentation was evaluated, to gain insight on what the optimal number of test participants is to uncover all use errors when specifically testing comprehension and understandability of written instructions.

Key Message:

UserWise analyzed User Manual Comprehension task data collected from the User Manual Comprehension portion of usability studies for two different medical devices. The goal was to determine the optimal sample size per user group for User Manual comprehension tasks, using Faulkner’s study design as outlined in the 2016 FDA Human Factors Guidance as a model.

The minimum viable sample size for usability testing has been widely debated, with some researchers claiming 5-users as a sufficient sample size and others arguing for larger sample sizes are necessary to uncover all usability issues. Despite the 5-user model presented by usability researchers Virzi and Nielsen, it is evident from the case studies conducted by UserWise, that a sample size of five test participants is not always sufficient to confidently capture all usability problems with user documentation.
A sample size of 5-users may theoretically be an ideal sample size to detect the most common usability problems. However, the optimal sample size for uncovering usability problems while minimizing financial burden varies. Sample size determination should be based on the predicted occurrence of the usability issue, the complexity of the device, and the aptitude of the participants.

This presentation will include a methodology for selecting an appropriate sample size for a given study, both building upon and bringing into question the FDA-required sample size of n=15 participants per user group.

Audience Takeaways:

UserWise recommends further investigation of historical usability study data, as performed in these diagnostic product and injection pen case studies, to determine the optimal sample size for evaluating User Manual Comprehension. The presentation proposes a new approach to sample size determination, pairing the approaches proposed Virzi, Neilsen, and Faulkner to determine an optimal middle ground for conducting User Manual Comprehension testing.