A Case Study: Remote Usability Evaluation on a Ventilator Developed to Address the COVID-19 Pandemic
Event Type
Oral Presentations
TimeThursday, April 1511:20am - 11:40am EDT
LocationMedical and Drug Delivery Devices
During the early stages of the COVID-19 Pandemic, most in-person usability evaluations were brought to a halt. Companies who were pursuing Emergency Use Authorization for Medical Counter Measure products for the pandemic, like ventilators and COVID-19 in vitro diagnostic test kits, still needed to proceed ahead with design and development in an accelerated fashion.
With the Emergency Use Authorization declaration in place, the FDA has allowed some products to be authorized for use without full Human Factors Validation. In order to reduce the risks around releasing a product without full Human Factors Validation data and uncovering serious use issues in the post-market surveillance phase, the authors of this proposal and the Manufacturer chose to conduct a remote usability evaluation of two-dimensional digital prototypes for a novel ventilator after carefully weighing the benefits and disadvantages to this methodology.

In this presentation the authors will share with the audience:

1) How and why the authors decided to conduct a remote evaluation with digital prototypes?
• Critical feedback was needed in an accelerated fashion to support the Emergency Use Authorization submission for the ventilator. There was little time to fully recruit and conduct an in-person usability study. The team was able to quickly identify a diverse set of participants in multiple geographical locations to participate in the sessions and mockup a fully digital prototype of the ventilator much quicker than producing hardware prototypes.
• Gathering user feedback on the user interface in a rapid fashion increased the confidence of business stakeholders to invest in the next phases of development for the project.
• Target healthcare user participants were actively working with COVID-19 patients early in the pandemic so an in-person study would have presented unacceptable risk to study personnel.
• Overall, we considered the advantages and disadvantages of remote usability testing (see details below), before moving ahead with this methodology.

Advantages of Remote Testing:
- Less exposure of study personnel to participants and vice versa
- Ability to recruit and test larger/more diverse groups of people
- Time/cost effective
- Can observer users in their natural environments (in medical office or home)
- May have decreased Hawthorne effect (compared to lab environment)

Disadvantages of Remote Testing:
- More prone to technology issues
- Higher likelihood of distraction in home as opposed to lab environment
- Constraints for viewing/listening (limited # of camera angles and mic positioning), limit the ability to identify all use errors or root causes
- Decreased security of intellectual property
- More difficult to extract participant feedback, non-verbal feedback not as salient

2) How readily available tools such as Zoom, Google Slides, and Adobe InDesign were used for conducting the remote session and simulating the ventilator user interface digitally.
• A digital rendition of the ventilator user interface, including knobs and LCD panel, was created in Adobe InDesign by the Manufacturer.
• The Google Slides application was used to present both stimuli (i.e., ventilator user interface, key animations) and key research questions to the participants. The authors will share select slides from this remote evaluation to illustrate how the stimuli was presented with animations.
• Separate images of knobs were used and manipulated so study personnel could adjust the settings on the ventilator to assess different use scenarios within the same session.
• LCD panel mocks ups with different alarm messaging and sound clips allowed animations of different alarms to be presented during the session.
• Digital prototyping was a significant cost savings over hardware prototyping.

3) How did key planning reduce negative impact of different events (i.e., internet issues, unexpected participant time constraints) on collecting important usability data?
• A pre-session check was performed at the beginning of the session by study personnel.
• A moderator and observer were both involved in the session to ensure continuity of the session should study personnel experience internet connectivity issues.
• Questions and tasks were prioritized in the event the session had to be cut short.

4) How did digital prototyping allow study personnel to quickly implement and evaluate design changes?
• Due to the digital nature of the ventilator prototypes and the ease with updating the Google Slides presentation, study personnel could rapidly make adjustments in alarm messaging, user interface design, and labeling between participants so feedback could be collected on the user interface changes.
• Gathering user feedback on modifications to the user interface implemented as a result of early participant feedback was a significant time savings for product development.

In a pre-COVID-pandemic world, we might have opted for an in-person usability study for this project. However, this case study illustrated that with remote testing of a digital prototype we were able to successfully gather quality user feedback on the ventilator user interface to improve the safety and customer acceptance of the product in a rapid, lower cost solution than conducting in-person testing with hardware prototypes.

(Acknowledgement: Thank you to Ashely Ngo for contributing to the content of this presentation.)
Senior Human Factors Engineer