Digital Testing in Human Factors: Creating Real Options for Virtually Testing Medical Devices
Event Type
Oral Presentations
TimeWednesday, April 1411:00am - 11:20am EDT
LocationEducation and Simulation
DescriptionThe COVID-19 global pandemic is an event that caught most of the world off-guard and forced businesses to be creative in how to solve problems on a scale that has never existed before. Chief among them for human factors professionals: how can practices that require participants to be in-person and interact with physical stimuli be adapted to allow effective testing to be completed at a high level through remote means? The needs of our time require creative innovation, and we are on the forefront of adapting technologies that will serve as precursors for augmented reality (AR) and virtual reality (VR) technologies for years to come.

In our talk we will briefly explore methodologies we implemented in our response to the COVID-19 pandemic, then discuss how other businesses in other industries have begun to implement AR and VR technologies. We will then segue into how these innovative remote technologies can be applied and implemented for medical and pharmaceutical device companies. We will end the presentation by discussing how we expect AR and VR technologies to continue to grow and supplement remote methodologies in the future of medical human factors testing.

In 2020, we worked with several clients in the medical device/pharmaceutical industry to pivot their research plans. It was important that we were still able to provide actionable findings from participants who took part in human factors studies from a remote setting. To do this effectively, we used a combination of different methodologies including screensharing, the creation of virtual interactive tabletops, and implementing a technique (virtual hands) where participants could direct a pair of hands on-screen to perform actions that they would if they were there in person.

Our adaptations to remote testing this year have demonstrated a need to test medical prototypes in an immersive, VR environment which can allow participants to closely interact with a new design concept or prototype while also allowing them to assess it in context. Our ‘virtual tabletop’ and ‘virtual hands’ serve as a precursor for having the participant put on a headset and then be virtually in a space with our moderator and with the prototype.

Internally, we began working to develop a method to expand user experience testing to VR technologies. We experimented with remote testing and discovered moderators could interview participants while the participant was in a VR headset quite easily – if security protocols were in place and safety measures were in place (to mitigate any VR sickness or nausea).

In the past four years we have conducted HF and UX tests using VR headsets in person. And when testing needs switched to completely remote due to COVID-19, we fast-tracked our efforts to develop remote VR testing and identified methods to remotely test VR devices, applications, and products. To date, we have performed several rounds of research with users utilizing their headsets remotely to test out and respond to applications and have another benchmarking test launching in early 2021.

There are several logistics to keep in mind when testing participants in a virtual setting (VR): the fidelity of the experience (how ‘real’ can it truly feel when you cannot physically hold the product), the cost of developing the 3D scan of the product or environment, sending and receiving of headsets and accessories to participants, and security measures for highly confidential work. We will share our approach and how we mitigate each of these factors when testing with VR remotely.

For example, many of our clients require high confidentiality when we test their products and prototypes and therefore need assurance that the prototypes can be secured. We have prepared logistics to ensure headsets with pre-loaded prototypes can be securely sent and monitored by researchers throughout the duration of a study. We have also modified headsets to include additional layers of security and which allow the researchers to ‘allow’ and/or ‘revoke’ access to prototypes within the headset.

As discussed above, testing in VR allows us to place the participant in a familiar or appropriate context when conducting formative user testing for prototypes and concepts. VR even allows for a closer ‘in-context’ experience for testing that is traditionally done in-lab and can offer a cheaper and more accessible alternative to testing in medical environments for the purposes of research. By using VR technologies, we can scan and recreate an environment in realistic 3D (e.g. Kitchens, living rooms, hospital rooms, pharmacies, etc.). This can allow participants to move around in the space while interacting with a digital version of the stimuli.

We are further developing our expertise and aiming to test with VR environments that allow us to perform HF/UX testing for early-stage prototypes without interactions (the participant examines the device, but does not ‘use it’), creating ‘Wizard of Oz’ interactions where the user can see how the prototype would work in VR, but isn’t actually interacting with it (the moderator ‘changes the state’), and also true interactions – where the user can manipulate the prototype or device in VR.

For example, imagine a test for the special refrigeration system concept for the new COVID vaccines. Due to the pandemic, accessing and testing this system in a real pharmacy could prove exceedingly difficult. With the use of VR technology, we would scan and replicate a pharmacy setting and use a 3D model of the refrigeration concept. Then, using our existing techniques, pharmacists could ‘enter’ the pharmacy in VR, interact with virtual design concepts, and participate in a remote interview with human factors researchers. This would allow the participants to assess the ‘refrigeration concept’ in the context in which they would typically use it, without having to subject the research team, participants, and facility staff to heightened safety protocols related to in-person testing.

We will conclude our talk by highlighting how each of these methodologies can be set up for testing for studies in the short term, expand on how each are important precursors to understand the cognitive load and physical responses being placed on the participant, and discussing future potential use-cases for AR and VR in medical device testing.