Conducting User Research Amidst Unpredictability and Constraints: Lessons Learned During the COVID Pandemic
Event Type
Oral Presentations
TimeThursday, April 1512:30pm - 12:50pm EDT
LocationMedical and Drug Delivery Devices
DescriptionSection 1: Introduction and Background
As user researchers, we know we should plan for and expect the unexpected, but the COVID pandemic threw the curveball of all curveballs. We were suddenly faced with a prohibition of any in-person work, navigating the overwhelming world of remote teamwork, and making sense of the constant influx of sobering COVID news.

Planning for usability studies during the COVID pandemic imposed constraints on nearly every aspect of the study design. We were forced to make decisions despite having no proof that a study riddled with COVID mitigations could even work, and no knowledge of how the COVID landscape would change day to day.

Now that the pandemic has been underway for a year, our team has conducted user research in various formats despite these constraints and uncertainties. While we have encountered challenges and surprises along the way, we have been able to meet our study objectives and have gained a better understanding of what to expect from studies conducted under these extreme conditions. In this presentation, we will discuss lessons learned through three case studies and share our recommendations for what to consider with future studies, both within the COVID landscape and in the inevitable “new normal.”

Section 2: Case Studies
To tackle study planning during the COVID pandemic, we started with the standard approaches of identifying study objectives, determining ways to collect the data required to meet those objectives, and assessing and mitigating against study risks. However, we discovered a need to balance study risks and study objectives more than we were accustomed to.

We will present three case studies of user studies we planned and conducted during the COVID pandemic. We will outline safety mitigations and logistics for each case study. We will then share how these mitigations impacted the study experience, as well as lessons learned for what worked well and what we could improve upon in the future. The chosen case studies present a range of research approaches, from fully remote to fully in-person:

1. Case Study 1: In-person simulated use training and interviews with participant teams

In 2020, we conducted a 3-month validation study to evaluate large capital equipment used by surgical teams, requiring a specific simulated use environment and in-person researchers to adequately collect the required data. This study presented the most complex logistical planning we’ve ever encountered, with numerous challenges around resourcing, participant availability, travel, and more.

2. Case Study 2: Hybrid remote / in-person 1-on-1 simulated use interviews

In Fall 2020, we were tasked with planning and executing a validation study of an injection device through one-on-one interviews with 45 participants. Since the interviews involved a simulated injection, the study design required at least one in-person moderator to ensure participants handled the product and other study materials safely. Due to a tight project budget and timeline, and COVID constraints on in-person researchers, we designed a hybrid in-person and remote study approach leveraging technological tools for remote data collection.

3. Case Study 3: Remote 1-on-1 user needs interviews

In Summer 2020, our team was involved with an early-stage internally funded project to understand unmet needs in the assistive technology space. Due to the front-end, exploratory nature of the work, we were able to conduct user research through one-on-one interviews over video conferencing. In the next phase of this project (expected to occur within the next month), we plan to again conduct remote interviews, this time sending “off the shelf” materials to each participant to evaluate their use process and further explore unmet needs in the space. We expect this will pose some technical challenges but that we will still be able to obtain an understanding of the user experience to inform early-stage product design without requiring in-person observation.

Section 3: Insights and Recommendations
Based on our case studies, we have learned that through roughly the same risk-based study design process we typically use, we were able to still conduct effective user research despite the numerous constraints and uncertainty presented by COVID. However, the experience of conducting the study was different than what we expected in several ways, such as:
--Shifts in researcher responsibility to effectively follow study protocols, safety protocols, and project contingency plans
--Changes in the researcher’s dynamic / rapport with participants in remote and distanced environments
--Increased stage-gated decision making to deal with uncertainty

While we hope that the restrictions presented by COVID are temporary, we expect to still deal with unexpected constraints as part of user research moving forward. With the appeal of remote usability testing and creative approaches to study execution that COVID has sparked, we will discuss study methods that can be leveraged for future research and what considerations to keep in mind as part of the planning process.
Principal Human Factors Engineer
Senior Human Factors Engineer