Usability Testing for the Mental Health Evaluation and Lookout Program (mHELP) Application for College Students
Event Type
Poster Presentation
TimeThursday, April 152:47pm - 2:49pm EDT
LocationDigital Health
DescriptionKeywords: Mental Health, College Student, Usability Testing

With the pandemic, there has been a severe increase in mental health conditions, especially among young adults (Czeisler et al., 2020). According to the Center for Disease Control and Prevention (CDC), the prevalence of symptoms of anxiety and depression have increased since 2019 from 8.1% to 25.5% and 6.5% to 24.3% respectively. Barriers, such as finances and the stigma surrounding mental health, prevent young adults from seeking professional help (Sareen et al., 2007). However, the amount of time a college student spends on their smartphone is approximately 8 to 10 hours a day (Wood, 2014). Thus, a potential solution is simply downloading an app.
When designing an app, it is crucial to perform usability tests. This is to ensure that the “user interface matches the natural human way of thinking and acting” (Kaikkonen et al., 2005). However, the pandemic prevents in-person studies. Therefore, online and remote forms of usability testing are used, which has its limitations, but also promotes some advantages in identifying certain usability issues (Jeong et al., 2020).
This study is part of a bigger project aimed in developing a mobile heath intervention for college students’ mental health self-management (mHelp). The overall purpose of this study was to identify usability issues of the mHelp app in the current iteration of design. The outcomes of this study will help shape future protocols for usability testing using online applications as well as identify and resolve major usability issues that users face.

A usability testing protocol was written to be semi-structured to understand any usability issues the design prototype presented. Participants were recruited from a large university system in the United States through email.
Sessions were conducted and recorded via Zoom on a desktop or personal computer lasting between 60 to 90 minutes. Participants received a compensation of $50 each and this study was approved by the university institutional review board.
The participant was accompanied by one interviewer who facilitated the session. During Zoom sessions, the participant interacted with Figma (Field, 2016) design prototypes of the mHelp app for the phone and smartwatch. Participants gave feedback by thinking aloud while navigating through the app design prototype.
The zoom test session contained five parts:
1. Introduction. The interviewer introduced the purpose of the study and stated a disclaimer of the limitation of the app design prototype.
2. Pre-Session Questionnaire. The interviewer’s screen was shared displaying the questionnaire. Background information collected included general demographics and any history with counseling and/or other mental health applications.
3. Prototype Tutorial. The interviewer showed the participant how to navigate through the Figma design prototype of the mHelp app.
4. Scenarios and Custom Questions. After the tutorial, the interviewer shared the design prototype link with the participant via the Zoom chat and the participant’s screen was shared to show their movement through the app. During the bulk of the usability session, the interviewer first read aloud a brief scenario, such as “From the home screen, you decide to check out what counseling services are provided,” then asked the participant to think aloud as they navigated through the prototype and commented on each screen. If a participant had difficulty completing a scenario, the interviewer would give simple guidance until the scenario was completed. Follow-up questions were asked to better understand any usability issues that the participant experienced. After going through a specific feature, the interviewer asked the participant to rate the ease of navigation on a scale from zero to ten, zero signifying very difficult and ten signifying very easy.
5. Post-Session Questionnaire. Once all scenarios were completed, the interviewer’s screen was shared displaying the questionnaire. This portion was contained two parts: (1) having the participant complete a System Usability Scale (SUS) questionnaire, (2) answering three open-ended question regarding the participant’s overall experience, favorite and least favorite parts of the app. The participant also had the freedom to ask the interviewer any questions or leave comments for the design team.

There was a total of 33 participants for the Zoom usability testing sessions, with a majority being female (70%). The participants ranged from undergraduate freshmen to graduate students in different majors. The data analysis and the recommended design changes were completed in January 2021. The new design changes will be tested soon on Maze and will be completed in the Spring of 2021.
From the first iteration of usability testing performed via Zoom, major usability issues that were identified included:
• Confusion with the meaning of the SOS feature. Originally, the design of the SOS feature was to record stress moments the user experiences with the options to broadcast their current location to emergency contacts and a shortcut to the counseling feature where other resources are provided. However, participants commented on how the meaning of SOS would be more urgent and have the same severity as calling 911. Therefore, the design and purpose of the SOS feature was changed to serve only for emergency situations.
• Confusion with the time management feature. A previous study showed time management being one of the most popular features recommended to have in a self-regulating, mental health app amongst the university’s student population. However, there was confusion as to why the calendar and list functions were on the same feature. Therefore, the new prototype now has a calendar and list in two separate features.
• The connection between the features and mental health. Some participants voiced how they liked the features provided but did not see how some of the features were connected to mental health. This issue has been solved by changing the designs of some features as well as adding an educational feature, which provides instructive videos and articles on how different aspects, such as sleep tracking and food intake, can affect mental health.
While usability analysis portrayed mostly minor aesthetic issues with the designs, the major usability issues brought up how effective the app can be in helping users self-regulate their mental health. Multiple participants liked the idea of having all the important features, such as time management and goal tracking, to be all in one app. However, more testing is needed to dig deeper in how to improve the connection between mental health and the app.

After analyzing the data, the usability testing team presented a list of major usability issues with recommended design changes to the design team. The next iteration of usability testing will then be performed on a different online platform, Maze (Maze, 2021), with a larger participant population.
There are many differences between the Maze protocol and the Zoom protocol. Instead of a scheduled video call between the participant and the interviewer, Maze is an online application the participant can independently interact with the design prototype and answer questions. The data is automatically recorded in a detailed report created by Maze, which will be analyzed by the usability testing team.

This research is a usability testing study on the use and effectiveness of the mHelp application to help college students self-regulate their mental health. We conducted usability testing sessions via Zoom with college students and summarized the major usability issues with the mHelp app. Remote and online usability testing has the advantage of providing participation from the comfort of their own homes. For the next iteration of testing, Maze will be used, which will mimic the natural environment as the participant will complete the session independently. This next phase of testing will discover and resolve more usability issues found in the mHelp app.