Usability Testing of Ultrasonic Surgical System for Lower Eyelid Fat Removal
TimeThursday, April 152:00pm - 3:00pm EDT
LocationMedical and Drug Delivery Devices
This work evaluated usability of ultrasonic surgical instrument for general liposuction & lower eyelid surgery to validate whether UI risk mitigation controls are properly established, and collected any opinions about the device. The usability test spanned the first three weeks of June 2020. The quantitative data: (1) task completion rate, (2) the number of use error occurred, (3) user satisfaction score, and qualitative data: subjective opinions (via survey and interview) were collected. The participants provided opinions about from overall device operating process to interface design, potential risk. Such as, most of them commented "need to clearly indicate the area where ultrasound is actually emitted." These test results could help medical device designers to understand human factor considerations.
According to the recent statistics from the International Society of Aesthetic Plastic Surgery(ISAPS), liposuction and lower eye surgery ranked second and third among the top five surgical procedures performed in 2018. The survey shows that two plastic surgeries have been widely performed. But because, for the these two procedures, there are some risks such as burns and organ perforation, physician's competence is critical. Particularly for lower eyelid surgery Doctor's knowledge of complications and anatomy is important. These risks could be reduced by improving usability of device as well as by enhancing physician's ability. Therefore, the evaluation of the user interface is needed prior to market release, since the user interface may reduce or eliminate side effects, and use-related error could lead harm to user and patient. One of the methods evaluating the medical device UI is the usability test. this study present the model of the summative test(Human factor Validation test) of the medical device conducted by applying related standard and guidance and the human factor considerations.
15 participants(appropriate for summative usability test) were recruited. They were all plastic surgeon from Yonsei University Severance Hospital and didn't have any interests, had not been involved in any development process.
The test environment was configured similarly to the general operating room environment.
Each test session included one participant and lasted up to about an hour. At the test session participant perform five use scenarios which reflects the actual use-flow. While participants use the device, test personnel observe and record any use problem. After whole use scenario, participants assessed the device with scoring(1-5 Likert-scale) and provided subjective opinion. The use scenarios consisted of the following: (1) Setup: Power On / Connect foot switch, (2) Connect Handpiece & probe(for abdomen), (3) Simulate surgical procedure (for abdomen), (4) Change Handpiece & probe(for lower eyelid), (5) Simulate surgical procedure (for lower eyelid). In the scenario 'simulate surgical procedure', Participants simulated for mannequins and could not actually insert probes. Therefore, the evaluation was carried out by focusing on the use of the device itself, except for the incision, suction process and so on. And the wearing actual surgical attire was excluded because it was not similar with the actual process and it might disturb participants. Such as, operating a device that was not actually sterilized with sterile latex gloves could have made participants uncomfortable.
At the test session, test personnel collected the following data: (1) Participant demographic and background information, (2) Use error, (3) task completion rate, (4) Satisfaction assessment score, (5) satisfaction score-related opinions, (6) survey response
Before conducting test, test goal was set for quantitative data. After all test completion, we analyzed the data to ensure the test goals are met, and to determine whether the UI is well designed for safe use.
A 60% task completion rate and six use errors were observed in one task, but this task was not a critical task. The type of six use errors was all same. The root cause of the use error was the difference with existing devices in how to use. The satisfaction score was for the UI interacting with each task. All the score of items were higher than a target score of 3.5. The item that received the lowest score of 3.93, was about an information-providing UI that tells us that ultrasound is emitted. Nine out of 16 participants provided the following comments on the item: "I wish I could adjust the volume of the sound that tells me that ultrasonic waves are emitted.", "I hope the sound is different by mode.", "Notify through not only the sound, but the light at the ultrasonic probe tip". These participants said they thought ultrasound was being released and carried out the surgery, but there were inconveniences because ultrasound was not actually being released.
The survey consisted of following contents: (1) Whether consider device is safe, (2) Important factors for safe use of this device, (3) Function required for safe use.
(1) On device safety, eight out of 16 respondents said it was safe.
(2) The important factors for safe use of the device (multiple responses) were answered in the order of physician proficiency (15 persons), the design of medical device itself (8 persons), and education for use (7 persons).
(3) As for the functions required to safe use (multiple responses), participant commented following:
First, "clearly indicate the area where ultrasound is actually emitted."(13 persons),
Second, "indicate the current probe's temperature."(12 persons),
Third, "possible to set the probe's temperature high limit (10 persons), and
Lastly, "heat protection measures in areas where ultrasound is not emitted to prevent unwanted areas from burning"(6 persons).
One of the test goals was to respond that the device was safe for more than 90% of the participants. But it was not achieved. Participants who thought the device was unsafe provided opinions about "need for safety verification such as clinical trials ", "need for clear presentation of the ultrasound emission area", "dangerous if education is not provided properly.", "burns and bleeding concern.", and "Protective measures are required for area not emitting ultrasound.", etc.
As the result of the summative evaluation analysis, we concluded that the safety-related test goal was not achieved, so design improvement is needed to improve the safety of the device. Therefore, another summative evaluation should be carried out after design improvements reflecting the evaluation results.