Novel Application of Cognitive Task Analysis: Development of a Neonatal Mask Ventilation Checklist for Instruction and Assessment.
TimeWednesday, April 142:00pm - 2:20pm EDT
LocationEducation and Simulation
Neonatal mask ventilation (NMV), learned and practiced during simulation-based resuscitation training, is a vital competency for neonatology fellows. NMV can be error-prone, especially for novices, who have difficulty creating a mask seal and providing appropriate breathing rates. It is well-documented that NMV delivery is inconsistent, even among experienced health care providers, and the deleterious effects of inconsistent NMV have been shown in human and animal studies. Despite gaps in NMV skill, no objective NMV instruction and assessment tools exist.
The development of checklists for instruction and assessment of psychomotor skills in health professions education has relied on literature review and subject matter expert (SME) consensus through free recall. However, studies show that unaided free recall of analytical decisions and problem-solving strategies is incomplete and/or imprecise. When recalling procedural steps, experts typically omit the cognitive aspects of a task, resulting in omission of up to 70% of essential information. This unintentional omission occurs because, as expertise is achieved, skills become automated and procedural knowledge and the associated cognitive processes become inaccessible to conscious memory. SMEs are therefore unable to completely and accurately articulate the knowledge and thought processes that drive their performance, which results in checklists that fail to highlight the underlying cognitive processes and steps that are error-prone and challenging to novices.
Cognitive Task Analysis (CTA) is a family of knowledge elicitation techniques used to elicit the automated knowledge, strategies, and cognitive and perceptual cues that underlie expert performance. CTA: 1) yields information about experts’ overt behaviors through observation and simulations, 2) highlights steps that are potentially challenging to novices, and 3) provides insight into experts’ implicit cognitive processes by eliciting examples of their lived experiences through task diagrams, semi-structured interviews and think-aloud protocols. The resulting comprehensive descriptions are used to inform the development of a variety of instructional materials.
In this study, we applied CTA to the development of a checklist for instruction and assessment of NMV using the flow-inflating resuscitation bag.
A convenience sample of five SMEs participated in the study. All SMEs were board certified neonatologists with at least five years of ongoing, current clinical experience in an academic level 4 (highest designation) neonatal intensive care unit and experienced in the use of the flow-inflating bag. Approval was obtained from the Institutional Review Board. All SMEs signed informed consent.
SMEs participated in CTA, which included: 1) simulated NMV performance, 2) creation of a dynamic checklist, and 3) semi-structured interview.
Simulated NMV performance:
To elucidate the steps of NMV, SMEs performed NMV on a neonatal mannequin, using the flow-inflating resuscitation bag. SMEs were instructed to “think out loud” and provide as thorough detail as possible, as if teaching a novice. Performance was video-recorded. Each SME’s video-recording was transcribed and a checklist of NMV was created for each SME.
To gain a broad overview of NMV and identify areas for further exploration during interviews, each SME created a dynamic checklist. To achieve this, SMEs configured individual laminated cards of the steps of NMV on a pin board to represent NMV based on their lived experiences. They used laminated arrows to represent sequence and laminated blank cards to modify steps and/or generate additional steps, as necessary. On their respective dynamic checklist, each SME marked the steps they perceived as cognitively difficult and challenging to novices. This highlighted areas to be probed during the interview. Each SME’s dynamic checklist was photographed and transcribed. A checklist of NMV was created for each SME.
Each SME individually participated in an in-depth semi-structured interview, conducted using a protocol based on the Concepts, Processes and Principles CTA method, which has been used to develop procedural checklists in medicine. The interviewer focused on the steps the SME identified on the dynamic checklist as challenging to novices. Interviews were recorded and transcribed verbatim. Sixteen procedure codes were created a priori, based on the published steps of NMV. To cross-check the codes and identify missing codes, the primary author and a second SME, who did not participate in the study, independently analyzed the de-identified transcripts. Based on the collective codes, the primary author reviewed and coded each of the interview transcripts.
Each SME’s coded interview transcript was used to create a detailed, step-by-step NMV checklist. Thereafter, each SME’s simulation checklist and dynamic checklist were superimposed on their respective interview checklist to ensure that all information had been captured. All five checklists were then aggregated into a preliminary checklist.
To generate SMEs’ feedback and agreement on the preliminary checklist, via electronic mail, we conducted two modified Delphi reviews. The SMEs evaluated the checklist on seven attributes and made modifications as deemed necessary. SMEs’ ratings of the checklist attributes were averaged. The checklist was revised following each Delphi review, based on SME feedback. A final checklist was created.
SMEs had an average of 19 years of clinical experience and each had performed NMV more than 200 times. Qualitative analysis of the CTA interviews confirmed the 16 a priori codes, one additional (procedure) code and six codes that were not procedural steps. The aspects of NMV that SMEs described as challenging to novices were used to detail the steps of NMV, and combined with the procedure codes, resulted in a detailed, 17-step checklist. The steps of NMV were: 1) clean hands, 2) put on gloves, 3) adjust bed height, 4) gather equipment, 5) set up equipment, 6) test bag, 7) assign assistants’ positions and roles, 8) position patient, 9) apply monitors, 10) obtain mask seal using one-hand technique, 11) obtain mask seal using two-hand technique, 12) assess mask seal efficacy, 13) squeeze bag to achieve appropriate rate, 14) squeeze bag to achieve appropriate pressure, 15) assess mask ventilation efficacy, 16) corrective steps for ineffective mask ventilation, and 17) discontinue mask ventilation. The six non-procedural steps were combined with the 17 procedural steps to create a separate, detailed instructional guide.
SME ratings of the checklist attributes scored on average ≥3.90 (out of 5). All SMEs agreed that no steps were missing and that all included steps were necessary. SMEs’ input collected during the second Delphi review related to the checklist format and assessment of steps of NMV that cannot be directly observed. The checklist was formatted to improve visual appeal and assessor prompts were added to steps that cannot be directly observed; for example: “How do you assess mask seal efficacy?”
By using CTA, we deconstructed SMEs’ automated skills into the discrete steps of NMV and identified the steps that are challenging to novices. We used the SMEs’ descriptions of strategies to detail the error prone steps, thereby stratifying the steps into distinct elements that can be taught and assessed during simulation-based training. The level of detail in our checklist – achievable only through CTA – is important, because the execution of complex psychomotor tasks such as NMV relies on both declarative and procedural knowledge - the “why” and “how,” respectively. For example, knowing that an effective mask seal should be created, and recognizing how to achieve an effective mask seal by: 1) opening the mouth, 2) placing mask over nose and mouth, 3) holding the mask in place with appropriate hand positioning, 4) pulling the jaw towards the mask (without placing fingers on eyes/soft tissues of neck), 5) gently pushing down on the mask while holding the head in sniffing position, and 6) periodically checking mask and head position, adjusting as necessary. Thus, through CTA, we direct the learner’s (and instructor’s) attention to the otherwise subtle complexities of this important, life-saving procedure.