Presentation
A New Baseline: Refining Observer Training for Studies in the Operating Room
Event Type
Oral Presentations
TimeTuesday, April 132:20pm - 2:40pm EDT
LocationPatient Safety Research and Initiatives
DescriptionTopic
New technologies in the healthcare setting carry the potential for improvements in patient outcomes. However, safe implementation of these technologies requires investigations that explore the relationship between new devices and those who use them. Direct observations can help explain complex interactions between technology and healthcare workers. However, reliable and precise observations of the healthcare environment require robust observer training. Few studies have demonstrated a methodology for training observers that enables critique and improvement. In order to achieve reliable, accurate observations, our group has created a web-based observer training course intended to help trainees record flow disruptions (FDs) during live robotic assisted surgeries (RAS). Since this study is occurring at multiple sites over a wide timeframe, the training course needed to be readily available, easily accessible, modifiable, and comprehensive to reflect the needs of the trainee. To verify that the observer has been successfully trained, the structure and content of the course must lead to a review of observational reliability that compares the trainee to an expert.
Our course offers new observers training through modules that overview the aims of this research project, introduce the trainee to RAS in the operating room (OR), explore current literature on FDs in the OR, elaborate on data collection techniques, review the possible implications of findings through other literature, and assess proficiency in identifying, categorizing, and rating FDs with quizzes. The course then instructs the trainee and a human factors expert to conduct joint-observations in order to evaluate interrater reliability (IRR). A novel tool, Research and Exploratory Analysis Driven – Time data Visualization (READ-TV), designed specifically for this research project, assisted in the easy comparison of data collection sheets between observers-in-training and professionals.
This course could be used for a variety of situations and offers a useful framework baseline for improving the rigor and assessment of direct observation methods in future HF healthcare research and practice. Our lecture will describe our experiences designing and testing the course so that other researchers who rely on direct observational data may further develop their training methods.
Application
Conducting direct observations is complex and challenging even for experienced researchers. This course was designed to train non-clinician observers in both the surgical and observation methodology topics necessary to collect observational data. While our content focused on training observers to identify FDs in RAS procedures, this course can be easily modified for a broad range of research projects involving direct observations. Similarly, READ-TV facilitates assessment of IRR and provides a much deeper examination of observer agreement to ensure observers are adequately trained. This course has proved useful for our multisite, multiyear study and can provide a model for similar research projects with its potential to increase observational consistency.
Background
As the OR changes, further research can support more effective use of new systems, devices, and technologies while reducing potential harms. Direct observation of work is one of the only ways to understand work “as done” in real clinical contexts and has informed a range of approaches to studying clinical systems ‘in the wild.’ Observers can come from diverse professional backgrounds – clinical, HF, design, engineering, quality improvement – so ensuring an appropriate level of shared knowledge and observation skill is vital for reliability and validity. Previous studies have trained their observers through approaches such as lectures, coaching, and practice observations. However, the training materials, processes, and evaluations of observers are rarely provided in replicable detail. It can therefore be difficult to see how IRR was achieved on the multiple dimensions that quantitative and qualitative observations are conducted. In deploying direct observation for our studies in robotic surgery, we recognized a more methodological and rigorous approach to observer training and assessment was required. This presented an opportunity to create and evaluate an observer training course that would cover case selection, sampling approaches, and methods for data recording, coding, and analysis.
Overview of Presentation
This lecture will detail why we developed an observer training course, how we chose its platform, and what our course can offer to trainees. We will share what we have learned through this development process so that other researchers might use similar training strategies in studies that rely on direct observations.
Despite the need for comprehensive observer training to ensure reliable data collection, few studies describe training methods and measures for IRR in detail in healthcare. A multisite, multiyear study requires a rigorous approach to the training and assessment of observers prior to collecting usable clinical observation data. This in turn benefits from an easily accessible training course, supplementing their background in human factors and systems analysis in the OR environment and surgical procedures, and training them in reliable recording of observational data.
At first, the OR environment might seem overwhelming for non-clinical observers; they can expect to learn a series of new skills while adapting to new environments. Trainees need to understand the scope of the project, HF in healthcare principles, and data collection techniques. These are not the only necessary competencies to have in the OR. Trainees must learn where to stand, where and when to move, when to speak, and how to gain and maintain the trust of the OR staff. Within our project, our course has helped observers become an accepted addition to the OR capable of recording reliable data. The course opens with an introduction that describes the roles, rules, and reality of the OR along with the basic operations of a da Vinci robot. After becoming familiar with their new work environment, the trainee reads literature prior to their first observation that elaborates on the HF implications of surgical robots. After a data collection guides and quizzes, the trainee attends a set of joint-observations with their expert mentor. The conclusion of the web-based course outlines instructions for IRR review and offers literature and guides for the observer’s contributions to the project outside of data collection. This includes studies that expand on the disruptions the observer may have already detected in the OR.
Both trainees reported that the web-based course was easily accessible, well-constructed, and useful. This course helped familiarize trainees with topics previously unknown to them. Trainees demonstrated growing proficiency as observers despite having no previous experience with these data collection techniques or the OR environment. We are currently in the process of developing additional assessments of skill prior to live observations in the OR which provide automatic feedback, including meeting specific IRR goals.
Conclusion
Direct observations are necessary to understand the interactions between humans and systems in the operating room. Since current literature does not provide detailed methods or standards for observer training, our research group has developed and implemented an observer training course with room for critique and improvement. This course demonstrates promise in training non-clinician observers to record observations in the OR. Currently, practice observations hold favorable outcomes when comparing the observational skills of trainees to human factors experts. Trainee feedback also reveals strongpoints as well as opportunities for improvement. This course could be the start of the model for observer training in future studies.
Funding Sources
This project was funded under grant number HS026491-01 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of AHRQ. Readers should not interpret any statement in this report as an official position of AHRQ or of HHS. None of the authors has any affiliation or financial involvement that conflicts with the material presented in this report.
New technologies in the healthcare setting carry the potential for improvements in patient outcomes. However, safe implementation of these technologies requires investigations that explore the relationship between new devices and those who use them. Direct observations can help explain complex interactions between technology and healthcare workers. However, reliable and precise observations of the healthcare environment require robust observer training. Few studies have demonstrated a methodology for training observers that enables critique and improvement. In order to achieve reliable, accurate observations, our group has created a web-based observer training course intended to help trainees record flow disruptions (FDs) during live robotic assisted surgeries (RAS). Since this study is occurring at multiple sites over a wide timeframe, the training course needed to be readily available, easily accessible, modifiable, and comprehensive to reflect the needs of the trainee. To verify that the observer has been successfully trained, the structure and content of the course must lead to a review of observational reliability that compares the trainee to an expert.
Our course offers new observers training through modules that overview the aims of this research project, introduce the trainee to RAS in the operating room (OR), explore current literature on FDs in the OR, elaborate on data collection techniques, review the possible implications of findings through other literature, and assess proficiency in identifying, categorizing, and rating FDs with quizzes. The course then instructs the trainee and a human factors expert to conduct joint-observations in order to evaluate interrater reliability (IRR). A novel tool, Research and Exploratory Analysis Driven – Time data Visualization (READ-TV), designed specifically for this research project, assisted in the easy comparison of data collection sheets between observers-in-training and professionals.
This course could be used for a variety of situations and offers a useful framework baseline for improving the rigor and assessment of direct observation methods in future HF healthcare research and practice. Our lecture will describe our experiences designing and testing the course so that other researchers who rely on direct observational data may further develop their training methods.
Application
Conducting direct observations is complex and challenging even for experienced researchers. This course was designed to train non-clinician observers in both the surgical and observation methodology topics necessary to collect observational data. While our content focused on training observers to identify FDs in RAS procedures, this course can be easily modified for a broad range of research projects involving direct observations. Similarly, READ-TV facilitates assessment of IRR and provides a much deeper examination of observer agreement to ensure observers are adequately trained. This course has proved useful for our multisite, multiyear study and can provide a model for similar research projects with its potential to increase observational consistency.
Background
As the OR changes, further research can support more effective use of new systems, devices, and technologies while reducing potential harms. Direct observation of work is one of the only ways to understand work “as done” in real clinical contexts and has informed a range of approaches to studying clinical systems ‘in the wild.’ Observers can come from diverse professional backgrounds – clinical, HF, design, engineering, quality improvement – so ensuring an appropriate level of shared knowledge and observation skill is vital for reliability and validity. Previous studies have trained their observers through approaches such as lectures, coaching, and practice observations. However, the training materials, processes, and evaluations of observers are rarely provided in replicable detail. It can therefore be difficult to see how IRR was achieved on the multiple dimensions that quantitative and qualitative observations are conducted. In deploying direct observation for our studies in robotic surgery, we recognized a more methodological and rigorous approach to observer training and assessment was required. This presented an opportunity to create and evaluate an observer training course that would cover case selection, sampling approaches, and methods for data recording, coding, and analysis.
Overview of Presentation
This lecture will detail why we developed an observer training course, how we chose its platform, and what our course can offer to trainees. We will share what we have learned through this development process so that other researchers might use similar training strategies in studies that rely on direct observations.
Despite the need for comprehensive observer training to ensure reliable data collection, few studies describe training methods and measures for IRR in detail in healthcare. A multisite, multiyear study requires a rigorous approach to the training and assessment of observers prior to collecting usable clinical observation data. This in turn benefits from an easily accessible training course, supplementing their background in human factors and systems analysis in the OR environment and surgical procedures, and training them in reliable recording of observational data.
At first, the OR environment might seem overwhelming for non-clinical observers; they can expect to learn a series of new skills while adapting to new environments. Trainees need to understand the scope of the project, HF in healthcare principles, and data collection techniques. These are not the only necessary competencies to have in the OR. Trainees must learn where to stand, where and when to move, when to speak, and how to gain and maintain the trust of the OR staff. Within our project, our course has helped observers become an accepted addition to the OR capable of recording reliable data. The course opens with an introduction that describes the roles, rules, and reality of the OR along with the basic operations of a da Vinci robot. After becoming familiar with their new work environment, the trainee reads literature prior to their first observation that elaborates on the HF implications of surgical robots. After a data collection guides and quizzes, the trainee attends a set of joint-observations with their expert mentor. The conclusion of the web-based course outlines instructions for IRR review and offers literature and guides for the observer’s contributions to the project outside of data collection. This includes studies that expand on the disruptions the observer may have already detected in the OR.
Both trainees reported that the web-based course was easily accessible, well-constructed, and useful. This course helped familiarize trainees with topics previously unknown to them. Trainees demonstrated growing proficiency as observers despite having no previous experience with these data collection techniques or the OR environment. We are currently in the process of developing additional assessments of skill prior to live observations in the OR which provide automatic feedback, including meeting specific IRR goals.
Conclusion
Direct observations are necessary to understand the interactions between humans and systems in the operating room. Since current literature does not provide detailed methods or standards for observer training, our research group has developed and implemented an observer training course with room for critique and improvement. This course demonstrates promise in training non-clinician observers to record observations in the OR. Currently, practice observations hold favorable outcomes when comparing the observational skills of trainees to human factors experts. Trainee feedback also reveals strongpoints as well as opportunities for improvement. This course could be the start of the model for observer training in future studies.
Funding Sources
This project was funded under grant number HS026491-01 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of AHRQ. Readers should not interpret any statement in this report as an official position of AHRQ or of HHS. None of the authors has any affiliation or financial involvement that conflicts with the material presented in this report.
Authors
Research Assistant Professor
