Easier Said Than Done – Challenges in Recognizing when Health Technology Contributes to Patient Safety Events
Event Type
Oral Presentations
TimeThursday, April 151:10pm - 1:30pm EDT
LocationPatient Safety Research and Initiatives
As hospitals continue to adopt health technology at a rapid rate, introducing new technologies can also create new areas of risk and possible harm to patients (Singh & Sittig 2016). Evaluating the safety implications of these technologies is both complex and challenging. The identification of technology-related safety events represents a significant opportunity for improvement. However, this step continues to pose challenges for most institutions. When the role of technology is not recognized in these events, opportunities for improvement may be missed. To better understand these factors at our institution, we set out to apply a human factors informed approach to identify ways that technology has played a role in our reporting of hospital safety events (Singh & Sittig 2017, Koppel et al 2016).

A valuable source of data comes from a hospital safety event reporting system where nurses, in the role of safety managers, review and code events voluntarily submitted by hospital staff (Fong et al 2018). Depending on the various codes, certain events are then collated and reported to a state-wide reporting agency, Pennsylvania Patient Safety Reporting System (PA-PSRS), which analyzes the collected data to identify trends and recommend changes in healthcare practices and procedures that may be implemented to reduce the number and severity of future serious events and incidents. A health information technology (HIT) related safety event is one of the categories reported to PSRS. Safety managers, while experts in safety, have varying degrees of experience with health information technology. Our objective was to systematically review safety events that potentially involved health information technology and compare findings to the current method established by PA-PSRS involving reporting of HIT safety events identified by safety managers.

We performed a systematic review of 1 year of safety event data from an academic pediatric medical system. We queried safety event data for events that either included HIT related terms or were coded by the safety manager as involving HIT. Using a classification system derived from the ECRI Health IT Hazard Manager, we reviewed the query results and categorized HIT safety events (ECRI, 2018). Results of the systematic review were compared to the safety manager coding of events.

The systematic review process was developed by a four-member team consisting of two physician informaticists, a safety expert with an advanced degree in health management systems, and a research specialist with an advanced degree in human computer interaction. A classification scheme of HIT safety events was developed based on the ECRI Health IT Hazard Event Reporting Form (ECRI, 2018). A sample of events were reviewed and compared by two team members. Classification discrepancies were addressed and added to a coding rule book. A sample of classified events was measured for inter-rater reliability.

We applied the HIT keyword query to our safety event reporting database between July 1, 2018 to June 30, 2019. The query returned a total of 1,247 reported safety events that either included free text terms associated with HIT and/or were coded by safety managers as involving HIT. A subset of 94 events were reviewed and classified by two team members. Classifications were reviewed, discrepancies were identified and documented in the coding rule book, and results were compared for inter-rater reliability. The overall Cohen’s Kappa for the two raters was 0.654. Reviewers were in complete agreement on classifying events as “not HIT.”

Of the 1247 events, there were significant differences between our systematic review and those identified as health information technology related events by safety managers. Of the 769 events identified in the systematic review as HIT related events, only 194 were classified as health IT and thus reported to PSRS. Additionally, of the 468 that were identified as not related to HIT, 81 were reported to PSRS as a health technology related safety event.

Classification results were as follows: Data Quality (477, 62.03%), Undetermined System Source (146, 18.99%), Usability (67, 8.71%), Decision Support (54, 7.02%), Organizational Factors (20, 2.60%), Local Implementation (4, 0.52%), Vendor Factors (1, 0.13%). The majority of discrepancies of under-recognized HIT events were classified as being related to Data Quality including incorrect or missing information, information in the wrong place, or information on the wrong patient.

Our study found that the current approach involving patient safety managers’ identification of HIT-related safety events, which is based on requirements established by PA-PSRS, may still result in under-recognition of when technology has contributed to a safety event. In addition, while the most common issue identified was “data quality,” review by our team which includes expertise in human-computer interaction and human factors suggested that these data quality issues likely reflect other upstream usability, technical or other sociotechnical issues. This indicates a need to support the culture and improve the understanding of HIT, so that safety event system users and safety managers are more likely to recognize these HIT-related events when they occur.

The implications of under-recognition of health technology and its contribution to safety events are consequential. While all required events were accurately reported to PA-PSRS, in conducting this systematic review, we found opportunities for improved classification by using a health technology and safety framework. When the contribution of health technology to safety events are attributed accurately, appropriate system level improvements in the health technology in question can be developed and implemented. When more of these events are then reported at the aggregate state level, there may also exist the opportunity for feedback to vendors or reporting of patterned events to the FDA for medical devices. Given that other studies have also found high incidence of under-recognition of health IT related safety events, it could be implied that governing bodies may be operating with incomplete information with current processes within voluntary reporting systems. Our findings suggest that using a health technology classification system such as the ECRI Health IT Hazard Manager for reviewing safety events can help improve the evaluation of safety events from a technology lens. Additionally, safety reporting organizations could consider adopting a health technology classification framework to improve the accuracy and consistency of capturing health information technology related safety events. Lastly, given the breadth of sociotechnical interactions identified, human factors and human-computer interaction experts have an essential role in advancing the understanding of the sociotechnical environment in safety moving forward.


ECRI Institute. (2018). Safe Practice Recommendations for Developing, Implementing, and Integrating a Health IT Safety Program. Retrieved from

Fong, A., Adams, K. T., Gaunt, M. J., Howe, J. L., Kellogg, K. M., & Ratwani, R. M. (2018). Identifying health information technology related safety event reports from patient safety event report databases. Journal of biomedical informatics, 86, 135–142.

Koppel, R. (2016). The health information technology safety framework: building great structures on vast voids. BMJ Qual Saf. 25(4), 218-20.

Singh, H., & Sittig, D.F. (2016). Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf, 25(4),226-32.

Sittig, D., & Singh, H. (2017). Toward More Proactive Approaches to Safety in the Electronic Health Record Era. Jt Comm J Qual Patient Saf. 43(10), 540–547.
Patient Safety and Quality Senior Analyst
Principal Human Computer Interaction Speciliast
Senior Medical Director Patient Safety, Attending Physician
Manager Human Factors Engineering & Patient Safety
Associate Chief Medical Information Officer
Chief Medical Information Officer