Measurement of Patient Safety

Background

The 1999 Institute of Medicine To Err Is Human report drew national attention to the problem of preventable harm in medicine and led to the creation of the modern patient safety field. Yet little of the material in To Err Is Human was new—the key studies on which the report (and its headline-grabbing estimate of up to 98,000 deaths every year due to preventable harm) was based were performed many years before the report was issued. The reason To Err Is Human had such a profound impact was largely because it was released from the IOM and estimated, for the first time, the toll of medical errors at the national level. Since the report was issued, several studies using similar retrospective chart review methodology have found that adverse events remain common and have likely decreased only slightly over time.

The impact of To Err Is Human demonstrates the importance of rigorously measuring the incidence and prevalence of preventable harm. However, measurement of patient safety is complex, and, while several different methods may be used, there is no single validated method for measuring the overall safety of care provided in a given health care setting. One commentator compared safety measurement to the fable in which five blind men describe an elephant in widely varying terms (as a wall, fan, spear, snake, or tree), depending on which part of the animal they touched. In this analogy, an institution's view of its safety issues inevitably depends on the method being used to measure safety, and a comprehensive picture can only be obtained by integrating multiple methods.

This primer will review methods of measuring patient safety. A related Patient Safety Primer discusses Detection of Safety Hazards.

A Framework for Measuring Patient Safety

In the 1960s, health services researcher Avedis Donabedian defined a taxonomy for measuring the quality of health care. The "Donabedian triad," which is still widely used today, defines three lenses through which quality may be viewed:

Structures—how care is organized
Processes—what is done to the patient
Outcomes—what ultimately happens to the patient

A structural measure of patient safety might assess whether a hospital has key resources in place to improve safety, such as an electronic health record or a mechanism to rapidly start the work of root cause analysis teams after a serious adverse event has occurred. Process measures assess adherence to safety standards, such as the proportion of surgical patients for whom a postoperative checklist is completed or the proportion of patients in a hospital receiving appropriate prophylaxis for venous thromboembolism. Outcome metrics can measure the incidence or prevalence of adverse events—harm experienced by patients as a result of interaction with the health care system. A related Patient Safety Primer on Adverse Events, Near Misses, and Errors discusses the definition and types of adverse events in more detail.

The choice of measurement method also depends on the reason measurement is being performed. Measurement is used for a variety of purposes: to evaluate the effectiveness of safety interventions, identify new or emerging safety threats, compare safety across hospitals and clinics, or to determine whether patient safety is improving over time. There is no one-size-fits-all approach to measurement—the choice of metric varies depending on the purpose of measurement. For example, studies of missed nursing care measure processes—the frequency with which required care elements are not completed. Studies of medication reconciliation may measure processes (such as the proportion of patients for whom a best possible medication history was documented at hospital admission) or outcomes (such as the preventable adverse drug events). The Leapfrog Hospital Survey evaluates hospitals based on structural metrics—the use of specific patient safety practices, such as computerized provider order entry.

Methods for Measuring Patient Safety

Several methods that can be used for measuring patient safety events are described in the Table below.

Table. Representative Examples of Safety Measurement Strategies

Measurement Strategies	Advantages	Disadvantages
Retrospective Chart Review (by itself or after use of a trigger tool)	Considered the "gold standard," contains rich and detailed clinical information	Costly, labor-intensive, data quality variable due to incomplete clinical information, retrospective review only. Efficiency improved by focusing chart reviews on cases identified by a reliable trigger tool or software tool
Voluntary Error Reporting Systems	Useful for internal quality improvement and case-finding, highlights adverse events that providers perceive as important	Capture a nonrepresentative fraction of adverse events (in hospitals, most reports are submitted by nurses; relatively few by doctors), retrospective review only based on provider self-reports
Automated Surveillance	Can be used retrospectively or prospectively, helpful in screening patients who may be at high risk for adverse events using standardized protocols	Need electronic data to run automated surveillance, high proportion of triggered cases are false positives
Administrative/Claims Data (e.g., AHRQ Patient Safety Indicators)	Low-cost, readily available data, useful for tracking events over time across large populations, can identify potential adverse events	Lack detailed clinical data, concerns over variability and inaccuracy of ICD-9-CM and ICD-10-CM codes across and within systems, may detect high proportion of false positives and false negatives
Patient Reports	Can capture errors not easily recognized by other methods (i.e., errors related to communication between providers)	Measurement tools are still in development

(Adapted from Wachter RM. Understanding Patient Safety, Second Edition. New York, NY: McGraw-Hill Professional; 2012. ISBN: 9780071765787.)

Retrospective chart review using a two-stage process was originally developed for the pioneering Harvard Medical Practice Study to detect and measure adverse events in hospitalized patients. In the first stage, screening criteria (which may include trigger tools) are used to identify cases where harm may have occurred. These cases are reviewed in depth, ideally by two independent clinicians, to determine whether harm occurred, the severity of injury to the patient, and if the harm was preventable. In addition to the advantages listed above, chart review can be used to measure process or outcome metrics and can reliably measure the frequency of specific adverse events. It is also useful for comparing estimates of overall patient safety events between hospitals or over different time periods. This method has been used in several influential studies of the prevalence of safety problems over time. Another seminal study by AHRQ used the Medicare Patient Safety Monitoring System to assess temporal trends in patient safety. This process also uses retrospective medical review with defined algorithms for detecting 21 measures of safety that can be reliably abstracted from medical records.

Problems and Controversies in Measuring Patient Safety

Despite the importance of accurately measuring adverse events, existing tools all have limitations, and controversy continues to plague efforts to measure safety and compare safety between organizations. Retrospective chart review using trigger tools or well-defined specific adverse events is often used in research studies, but it is so labor-intensive that most hospitals do not routinely monitor safety performance in this fashion. Other commonly used methods, such as voluntary error reporting systems and the AHRQ Patient Safety Indicators, are useful for screening purposes but cannot be reliably used for measuring incidence or prevalence of most safety problems. Certain types of errors—such as diagnostic errors—still lack standardized and reliable measurement strategies, and studies have shown that variation in how medication errors are defined can result in widely varying estimates of error prevalence. As a result, 15 years into the safety movement, evaluating the effectiveness of safety programs remains a challenge for most organizations.

Even when adverse events can be measured, an additional layer of controversy concerns determining whether the event was preventable. In the seminal studies that formed the basis of To Err Is Human, experienced clinicians often disagreed on whether an error was preventable. Differences in definitions of errors account for some of the wide variation in estimates of the proportion of hospitalized patients who experience preventable harm, which ranges from about 12% among Medicare patients in the 2010 Office of the Inspector General study to nearly 33% in a 2011 study.

Current Context

Progress has been made toward measuring trends in safety at the national level. The Partnership for Patients—a public-private initiative launched in 2011—uses a combination of 28 metrics to determine a national Hospital-Acquired Conditions (HAC) rate. The HACs include certain health care–associated infections, medication errors, and never events. Analysis by AHRQ has demonstrated a consistent and sustained decline in HAC rates since the inception of the Partnership for Patients.

Nevertheless, accurate and reliable measurement of errors and adverse events remains a major challenge for the patient safety field. The 2015 Free From Harm report by the National Patient Safety Foundation called for the creation of a "common set of safety metrics that reflect meaningful outcomes" as one of eight recommendations for advancing patient safety. Their specific recommendations included establishing a standardized set of process and outcome measures for use on a national basis (as well as retiring outdated measures), creating measures of patient safety for settings outside the hospital, improving the quality of safety reporting systems, and developing ways of measuring safety in real time (as opposed to retrospective measurement). The AHRQ Common Formats were developed in order to "help providers uniformly report patient safety events and to improve health care providers' efforts to eliminate harm." The Common Formats represent an important step toward achieving the goal of developing a universal framework for measurement of safety processes and outcomes.