Large-scale empirical optimisation of statistical control charts to detect clinically relevant increases in surgical site infection rates

Iulian Ilieş 1, Deverick J Anderson 2 3, Joseph Salem 1, Arthur W Baker 2 3, Margo Jacobsen 1, James C Benneyan 4 5

Affiliations

PMID: 31704893
DOI: 10.1136/bmjqs-2018-008976

Abstract

Objective: Surgical site infections (SSIs) are common costly hospital-acquired conditions. While statistical process control (SPC) use in healthcare has increased, limited rigorous empirical research compares and optimises these methods for SSI surveillance. We sought to determine which SPC chart types and design parameters maximise the detection of clinically relevant SSI rate increases while minimising false alarms.

Design: Systematic retrospective data analysis and empirical optimisation.

Methods: We analysed 12 years of data on 13 surgical procedures from a network of 58 community hospitals. Statistically significant SSI rate increases (signals) at individual hospitals initially were identified using 50 different SPC chart variations (Shewhart or exponentially weighted moving average, 5 baseline periods, 5 baseline types). Blinded epidemiologists evaluated the clinical significance of 2709 representative signals of potential outbreaks (out of 5536 generated), rating them as requiring 'action' or 'no action'. These ratings were used to identify which SPC approaches maximised sensitivity and specificity within a broader set of 3600 individual chart variations (additional baseline variations and chart types, including moving average (MA), and five control limit widths) and over 32 million dual-chart combinations based on different baseline periods, reference data (network-wide vs local hospital SSI rates), control limit widths and other calculation considerations. Results were validated with an additional year of data from the same hospital cohort.

Results: The optimal SPC approach to detect clinically important SSI rate increases used two simultaneous MA charts calculated using lagged rolling baseline windows and 1 SD limits. The first chart used 12-month MAs with 18-month baselines and best identified small sustained increases above network-wide SSI rates. The second chart used 6-month MAs with 3-month baselines and best detected large short-term increases above individual hospital SSI rates. This combination outperformed more commonly used charts, with high sensitivity (0.90; positive predictive value=0.56) and practical specificity (0.67; negative predictive value=0.94).

Conclusions: An optimised combination of two MA charts had the best performance for identifying clinically relevant small but sustained above-network SSI rates and large short-term individual hospital increases.

Keywords: Infection outbreaks; statistical process control; surgical site infections; surveillance.

Conflict of interest statement

Competing interests: None declared.

Comment in

Applying rigour to the interpretation of surgical site infection rates.
Williams V, Leis JA.BMJ Qual Saf. 2020 Jun;29(6):446-448. doi: 10.1136/bmjqs-2019-009964. Epub 2019 Dec 13.PMID: 31836626 No abstract available.