Temporal Trends and Understanding the Impact of an Artificial Intelligence Abstraction Tool on the Discordance of NHSN and ACS NSQIP Surgical Site Infection Reporting

Authors:
Brayden Seal, Evan Abbey, Samantha Hendren, Anthony Yang, Sanjay Mohanty, Elizabeth Danielson, Ryan Merkow

Body of Abstract:
BACKGROUND

Accurate surveillance of surgical site infections (SSIs) is critical for patient safety, quality improvement, and institutional benchmarking. Although both the Centers for Disease Control and Prevention’s National Healthcare Safety Network (NHSN) and the American College of Surgeons’ National Surgical Quality Improvement Program (ACS NSQIP) monitor SSIs, prior studies have consistently shown substantial discrepancies between the two, with NHSN identifying fewer cases. As data abstraction grows increasingly automated and integrated into the electronic health record (EHR), it remains unclear whether these advances will meaningfully enhance data accuracy or mitigate discrepancies. Our objectives were to (1) evaluate temporal trends in discordance between NHSN and ACS NSQIP and (2) assess the impact of an automated artificial intelligence (AI) abstraction tool on NHSN data accuracy.

METHODS

We analyzed SSIs (superficial, deep, and organ space) captured by NHSN and ACS NSQIP for adults undergoing colon surgery at a large urban academic medical center (2015-2024). Agreement in SSI identification for overlapping encounters was assessed using Cohen’s kappa. Temporal trends in the NHSN–NSQIP incidence gap and Cohen’s kappa were evaluated using weighted least squares regression with inverse variance as weights. In 2023, our institution implemented an EHR-based AI data abstraction tool.

RESULTS

Over the ten-year study period, we analyzed 7,720 patient encounters: 5,242 surveyed by NHSN, 2,478 surveyed by ACS NSQIP, and 2,082 surveyed by both programs. Cumulative SSI incidence was 6.5% (95% CI, 5.8 – 7.1) in NHSN encounters and 10.7% (95% CI, 9.5 – 11.9) in ACS NSQIP. Among overlapping encounters, agreement was moderate (kappa = 0.43). In 2015-2017, NHSN rates were substantially lower than ACS NSQIP, but this gap narrowed beginning in 2018 (Figure). Regression analysis showed the incidence gap decreased by 6% per year (p < 0.05). Yearly kappa values remained approximately 0.3 - 0.5 (p = 0.15). Cohen’s kappa did not significantly change after introduction of the AI abstraction tool (0.366 in 2022 and 0.567 in 2024, p = 0.22). CONCLUSIONS The gap in SSI rates reported by NHSN and ACS NSQIP has narrowed over the past decade, yet concordance remains only moderate even after introduction of AI-facilitated data abstraction. This suggests that although more efficient and reproducible workflows have improved NHSN case capture, subtle inaccuracies persist. Both systems can track meaningful trends, but case-level data from each should guide targeted interventions.