Use and reporting of statistics and statistical methods in the Surgical Infections Journal

Authors:
Daithi Heffernan, Matthew Kwasnik

Body of Abstract:
Background: The proper use of statistical methods in the communication of scientific results has come under increasing scrutiny for a well-documented lack of rigor. As the medical community reflects on its reporting habits and best practices, surgeons must be leaders in this field. We hypothesize that the nature and type of statistics and statistical reporting in the Surgical Infections Journal has improved between the year 2015 and 2023.

Methods: All 2015 and 2023 articles of the Surgical Infections Journal in which statistical analysis was undertaken were reviewed. Excluded articles included editorials, letters to the editor, literature reviews, articles lacking any statistical methods, and those using descriptive statistics only. The nature and type of statical reporting were extracted from these eligible articles. Dichotomous variables were analyzed with Fisher’s Exact Test, and continuous variables were analyzed with a student’s t-test. Results with p-values less than 0.05 were considered statistically significant.

Results: Overall, 188 articles were included, 102 articles from 2015 and 86 articles from 2023.
In 2023, when compared to articles from 2015, though not statistically significant, the reporting of a priori power analyses was higher (15.1% vs 8.8%;p=0.3), while reporting the method used for power analyses was lower (7.7% vs 44%;p=0.1). Furthermore, reporting the level of significance ( ) was lower in 2023 (67.4% vs 79.4%;p=0.07). Among these articles which reported an , the use of “p<0.05” as the cut-off value significantly increased in 2023 (98.3% vs 86.4%;p=0.01). The use of confidence intervals was higher in 2023 (61.6% vs 56.9%;p=0.6). Beyond reporting the actual p-value, 32 additional modalities of p-value (e.g., <0.001, <0.0001, etc.) were used in 2023, increased from 27 additional modalities in 2015. However, the mean number of unique p-value modalities employed per article significantly decreased in 2023 from 2.8 (SD 1.7) to 2.3 (SD 1.7) (p=0.02). Articles strictly reporting the actual p-value was lower in 2023 (18.6% vs 27.5%;p=0.2). The use of graded significance levels denoted by asterisks (e.g. *p<0.05, **p<0.01, etc.) was higher in 2023 (5.8% vs 2.9%;p=0.5). The achievement of significance in at least one variable significantly decreased in 2023 (79.1% vs 92.2%;p=0.01). Overall, it was noted that in 2023 there was a significantly increased number of articles which claimed “non-significant” results despite not reporting an a priori (8.1% vs 1%;p=0.03). Conclusion: We demonstrate no improvement in best practice for statistical reporting. Lack of transparency and consistency in reporting significance levels, power analyses, and actual p-values, coupled with unverifiable claims of “significance,” make it difficult for the reader to assess for themselves the importance of study results to their surgical practice. Surgeons should be the leaders in driving excellence in statistical analysis for clinical studies.