#multiple comparisons

Introduction In the pursuit of justice, forensic science plays a pivotal role in uncovering the truth and securing convictions. However, a recent study published in the esteemed Proceedings of the National Academy of Sciences (PNAS) has brought to light a concerning issue that could potentially undermine the credibility of forensic evidence in courtrooms. The research delves into the intricate…

This deserves to be read, re-read, re-re-read, and taken to heart. The ASA makes their statement on p-values. Some key points:

p-values by themselves offer very little information on the utility of a model, the truth of a statement, or what is behind a data process

Full documentation and transparency of data collection, handling, and analysis are essential to interpret results

This full transparency has not taken place historically for many reasons

Multiple comparisons, even if done informally, affect the interpretation of a p-value

Replicability and reproducibility are essential for scientific progress, and this full transparency is essential to replicability and reproducibility

Hopefully this statement will drive field journals, professors, and regulators to consider more than the p-value (the FDA in fact already does consider more than the p-value, but they still want to see that 0.05 threshold met twice in most cases).