Prof. Dr. Duane T. Wegener (The Ohio State University, USA) will give a lecture entitled “Frequentist Statistics and Scientific Claims: Statistical Power and Researcher Degrees of Freedom in Single-Study and Multi-Study Empirical Cases”.
When/Where: 20th October 2025, 14h30 – 16h00, at Ispa – Instituto Universitário (Vítor Almada Auditorium).
Abstract:
In traditional research methods, statistical conclusion validity was largely evaluated according to statistical evidence such as p-values produced by typical frequentist statistics.
Over the last 10-15 years, two key reasons have been given for questioning the credibility of results that were statistically significant. That is, methodologists have suggested that, even for results associated with low p-values, a low level of statistical power would be associated with a larger proportion of significant findings that are false (i.e., a high false-finding rate; FFR). Other methodologists have suggested that p-values might not be accurate because of flexibilities in statistical analyses (i.e., researcher degrees of freedom), where researchers analyze data in different ways until significance is reached.
This talk reviews the support for these concerns and their relevance to both single-study and multi-study data in psychology. According to the mathematical bases linking power to FFR, power has only weak influences on FFR in settings most typical for empirical cases in psychology. Researcher degrees of freedom do pose substantial challenges to single-study empirical cases (regardless of sample size) but appear largely incapable of producing sets of methodologically consistent results, even when allowing for substantial selectivity (file drawer) in which studies are presented. Implications for research evaluation are discussed.
WJCR 