The Problem with “Magnitude-Based Inference”

Purpose A statistical method called "Magnitude-Based Inference" (MBI) has gained a following in the sports science literature, despite concerns voiced by statisticians. Its proponents have claimed that MBI exhibits superior Type I and Type II error rates compared with standard null hypothesis testing for most cases. I have performed a re-analysis to evaluate this claim. Methods Using simulation code provided by MBI's proponents, I estimated Type I and Type II error rates for clinical and non-clinical MBI for a range of effect sizes, sample sizes, and smallest important effects. I plotted these results in a way that makes transparent the empirical behavior of MBI. I also re-ran the simulations after correcting mistakes in the definitions of Type I and Type II error provided by MBI's proponents. Finally, I confirmed the findings mathematically; and I provide general equations for calculating MBI's error rates without the need for simulation. Results Contrary to what MBI's proponents have claimed, MBI does not exhibit "superior" Type I and Type II error rates to standard null hypothesis testing. As expected, there is a tradeoff between Type I and Type II error. At precisely the small-to-moderate sample sizes that MBI's proponents deem "optimal," MBI reduces the Type II error rate at the cost of greatly inflating the Type I error rate—to two to six times that of standard hypothesis testing. Conclusions MBI exhibits worrisome empirical behavior. In contrast to standard null hypothesis testing, which has predictable Type I error rates, the Type I error rates for MBI vary widely depending on the sample size and choice of smallest important effect, and are often unacceptably high. MBI should not be used. This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. Corresponding Author: Dr. Kristin L. Sainani, Department of Health Research and Policy, 150 Governor's Lane, HRP Redwood Building, Stanford, California 94305. The author did not receive financial support and has no conflicts of interest to disclose related to this work. The results of the study are presented clearly, honestly, and without fabrication, falsification, or inappropriate data manipulation. The results of the present study do not constitute endorsement by ACSM. Accepted for Publication: 6 April 2018 © 2018 American College of Sports Medicine

from Sports Medicine via xlomafota13 on Inoreader https://ift.tt/2qPTmki
via IFTTT