April 25, 2019 –The end of p-values?

April 25, 2019 - Boulder, CO, US

As we discussed in an earlier news article, focusing solely on p-values when performing statistical analysis can be misleading and doesn’t make best use of all the information in the data. These concepts have been further reinforced in a recent special issuespecial issue in The American Statistician journal titled Statistical Inference in the 21st Century: A World Beyond p < 0.05Statistical Inference in the 21st Century: A World Beyond p < 0.05. Their recommendations were summarized by the editorssummarized by the editors as follows, “We summarize our recommendations in two sentences totaling seven words: Accept uncertainty. Be Thoughtful, Open, and Modest. Remember ATOM.” Much of what’s proposed by the editors is fundamental to the practice of risk analysis, where the explicit modeling of uncertaintymodeling of uncertainty and the use of more diverse and flexible statistical distributionsstatistical distributions has been advocated for decades, so we certainly agree with their recommendations.

With the availability of modern methods such as Bayesian statisticsBayesian statistics and Monte Carlo simulationMonte Carlo simulation, we have much more flexibility to produce results that are relevant to our audience. In our case, being a consultancy we are typically asked to use modeling to support decisions that affect an institution’s goals, so rather than using p-values we talk about confidence in the superiority or optimality of different actionable options, rankings between them, and the magnitude and impact of those options. A hypothetical example could be to inform if option A is better than option B to reach a company’s growth target based on an analysis of the means using historical data. Using a one-sided test (A>B) one could only say, “we reject the null hypothesis that A=B, (P<.05)”, and perhaps state that option A yielded an average $5M more revenues than option B. At this point we would be escorted out of the client’s offices as this information would be largely useless to the decision maker. If we instead use Bayesian methods, an alternative answer could be “based on historical information we are 99% confident that your revenues will increase by at least $5M if you stay with option A…but there is also a 25% chance that future revenues would be higher if you implement option B”. With this information together with the costs of each option and the risk appetite of the institution, we can incorporate a decision-analytical framework to help the institution decide what’s the optimal option. Furthermore, one can then use simulation methods to create different scenarios going forward and see how the conclusions from historical data could be affected by variables we might encounter in the future. We invite you to review ModalAssistModalAssist, which includes a repository of freely available applied example modelsapplied example models illustrating these and other principles.

Projects

We believe in seeing to believe.

View our client success stories to read about our projects.

Testimonials

Once in a while, we ask our clients, course participants and co-investigators to provide us with testimonials of their experience working with us. Read what they're saying about us.

Brief History of EpiX Analytics

EpiX Analytics was founded in 2003 in Princeton, NJ under the Vose Consulting name, and among its offerings it included specialized software to perform Monte Carlo simulations. Read More >>