April 25, 2019 - Boulder, CO, US
As we discussed in an earlier news article, focusing solely on p-values when performing statistical analysis can be misleading and doesn’t make best use of all the information in the data. These concepts have been further reinforced in a recent special issue in The American Statistician journal titled Statistical Inference in the 21st Century: A World Beyond p < 0.05. Their recommendations were summarized by the editors as follows, “We summarize our recommendations in two sentences totaling seven words: Accept uncertainty. Be Thoughtful, Open, and Modest. Remember ATOM.” Much of what’s proposed by the editors is fundamental to the practice of risk analysis, where the explicit modeling of uncertainty and the use of more diverse and flexible statistical distributions has been advocated for decades, so we certainly agree with their recommendations.
With the availability of modern methods such as Bayesian statistics and Monte Carlo simulation, we have much more flexibility to produce results that are relevant to our audience. In our case, being a consultancy we are typically asked to use modeling to support decisions that affect an institution’s goals, so rather than using p-values we talk about confidence in the superiority or optimality of different actionable options, rankings between them, and the magnitude and impact of those options. A hypothetical example could be to inform if option A is better than option B to reach a company’s growth target based on an analysis of the means using historical data. Using a one-sided test (A>B) one could only say, “we reject the null hypothesis that A=B, (P<.05)”, and perhaps state that option A yielded an average $5M more revenues than option B. At this point we would be escorted out of the client’s offices as this information would be largely useless to the decision maker. If we instead use Bayesian methods, an alternative answer could be “based on historical information we are 99% confident that your revenues will increase by at least $5M if you stay with option A…but there is also a 25% chance that future revenues would be higher if you implement option B”. With this information together with the costs of each option and the risk appetite of the institution, we can incorporate a decision-analytical framework to help the institution decide what’s the optimal option. Furthermore, one can then use simulation methods to create different scenarios going forward and see how the conclusions from historical data could be affected by variables we might encounter in the future. We invite you to review ModalAssist, which includes a repository of freely available applied example models illustrating these and other principles.