The real problem with statistics is that people want an answer to an impossible problem. Namely they want to be told the probability of the world being a particular way. But anyone familiar with conditional probability can easily see that there is no way to come up with that answer, because the conditional probability of something after an observation depends on the assumed probability before the observation.
There are multiple approaches to this problem. What frequentist statistics does is answers a different question. Namely, "What is the probability of getting a result this strongly against the null hypothesis if the null hypothesis is true." This is appealing in that it is an objective probability that seems to say something about the problem under discussion. However people consistently read it as, "What is the probability that the null hypothesis is true?" Which is simply wrong.
There is a second problem with frequentist statistics, which is what this article is about. Which is that the objective-looking probability you get depends on the null hypothesis chosen in ways it shouldn't. Basic familiarity with Bayes' Theorem and conditional probability demonstrates that it is impossible for it to matter whether your intention was to flip a coin 6 times, flip it until you get both heads and tails, or flip it until you get heads. This factor cannot affect the conditional probability of the coin being biased. But those three different intentions translate into 3 different calculations, and 3 different answers in frequentist statistics.
So that's the frequentist approach. What is the Bayesian approach? It is to come up with statistics that inform us on how prior beliefs before observation turn into posterior beliefs after observation. The advantage of this method is that it is intellectually honest. The disadvantages are that it is complicated and people notice that it is confusing. (The frequentist approach is confusing as well, but people don't notice their confusion. Instead the confidently draw the wrong conclusion that the null hypothesis has been proven.)
There are other approaches. The article touched on my favorite when it pointed out that we should report likelyhood ratios rather than probabilities. This is absolutely right. The effect of an observation is to modify our prior beliefs, and likelyhood ratios concisely describe how we should modify them. Plus they have the great ability to stack - you can take 3 experiments and combine their likelyhood ratios to come up with the likelyhood ratio for having seen all three experiments.
Unfortunately, though, everyone knows frequentist methods, people accept them, and it is very hard to get people to see what is wrong with them. So alternate approaches, though theoretically superior, face an uphill battle towards acceptance.
The difficulty I tend to encounter in promoting Bayesian statistics among scientists is the "sudden" appearance of a statistical model. Too many people go one further than believing frequentist statistics are answering "What is the probability that H0 is true" but instead just fully over the analysis and believe that frequentist methods tell you, simply, accurately, objectively, whether an experimental treatment is "significant".
If you try to press on what people believe "significant" to mean it gets ugly fast, but it's generally a good thing and definitely necessary to publish. If you don't get significance it's just because you need a bigger n. If you can think of some factors or covariates then you really need to use ANOVA.
Stating that there is anything more complex to looking at data and deciding what it means is practically unthinkable.
Likelihood ratios are definitely nice, though. I've sort of gotten people to think about it at a high level by talking about "information flows" and log likelihood values.
A separate problem, not dealt with in that particular essay, is the quite hideous degree to which average scientists don't understand the statistics they use.
I would put a good deal of the blame for this squarely on frequentism as well. Bayesianism isn't hard to understand, it's just takes an effort of the teacher to explain well - I've made certain notable efforts in that direction myself. Once you do get it, you get it.
I think, roughly, the blame goes out to Fisher and anyone else who promoted the "Recipe for Understanding the World" style statistics. It's not that people are being blocked by their understanding of complex frequentist methods but instead the idea that they don't need to understand anything more because statistics is just a black box you use for verification.
Insert results, get a green or red "significance" light, move on.
I think you're on to something regarding "significance." Over in my dept. we like to say that significance is a measure of sample size. The question, then, hinges on whether or not something has practical significance. Because we've built the whole research reputation incentive structure on the .05 significance level, studies can be designed to get that.
The punchline seems to be, well, that there's also a large human element that contributes to the problem. I think it's one thing to rail on the Frequentist way-of-thinking; it's entirely another to state that the institution of scientific research built on it creates unwanted incentives.
The relationship between significance level and sample size is reliant on a complex set of assumptions to say the least, and, when everything is stripped away, is perhaps best seen as a way of discovering just how difficult it will be to deblur the world. What power prescription we need.
Often (always?) these constraints are all so very much more complex than Gaussian power analysis states. You do it as a way of sketching the depth of a problem I think, not much more.
The linked paper is a pretty clear introduction of the high level problems. I think it's perhaps a little more grim than necessary, but then again that might just be my own bias.
There are multiple approaches to this problem. What frequentist statistics does is answers a different question. Namely, "What is the probability of getting a result this strongly against the null hypothesis if the null hypothesis is true." This is appealing in that it is an objective probability that seems to say something about the problem under discussion. However people consistently read it as, "What is the probability that the null hypothesis is true?" Which is simply wrong.
There is a second problem with frequentist statistics, which is what this article is about. Which is that the objective-looking probability you get depends on the null hypothesis chosen in ways it shouldn't. Basic familiarity with Bayes' Theorem and conditional probability demonstrates that it is impossible for it to matter whether your intention was to flip a coin 6 times, flip it until you get both heads and tails, or flip it until you get heads. This factor cannot affect the conditional probability of the coin being biased. But those three different intentions translate into 3 different calculations, and 3 different answers in frequentist statistics.
So that's the frequentist approach. What is the Bayesian approach? It is to come up with statistics that inform us on how prior beliefs before observation turn into posterior beliefs after observation. The advantage of this method is that it is intellectually honest. The disadvantages are that it is complicated and people notice that it is confusing. (The frequentist approach is confusing as well, but people don't notice their confusion. Instead the confidently draw the wrong conclusion that the null hypothesis has been proven.)
There are other approaches. The article touched on my favorite when it pointed out that we should report likelyhood ratios rather than probabilities. This is absolutely right. The effect of an observation is to modify our prior beliefs, and likelyhood ratios concisely describe how we should modify them. Plus they have the great ability to stack - you can take 3 experiments and combine their likelyhood ratios to come up with the likelyhood ratio for having seen all three experiments.
Unfortunately, though, everyone knows frequentist methods, people accept them, and it is very hard to get people to see what is wrong with them. So alternate approaches, though theoretically superior, face an uphill battle towards acceptance.