Well, for one, because the standard for publication in many fields is “there’s a less-than-5% chance that we observed these effects because of coincidence”.

Combine that with people not publishing negative-results studies, and all it takes is someone doing 20 studies (or worse, 20 analyses on the same data set) in order to find that 1-in-20 chance occurring...by chance.

Of course, this has little to do with research itself and much more to do with the standards researchers hold each other to.

While you're right about the 5%-or-less chance, it's still a 5% chance we're aiming for. If we end up with 40% over a large number of papers, something went wrong.

Not publishing the negative results shouldn't affect this number. They're not included in the 5% chance in the first place.

This is a common misunderstanding of p-values.

There is a 5% chance of observing the effect, if the effect is not there.

I think the difference is best illustrated by an example:

If you have 20 researchers investigating a hypothesis that turns out to be false 1 of them will report evidence for a false hypothesis, which will likely not replicate. Thus, if for every true hypotheses investigated, 20 false hypotheses are investigated, and the 19 researchers with false hypotheses do not report their result, that means that 1/2 of the reported positive results will not replicate.

That's a cyclical argument though to what the parent asked though.

"Research is OK to have ac40% non-replication rate" because "the standard for publication in many fields is low"

What is not being considered here at all is that different sciences are different.

First, you have purely theoretical sciences. "Math" (even though quite a bit of these disciplines are actually not quite in math, but in physics, economics, philosophy or computer science. There's others). Obviously they're not just replicate-able, they would never get published if they weren't. Furthermore replicate-ability is absolute, because it's theoretical.

Then you have positivist sciences. Essentially Physics and Chemistry. These are sciences where you can actually experiment, and therefore you can have bounds (like the 5/6 sigma bound in Physics). Replicate-ability is not absolutely guaranteed like in Maths, but it's going to work.

Then you have statistical sciences, like medicine, climate science, experimental economics where due to practical limitations (need actual people that might die as a result of the study, we don't have many test-planets, ...) the number of examples is extremely limited, and in general, far more limited than the complexity of the system would require according to actual statistics.

So replicate-ability is going to take a further BIG hit.

And then you have the sciences where we are studying the existence of a phenomenon in the first place. Biology, social sciences, psychology, political sciences, language studies, parts of experimental economics. In these sciences something gets studied because it exists. We study Siberian tigers, because they exist. We read Hamlet because it exists. Replication ? What do you even mean ? Replicate-ability is essentially zero, and everything is just based on judgement of famous characters.

Add to that that quite a few of these sciences (social sciences, political sciences, English) are up-front biased. They start from the point that they want to find/push X. This can be a value system, a way of thinking, or even spelling or grammar. Needless to say, these studies can be criticized from a neutral perspective.

But critically, thinking about it, you will realize: it would not make sense for social studies or English studies to be neutral.

The further you move down the line, the less the demands of the field on replication are, and the more problems replicate-ability has.

And yes, by the time you hit social studies, the norm is far past the point where a physicist would be fired for scientific fraud.

That does not make social sciences useless, it just means that people's evaluation of scientific results needs to go deeper than "oh scientist said X" (or even "consensus is Y").

> something gets studied because it exists

That's an interesting way to look at it

> Replicate-ability is essentially zero, and everything is just based on judgement of famous characters.

That doesn't fit what we see even in the linked article and the older story about psychology replication: All these experiments had methods that could be repeated (that's what the replication project people did), the effects of almost all the experiments were replicated, and for about half the research the effects were as strong as before. That's a long way from methods that are "just ... judgement of famous character" and "essentially zero" replication.

> quite a few of these sciences (social sciences, political sciences, English) are up-front biased. They start from the point that they want to find/push X

What is that based on? Is there someone in those fields that has written about it? Have you done any work in them?

Having some minor, intimate familiarity with these fields, though from an outside perspective, I don't agree. Political science, for example, aims to identify political phenomena and how they function. Every experiment in any science starts with a hypothesis, the theory the experimenter sets out to prove, and generally experimenters have a career based on promoting a certain model or idea. That is true of mathematicians and chemists too.

Also, I wouldn't group English, one of the humanities, with the social sciences.