Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments.

Ghost hunters and anti-vaxxers have better science behind their pet theories.

We expect this type of endeavour to be extraordinarily prone to influence from noise. The entire modern process of science has been built to control for external factors that Roberts is ignoring or outright embracing.

And no one should be surprised when he finds links between brain function and dental amalgam, which is both preposterous and specific to an existing hypothesis common among other pseudosciences. Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias?

That doesn't mean that some of the things he has found aren't true but 100+ years of trial and error and deep thought on this single issue by an enormous number of intelligent people taking the most meticulous records humanity has ever produced all around the world have determined that his methods will produce more false correlations than true ones.

The failure to produce previous controlled experiments is alarming. I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

> Roberts is collected one long detailed case study. Sample size of 1, no controls, no blinding and tiny experiments.

You might want to actually read what Roberts collects, rather than what you think he collects, because it is apparent that you extrapolate from this post. It's a good way to stroke your ego, but not a good way to have a discussion.

> Isn't this just more evidence that the unblinded nature of Roberts anomaly hunting is being skewed by bias?

No, it isn't, and if you read what he publishes rather than have a knee-jerk reaction to one short summary, you might be able to appreciate that.

> have determined that his methods will produce more false correlations than true ones.

That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer)

What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science".

> The failure to produce previous controlled experiments is alarming.

What exactly are you referring to?

> I'm just baffled as to why you think that removing controls and oversight and experimenter blinding and lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well. He has stumbled on a diet that seems to be more effective than any other out there (in real world terms, that of keeping weight off years later). It only works for 80% of the people, but for those it works wonders. If you can describe a plausible placebo mechanism for this result, please do - I would love to hear one (and why it seems to skip every other popular diet out there)

I do believe that a cheap, safe, easy to do experiment that is not blinded is infinitely more useful than an expensive one that will never be carried out.

The first is not perfect, but yields results that are easy to later work with, and easy to verify whether they are right or wrong by doing the same test again or other test that would pinpoint the effect. It is a weak result, but it is a result of some kind.

The second yields zero information, since it will never be done.

> lowering the sample size, TO ONE, as well as taking a single data point for some "experiments" would in any way make it better?

You don't have to agree with what Roberts is doing. But you will have to excuse me for pointing out that making comments based on what you infer he is doing or claiming from a ten line summary makes you look like an idiot, even if it makes you feel superior.

> That's actually not true, as far as I know, and I would appreciate if you specified exactly what you are referring to. (saying "everyone knows" is not a good enough answer)

Let's take one, specific, example: Blinding. So you think that science added the concept of blinding and double blinded experiments because it didn't result in an improvement in results and a decrease in bias? And a sample size of more than 1? Are you saying there is no evidence that a sample size larger than one leads to better results?

What about my statement that having, rather than not having, basic scientific controls in place results in more trustworthy evidence is "actually not true"

> What I do know is that this meticulous record keeping produces bogus results at an alarming rate because of the things it does not consider, like "publish or perish" pressure and the "negative reuslts go in the drawer" effect. This xkcd describes the mechanism http://xkcd.com/882/ , and these extremely well written, peer-reviewed papers, show that most published results (which are golden according to the school of thought you seem to belong to) is actually wrong: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.reuters.com/article/2012/03/28/us-science-cancer-.... (points to a paywalled nature article). In case you are unable to get a copy of the nature paper: 88% of celebrated cancer results published in the last 10 years CAN NOT BE REPLICATED (that is, are most probably WRONG), despite following the "golden rules of science".

> Because I actually live in the real world and not in some fantasy world. I don't think removing controls and oversight is a good idea. However, Roberts has no funding for these experiments because there's no way to make money off them. So he does what he finds interesting, in the best way he can, and lets others try to replicate -- and then collects their results as well.

Which side are you arguing? So science is hard and there are a ton of shitty results accepted as valid. But you seem to be suggesting that the solution to this is good intentions and a scrappy can-do attitude. If the well funded experiments with large sample sizes, double blinding and excellent controls still screw it up most of the time because of some bias or error in procedure then why would I expect removing all controls would result in something even worth looking at?

PS Just to clear something up, in science unreplicated experiements are weak evidence, most are expected to be wrong. replicated experiments are stronger. and theories with multiple lines of evidence from multiple experiments in multiple fields and have emerged as the consensus opinion among experts of how things work are strong. The way you pointed out that unreplicated papers are probably wrong seemed like you thought you had some kind of gotcha. Well the "golden rules of science" are pretty clear on the value to be placed on unreplicated papers in a field overrun by media attention.

> Blinding. So you think that science added the concept of blinding and double blinded experiments because it didn't result in an improvement in results and a decrease in bias?

There's theory, and then there's practice. In 3 clinical experiments that I've followed closely, it was officially "double blinded" (and at least one was published as such), but because the placebo was sugar, and the substance or treatment under test was not inert (one caused dry mouth, one caused flushing, and in one laser treatment, people found the real thing painful, unlike the placebo light). So the patients knew about it for sure, and the experimenters had to be idiots to not know.

Do you think the published results eventually reported that? The answer is no. That would have killed FDA approval, and potentially cost hundreds of millions of dollars. Is that scientific fraud? I would say yes, but you'd be surprised at the answers you'd get if you ask the people involved (rather than the theoretical question).

Do you think this situation is uncommon? I don't. I might be biased here by my own experience, but I would assume that when hundreds of millions of $ are on the line, that science takes a back seat. FDA slap-on-wrist fines seem to agree with that.

> Are you saying there is no evidence that a sample size larger than one leads to better results?

No, I was not saying that. I was referring to the "hundreds of years, hundreds of smart people" statement which was vague and did not point to anything in particular.

> But you seem to be suggesting that the solution to this is good intentions and a scrappy can-do attitude. If the well funded experiments with large sample sizes, double blinding and excellent controls still screw it up most of the time because of some bias or error in procedure then why would I expect removing all controls would result in something even worth looking at?

Read what you just wrote. Science's "golden standard" is essentially unattainable: 88% of the most (assumed) useful results about cancer in the last 10 years cannot be replicated (therefore, scientifically wrong , or in your own words, shitty results accepted as valid). Read that again. And again, and let that sink. That's not an accident and that's not just because science is hard. That's a direct result of the way science is practiced these days (The xkcd is the super-short summary of one of the reasons, the Ioanaddis piece is the "scientific" version with many identified reasons).

Guess what - if you do not change how science is practiced, you're guaranteed to get more shitty results. In fact, you're guaranteed to get a higher percentage of them with time (I don't remember if this piece by Ioanaddis touches that, but he did in other places). People in science I've talked to expected a 50% "wrong result" ratio from the golden standard that's supposed to give you good faith. No one expected the 88% failure result, but no one thinks it is exaggerated.

Again, these conclusions (results done the same double blind, N>>1) are going to get shittier and shittier, unless you change something about how you do stuff.

> The way you pointed out that unreplicated papers are probably wrong seemed like you thought you had some kind of gotcha.

Question, if Seth did acceptable double-blind N=30 (assume sufficient) experiments, would have pointed out that "well, that's not scientific because it wasn't replicated?", if so - kudos. But you'd be hard pressed to accept any nutrition or medical result published in the last 30 years, because the vast majority does not comply with that standard. Do you have that response to any scientific news you hear?

If indeed you hadn't replied like that, then you are moving the goalposts. But I'll give you the extremely unlikely benefit that you would have.

> Well the "golden rules of science" are pretty clear on the value to be placed on unreplicated papers in a field overrun by media attention.

That is wishful thinking. For a few years, I read the medical literature and followed how things became standard practice (my SO at the time was a doctor, I'm an engineer). Results such as described by Ioannadis and the Amgen guy become accepted in medicine and nutrition without any attempt at replication. And then pulled back 10-20 years later when people are dying of e.g. Vioxx, or statins (the sh*t has to hit the fan on this one; wait another 10 years), or realize that most of the Prozac data was cooked to look much more significant than it was.

> Which side are you arguing?

I'm arguing medical/nutritional science as practiced today (especially its relationship with "publish or perish" academia and "a bad result will cost us upward of $400M" pharma) has painted itself into a corner. The incentive structure essentially guarantees that the percentage of wrong answers is going up. Replication is part of the "gold standard", but is rarely ever practiced. (The Amgen paper is the only modern systematic attempt at replication that I'm aware of; and of the few casual replication attempts that fail, most don't get published).

It's converging into an insane "what must do this, even though we don't have the facilities, but anything else is rotten, so we'll pretend it's ok and not bother replicate unless someone points out we're wrong".

What I'm arguing is not "science is hard" (it is!) or "science is easy" (it isn't). It is "we need to find a new way to do science". The golden "double blinded, n->inf, multiple replication" is the easiest to be convinced of, if done right, but definitely not the only way to do science, and definitely not often done right (and with the existing incentive structure, will be done less and less so).

I am arguing that we should find other ways to make progress, and I think Roberts is on the right track. Following him for the last 7 years, I know that he produced more useful results (safe, easily testable, showing improvement for a large percentage of people who attempt to follow) than most hundreds-of-million-dollars scientifically-golden experiments. And he did it in his free time, and with no funding.

The way science is done needs a disruption. Double blind n->inf multiple independent confirmation is indeed the easiest way to be convinced of a result, but it is NOT the only way, and it is becoming increasingly rare/impossible to carry out.

What I'm arguing is that the gold standard is actually not achieved; You don't disagree, I think. What we disagree about is how meaningful the results that are coming out are compared to the effort/cost. I believe that the existing tradeoff is not acceptable; you seem to believe there is no tradeoff as there is only one way to do science (it's just that the ideal is not achieved).

What I'm arguing is that Roberts is giving an example about how the way science is done could be disrupted. It's immature, it's not easy to replicate (I'm not as placebo-resistant as Seth is, for example). But it has applicable results, and is cheap and effective.