First of all, do we have a clear evaluation criterion for "better" outcomes? If the question is about writing better code, is there agreement at the start of an experiment about what "better" means? I'm not experienced in evaluating software code (I'm much more experienced in evaluating psychological and medical research), but it seems to me, from reading HN, that better code could be code
a) that is able to pass all the unit tests with an earlier deadline,
b) that has a smaller total bug count when subjected to code review,
c) that solves a problem that other programmers in the same workplace didn't solve until later,
d) that works around an intellectual property claim by a competing company more convincingly,
e) that meets company stylesheet requirements more exactly,
f) that can be maintained by less skilled subsequent programmers,
and perhaps other criteria.
Any sound experimental study has to meet a lot of other criteria
to be worthy of believing, and in human drug studies the criteria include
1) sufficiently large sample size (n = 1 just doesn't do the job)
2) treatment-control design (so that some subjects of the experiment get the drug, and some subjects do not)
3) double-blind administration (the subjects of the experiment should not know if they are getting the drug, nor should the experimenters know which subjects are getting the drug)
and many more, especially
4) rigorous statistical analysis afterward.
As Richard Feynman said, "The first principle is that you must not fool yourself--and you are the easiest person to fool. So you have to be very careful about that."
So anecdotes along the lines of "I took this drug, and I wrote much better code" can be completely discounted, because we don't have access to the coder's before-and-after work product (nor to the coder's detailed dosage history) to know what really happened.
http://www.sciencebasedmedicine.org/index.php/about-science-...
First of all, do we have a clear evaluation criterion for "better" outcomes? If the question is about writing better code, is there agreement at the start of an experiment about what "better" means? I'm not experienced in evaluating software code (I'm much more experienced in evaluating psychological and medical research), but it seems to me, from reading HN, that better code could be code
a) that is able to pass all the unit tests with an earlier deadline,
b) that has a smaller total bug count when subjected to code review,
c) that solves a problem that other programmers in the same workplace didn't solve until later,
d) that works around an intellectual property claim by a competing company more convincingly,
e) that meets company stylesheet requirements more exactly,
f) that can be maintained by less skilled subsequent programmers,
and perhaps other criteria.
Any sound experimental study has to meet a lot of other criteria
http://norvig.com/experiment-design.html
to be worthy of believing, and in human drug studies the criteria include
1) sufficiently large sample size (n = 1 just doesn't do the job)
2) treatment-control design (so that some subjects of the experiment get the drug, and some subjects do not)
3) double-blind administration (the subjects of the experiment should not know if they are getting the drug, nor should the experimenters know which subjects are getting the drug)
and many more, especially
4) rigorous statistical analysis afterward.
As Richard Feynman said, "The first principle is that you must not fool yourself--and you are the easiest person to fool. So you have to be very careful about that."
http://www.lhup.edu/~DSIMANEK/cargocul.htm
So anecdotes along the lines of "I took this drug, and I wrote much better code" can be completely discounted, because we don't have access to the coder's before-and-after work product (nor to the coder's detailed dosage history) to know what really happened.