The idea that TDD involves some kind of blind faith that the tests will generate grand designs and beautiful code is both silly and wrong. You are right about that. Good design and good code require skill, thought, and knowledge irrespective of whether you are using TDD. So as I practice the discipline, I am thinking all the time about larger scale issues, and I am _not_ being blinded to those concerns.
However, the act of writing tests first has a powerful benefit: the code you write _must_ be testable. It is hard to understate this benefit. If forces a level of decoupling that most programmers, even very experience programmers, would not otherwise engage in.
It also has a psychological impact on the programmer. If every line of production code you write is in response to a failing test, you will _trust_ your test suite. And when you trust your test suite, you can make fearless changes to the code on a whim. You can _clean_ it and improve the design without trepidation.
Gaining these benefits without writing tests first is possible, but much less reliable. And yet the cost of writing the tests first is no greater than writing the tests second.
> However, the act of writing tests first has a powerful benefit: the code you write _must_ be testable.
No, the act of writing well-tested code at all has that benefit. Whether you write the tests before or after the code, one at a time or in module-sized groups, writing code that's hard to test has immediate and obvious penalties when you test it (e.g., tedious rework), and you'll quickly learn to avoid those penalties. So just having the discipline to write well-tested code at all forces you to write code that's not only testable but easily testable. This benefit is not unique to TDD.
> It also has a psychological impact on the programmer. If every line of production code you write is in response to a failing test, you will _trust_ your test suite.
It's not enough to trust that your tests actually test your code. You also need to trust that your tests express your desired semantics. And that's harder to do when the semantics is not designed in whatever form and grouping is most natural to its representation but rather is extruded, one test at a time, through the pinhole view that TDD imposes upon programmers.
> And yet the cost of writing the tests first is no greater than writing the tests second.
What you seem to be overlooking is that TDD not only forces you to write tests first but also in tiny baby-steps that cause programmers to focus only on satisfying one test at a time. As a result, the initial code that is written satisfies only a small portion of the system's overall semantics (the portion that's been expressed as tests so far), and a lot of that code ends up having to be reworked when later tests finally uncover other requirements that affect it. This leads to rework that would have been avoidable had the programmers not been blinded to those requirements earlier on.
The problem with TDD isn't so much that it's test first but that it promotes a pinhole view of subjects that are not narrow.
"However, the act of writing tests first has a powerful benefit: the code you write _must_ be testable. It is hard to understate this benefit. If forces a level of decoupling that most programmers, even very experience programmers, would not otherwise engage in".
I'm sorry but I really find that sentiment to be totally inaccurate. I've never seen a good experienced programmer write hard-to-test or coupled code, regardless of whether they are using TDD or not. A hallmark of when makes them good is that they all have some testing methodology that enforces this and works for them. TDD is one, but there are many others (and yes, that includes good manual-only testing).
I also don't see why you believe TDD is the only way to successfully refactor code, or that only developers who use TDD continually refactor their code to eliminate technical debt and increase productivity. Again, every good programmer does this. TDD is one way to get there. It is not the only way.
However, the act of writing tests first has a powerful benefit: the code you write _must_ be testable. It is hard to understate this benefit. If forces a level of decoupling that most programmers, even very experience programmers, would not otherwise engage in.
It also has a psychological impact on the programmer. If every line of production code you write is in response to a failing test, you will _trust_ your test suite. And when you trust your test suite, you can make fearless changes to the code on a whim. You can _clean_ it and improve the design without trepidation.
Gaining these benefits without writing tests first is possible, but much less reliable. And yet the cost of writing the tests first is no greater than writing the tests second.