It doesn’t even have to be a big construction project to hit budget and schedule issues. Rebuilding our house had us going right back to the drawing board having to get new plans drafted and approved. It worked out much better in the end but the unforeseen cost and time overrun was quite anxiety inducing at the time.
But yeah the idea anyone has magically solved accurately predicting the future is completely bonkers.
I like to remember "plans are useless but planning is indispensable". The plan is useless because of the challenge of predicting the future. But the planning is useful because trying to predict the future gets you asking good questions.
What is then terrifying to me then is when I find myself working with people who manage to, by way of treating "estimates" as "quotes", make it so demoralizing to plan anything--as it will be "used against you" later (which they do even if they were adamant they wouldn't)--that everyone does everything they can to discourage and sabotage any attempt at planning, because planning is valuable :(.
Also because planning gets you to make your desires explicit and to put them in a useful framework. Unexpected problems, disruptive as they may be, don't usually derail the whole plan. Some parts may need to be rethought, but others are still valid, and the whole thing keeps you focused.
There was a submission on this (in the annals of HN) where someone made the point that, more often than not and under normal project scenarios, unexpected problems usually do derail the entire software engineering plan.
The gist being: estimates are given with zero as a lower bound, and some upper bound. In reality, there is no upper bound. Software engineering can and does involve unexpected problems that would take infinite time to solve.
Ergo, project timelines are more accurately averaged between {min_time} and {infinite}, and, given that, remaining risk is the only true metric that should be reported. I.e. how far are we along in terms of project completed, such that we can say only x% of the project remains, and therefore that 1-x% did not explode.
The fundamental problem with estimates is that the recipient believes that the estimate will always turn out to be true, which is obviously not correct.
So, why can't we express the upper bound in combination with uncertainty?
Example: "There is a 50% chance that the feature will be delivered within 1 month."
As time progresses confidence starts increasing and we get closer and closer to how much time it actually takes (be that 5 days, 5 months or 5 years). On the day the feature is completed, confidence is 100% because the real delivery date has become known.
Programming is isomorphic to theorem proving (according to Curry-Howard).
Therefore trying to estimate a software project is a bit like trying to estimate how long it will take to prove a new theorem in mathematics, assuming you would know which exact theorem you want to prove.
In a way I think yes but just getting the type-check right would be like proving ANY mathematical theorem. It is very easy to prove mathematical theorems if you don't care what those theorems are.
The challenge in mathematics, and programming, I believe, is what ("lemmas") to prove so that you can then prove more things that are useful for engineering, science, etc. . What functions to write in your program so that you can get its "main function" to do something useful.
Add to that the fact that much of the "type-checking" with current practical programming languages still has to be done informally, to describe what we expect of the result and then prove (to yourself and the client) that the program does that.
A lot of the thought-process that goes on in programmers' heads is informal reasoning proving things to yourself, "If I make this change here then that thing there must have a correct value, and then ..."
its more than that. there is a loose correspondence between plan and work. so changes in plans can often be projected onto changes in work. the same goes for expectations and dependencies.
the other thing about the plan is that it attempts to capture what completion might look like. where i come from, the things software developers do right before releasing code look very different than the things they do when they are starting a new project. the plan can allow you to try to do some orchestration, ensure minimum quality bars are met, etc.
having a fuzzy map of the future that you're constantly having to correct sucks. having no map at all leads to the kind of carpe diem development organizations that never seem to finish anything.
More than that we need to be flexible to adjust a plan. You’d be amazed at how many people who quote that aphorism aren’t actually flexible. At least with a building they are also motivated by not leaving a physical monument to their failure.
I'll raise (shrink) you an estimated-at-1-week landscaping project for my house that ended up taking just over 2. Wasn't particularly anxiety inducing, though, as we expected some overrun and both parties (re)negotiated in good faith.
But yeah the idea anyone has magically solved accurately predicting the future is completely bonkers.