> do estimation the same exact way, but rather than arbitrary deadlines,
The core problem with that is that effort - consciously or not - gets immediately turned into (start day + estimate) => deadline which only works when you consider spherical cows, not accounting for the hard true practical fact that reality is messy.
And it is quite messy, by the very nature of the work which is to do something that has not been done before (otherwise by the very nature of software you'd just reuse it) and thus carries unknown unknowns. Everyone has experienced this "one-line fix, should be done tomorrow" that turns into a bind-mending multi-week hunt down the rabbit hole; replace "fix" with "feature" at your discretion; replace rabbit hole with "that very urgent task preempting anything else".
The second-order problem is that deadlines are used to schedule higher level dependencies between teams, all the way up to product, and possibly customers, and then it becomes the coupling interface and everything falls apart when there's a delay that inevitably trickles up with rippling consequences, because the system is not designed to handle such a failure mode, having this deadline dependency as its core interface between all the parts.
If you're not convinced of that, "follow the money": the consistent metric across the software industry to evaluate an employee's performance boils down to "deliver on time". Three months down the road, the rational explanation to delays of mandatory rabbit holes and very legit preempting tasks is forgotten, even when acknowledged by management. You're told to factor all that in, but estimating unknown unknowns is by definition impossible (best case you go statistical, with deadly outliers around the corner, which amounts to say it's a bet). Experience reduces these unknowns but they're still around, everywhere. Overall, you tried telling the hard truth with absolute candor were left helpless when it backfired.
Ultimately, whatever the process, the stark reality is that to get shit done you have to take part in the make-believe dance and lie - white lies, because soon enough one realises that the other kind of lies also backfires very quickly - in a way that still communicates some form of deeper truth: "progress is being made, it'll be done (when it's done)". Inflate some numbers, tune wording when reporting, steal time by bleeding some for task X (reported or skunkworks) into task Y. All the reporting meetings and documents entirely become smoke and mirrors, but are somehow important as they're oil to the whole machinery. Two parallel universes emerge and graciously evolve in parallel.
This sleigh of hand is the missing buffer catering for the lack of acknowledgment that producing software is entirely different in nature than producing hardware in a factory. It is in essence more of a creative act, albeit a strangely misleading one because of its technical component. Lots would be amazed as to how much closer it is to advanced drawing or musical composition than it is to more "material world" engineering (although the rigorous process of the latter certainly helps for technical aspects), and taking classes of the former would probably do a lot of good.
The core problem with that is that effort - consciously or not - gets immediately turned into (start day + estimate) => deadline which only works when you consider spherical cows, not accounting for the hard true practical fact that reality is messy.
And it is quite messy, by the very nature of the work which is to do something that has not been done before (otherwise by the very nature of software you'd just reuse it) and thus carries unknown unknowns. Everyone has experienced this "one-line fix, should be done tomorrow" that turns into a bind-mending multi-week hunt down the rabbit hole; replace "fix" with "feature" at your discretion; replace rabbit hole with "that very urgent task preempting anything else".
The second-order problem is that deadlines are used to schedule higher level dependencies between teams, all the way up to product, and possibly customers, and then it becomes the coupling interface and everything falls apart when there's a delay that inevitably trickles up with rippling consequences, because the system is not designed to handle such a failure mode, having this deadline dependency as its core interface between all the parts.
If you're not convinced of that, "follow the money": the consistent metric across the software industry to evaluate an employee's performance boils down to "deliver on time". Three months down the road, the rational explanation to delays of mandatory rabbit holes and very legit preempting tasks is forgotten, even when acknowledged by management. You're told to factor all that in, but estimating unknown unknowns is by definition impossible (best case you go statistical, with deadly outliers around the corner, which amounts to say it's a bet). Experience reduces these unknowns but they're still around, everywhere. Overall, you tried telling the hard truth with absolute candor were left helpless when it backfired.
Ultimately, whatever the process, the stark reality is that to get shit done you have to take part in the make-believe dance and lie - white lies, because soon enough one realises that the other kind of lies also backfires very quickly - in a way that still communicates some form of deeper truth: "progress is being made, it'll be done (when it's done)". Inflate some numbers, tune wording when reporting, steal time by bleeding some for task X (reported or skunkworks) into task Y. All the reporting meetings and documents entirely become smoke and mirrors, but are somehow important as they're oil to the whole machinery. Two parallel universes emerge and graciously evolve in parallel.
This sleigh of hand is the missing buffer catering for the lack of acknowledgment that producing software is entirely different in nature than producing hardware in a factory. It is in essence more of a creative act, albeit a strangely misleading one because of its technical component. Lots would be amazed as to how much closer it is to advanced drawing or musical composition than it is to more "material world" engineering (although the rigorous process of the latter certainly helps for technical aspects), and taking classes of the former would probably do a lot of good.