The idea of fixing a whole class of problems is common in safety critical software. When you find the cause of a bug its not just about fixing the bug but looking for this pattern of failure everywhere and fixing that and then understanding the aspects that led to this class of bugs to begin with and eliminating those. Its just good engineering to solve the class of problems not just the bug in front of you.
But I have also been part of a team that was replaced by another because we weren't heroic enough, because we had no bugs in our software and there was no drama for the business to get what it wanted and needed. Management rarely values this type of engineering.
I like to rate engineers across various categories, but really I found that there were two general classes of great engineers: fast and slow. For marketing tests, etc., the whole team would be fast engineers. For payments, the whole team would be slow. Everything else I would try to have tension -- a mix of the two so they learn and appreciate each other.
This is condensing a multidimensional vector into just a line, but effective enough to explain to non-engineers.
If you are a precision (slow) engineer, then you are more likely on the backend, more likely to write tests, more likely to avoid costly errors. This will be wasted on some kinds of tasks (trying out new things, for example, but as a class these tasks can generate a lot of management interest). The only kind of management interest in the precision stuff is failure, and it is usually doomsday failure.
That said, security is always present, and I am noodling ideas right now on how to eliminate whole classes of security and privacy issues while making it even easier faster for all engineer types.
> two general classes of great engineers: fast and slow ... Everything else I would try to have tension -- a mix of the two
To add to this, if you are one of these engineers a great career hack is to find an engineer you respect on the other end of the spectrum and partner with them.
As a "slow" engineer I make sure our key interfaces are abstractions are correct and my "fast" partner ensures that everything gets shipped and that I don't sweat the small stuff. This has lead to a lot more successful and impactful projects than I could manage on my own or with another "slow" engineer.
Interesting. I never thought about dividing engineers this way. I do know that the two best engineers I've ever met were slow. Not just slow in working, but slow in everything. Speaking, moving, etc. This frustrated some people, but these engineers saved everyone else much time and frustration because their work was rock solid, well thought-out, and complete.
And, once you accounted for time spent bug-fixing and validating things, they weren't actually slow.
Yeah, I gotta say, it seems like one of the key skills that differentiates a mid-level dev from a senior one is knowing whether the current task calls for being a slow engineer vs being a fast engineer.
Since I consider "Move fast and break things" to be one of the worst and most destructive philosophies to find adoption in software engineering, I guess it's obvious which side I favor.
It depends on what you're building, tbh. If you run a large b2c site and want to outpace your competition then you default to shipping. If the costs of failures are much higher, you are better off defaulting to a slow and steady pace.
True, from a certain business point of view. As a customer, though, I want nothing to do with such software and I don't want to work on software that I wouldn't personally use. That's why I say it's obvious which side I fall on.
Some of that is deeply ingrained, but it’s useful to be able to swap between different kinds of problem solving.
Ultimately, there isn’t a single best approach to software development just different tradeoffs. Being able to whip up a bug ridden happy path that only works for some set of data is useful when trying to build an understanding of some new system. At the other end even if most of what you do is short lived demos creating a few rock solid building blocks can save you a great deal of pain.
That's how I see brains work differently: fast and slow - one brain can quickly associate, find information; another brain is slow to load, in the end, you get 100% of what you need. I appreciate slow brains.
This is an interesting observation. I like to think myself as a slow programmer. Do you think discussing this in an interview before you actually got the job in order to evaluate if you are fit to the role joining is a good idea ?
The heroes who are up late, solving a page or mitigating an outage are often the ones remembered and rewarded.
Meanwhile, a dependable, resourceful, and independent IC who “picks up trash on the floor”, promotes good work habits, and is dead reliable - no praise, they are just “doing their job”
Managers and leaders like drama, most staff and senior engineers gravitate to drama and love talking about it. Their day to to day work, often subpar and often not team players
Yes, this is my problem. I'm seen as slow, because I take my time to deliver my stuff. And I'm rarely the hero of fixing some outage or some critical bug. Because my stuff works, quietly. The heros are always busy; fixing that bug and extinguishing that fire and delivers the new feature in a day (as soon as they clean up the latest incident). What people see is that I take three days to deliver the same feature. What they don't see is that I don't have to spend in total two weeks (spread over some time), fixing bugs and cleaning up corrupt data for the same feature.
Reminds me of my time as a sysadmin managing over 100 individual linux boxes by myself. Nothing ever broke (in dramatic fashion, I mean) and new stuff were delivered on a steady schedule. I didn't spend all my time fighting fires, so the perception was that I wasn't doing anything amazing. I have to constantly point out the uptime, that every single case that needed it, we always had data recovery, but that was just considered part of my job.
I once knew a soldier that had reconfigured his tent heater to be out of the way by placing it against the side of his tent. The tent caught fire. He received and award for putting out the fire.
I have seen this pattern repeated a few times throughout my career. People are rewarded for putting out fires they created (metaphorically), but people who are diligent and don't create problems to solve are overlooked or seen as less than capable.
On the field there's no time for false drama or fluff, you have resources (time, energy, devices) and you keep doing the best you can at every step. And if you don't think enough about how you plan your operations, you die.
I don't want my teammates to feel on the verge of death, but I really, really work better if I'm operating at high pace and density and if the team also does that, like swarm of people attacking all problems at all levels on the job.
Yeah the only way to get rewarded for working on resiliency like this is if you do it very loudly. If you spend an extra day or two refactoring a feature so that it is more robust you will be looked at poorly versus the engineer who slaps a few more conditionals onto it to keep it working.
Raise the issue with higher ups, maybe create some fancy charts about lost engineering time in the future, spin up specific tickets for refactoring, turn it into a two week project and you will get recognition. Management loves a chart about improving X by N% almost more than a shiny feature.
When the org that I was working for was doing TSP (https://segoldmine.ppi-int.com/node/67631), our coach told me a story about another team and an engineer I knew very well. He got high praise for all the late nights and weekends he worked to get a product out the door. But on analysis of what he was working on and the bugs he had to deal with, if he had taken the TSP approach that my team was using, most of those bugs would never have been created in the first place and the product would have been finished much sooner.
Drama gets noticed, just quietly ticking along, producing high-quality output really doesn't.
Exactly. I've worked at software companies where the executives wouldn't believe any work was being done unless they could visibly see activity, and hear the "buzz" of "people doing things" and feel the drama of production emergencies and heroics. It felt like they were listening for intense movie-like typing on keyboards, watching for theatrics in front of whiteboards, project leads calling for standups, and so on. Those were the teams truly DoingThings™ and those teams were rewarded for their performance art. The team silently plugging away at their desks in chat, while they calmly deployed another build that passed all test automation--I'm not sure if leadership even knew who they were.
EDIT: These folks almost certainly overlap with the ones pining for Return To Office instead of remote work: They miss the "hum" and "buzz" of SeriousBusiness™ happening all around them in the physical office.
Its exactly this. Their approach produced lots of weekends and late nights and broken releases and they could be seen to be fixing things and responding like lightening to every issue. My team on the other hand was 9 to 5, everything just worked when we released it with few bugs and we and the business worked like normal human beings. The problem is that its invisible and there are no heroics, its just good solid engineering. Which considering it was a back end system for a bank is the right way for things to be.
Management likes people who make heroic effort even if they are the cause of needing it they are visibly working hard even if they are making less progress.
Management also often doesn't track the amount of drama products created by "fast" teams cause later in production. Because the negative impact is often delayed, those problems are rarely attributed to the original authors of the code, who often move to new projects by that time. I've seen it so many times: a "hero" gets praised for writing a software component in a day and putting it into production quickly, despite a massive evidence showing that the maintenance of the previous N projects done by that person turned out to be literally a PITA and a constant source of drama later.
I'd love to see engineering bonuses / promotions work similar to how hiring bonuses work. You don't get a hiring bonus immediately when you recommend a new hire, but you get it once the new hire stays for N months. You shouldn't get a bonus/praise/promotion for just delivering software quickly. You should get it after it runs consistently and painlessly in production for N months.
Management can be assured of worker productivity by evidence of activity or evidence of output.
In some situations (probably a lot of software engineering situations) output is difficult to measure, and so the habit of tuning in to activity is adopted instead. Some may even forget the difference.
But in software engineering, it isn't difficult to measure at all. We're developing a deliverable. You can measure if the deliverable happens on time and with acceptable quality.
Not exactly. Your approach works if everyone knows ahead-of-time the exact amount of effort that something will take. In software, it’s rarely the case that a project complexity is fully understood from the beginning; at best, you can make a ballpark estimate.
If you have a rock solid approach and design a system with little drama, which is released on time and with high quality, then you look like it wasn’t a very ambitious project. Or easier than expected. Maybe your team padded the estimates a lot and didn’t need to work that hard.
If you are putting in late nights and weekends and constantly fighting to get features working, maybe management thinks that the project was way harder than expected. They’re so lucky to have someone as hardworking as you or this project never would’ve been done!
Obviously there is a flaw in the logic — it’s possible that those assumptions are correct, and person 1 really was under-ambitious and person 2 is an incredible and dedicated engineer working on crazy hard problems. But it’s also possible that the first engineer was just better, and the second had terrible system design skills and constant spaghetti code that made a simple project seem complex.
It can be really hard to tell the difference. Even if both end up delivering on time, the second looks more ambitious, like they’re taking on harder problems.
> Your approach works if everyone knows ahead-of-time the exact amount of effort that something will take.
No, it really doesn't require that. But we're getting into the topic of project planning, which is a larger subject than we can tackle in the comments here. Fortunately, this is a topic discussed in great detail elsewhere.
> But it’s also possible that the first engineer was just better, and the second had terrible system design skills and constant spaghetti code that made a simple project seem complex.
Right, I was intending to cover that with my "acceptable quality" conditional.
I have no single source. I only meant that this is a topic that is widely covered, and collectively with greater depth than is appropriate in this thread. Read widely on this topic, and don't focus just on one perspective. That way, needed nuance can be retained.
It comes down to "the squeaky wheel gets the grease", if you sit quietly in the corner and get things done, and there are no problems with the work you do. You will be invisible to everyone outside those you work with directly.
The person to makes a lot of noise, good or bad, is noticed
Everyone say they want well engineered solutions, but in practice it's not what makes an impact.
I think OP was trying to say something like, "Management replaced our team because we were quiet and didn't rally around problems like other teams". I do think it was poorly worded and a bit jaded.
Just wanted to say feel your pain. My team was recently disbanded to other teams because our product has no issues and we were considered too slow. Fast forward just released a shiny new front end that is buggy and slow. 4 hot fixes in a month. At my review they said see getting you on a new team was all it took to speed you up. Well ya I guess no unit testing, and no documentation. At least I got the team to start using the linter. Another benefit is I have more family time and been working on a side project to ensure I don’t forget how to do things correctly lol. Always think this place is going to be different but timelines are king and those who hit them will be rewarded. Also can’t get much faster than 4 releases in a month which is all the boss sees on paper…
Excellent article; thank you. I have written similarly in my notes, focused on the code maintainability (or component quality) part, but not in a shareable format yet. It is a sort of maturity model (I hope eventually to turn it into a grid with practices on the left side and levels across the top, so the RH column is near-ideal behaviors personally and organizationally). But the concept begins with a self-code review for each commit or PR, followed a code review from someone else (who ideally was not part of the design discussion) who asks questions like: is this code easily understood for a new maintainer? would I want to maintain this? How was it tested? Is any coordination in order with other stakeholders, or does documenation need to be updated? Is it clear why the code exists and the tests do what they do?
And then improving over time. This came up for me after repeatedly encountering code that no one could understand but the original author ("bus count of one", or if one person gets hit by a bus we would be in trouble), or that was needlessly complex for the enjoyment of complexity (really), or that had tests where no one knew why they tested for certain behavior (until fortunately someone else came back from vacation after I was about to commit a change to the test). Etc, all for comprehensibility and maintainability, and reliability.
Ps: overall, this is one of the most enjoyable HN discussions I have seen.
pps: I also one started encouraging our team and others to maintain a set of wiki pages, listing all projects for which we were responsible, then for each one at least one simple page of documenation listing things like who are the stakeholders, where is the source code, any odd build or deployment steps, the key (one sentence) inputs and outputs, where it runs, who does backups of what, etc. Before that it was haphazard. This is short of a real ops manual and could be done better of course, and would change as other org. practices change, but it was far better than nothing, and could be created in 10 minutes from a template. Great for bringing new teammates on board. I think an organization should have something like that for the whole org, listing teams and each team having such a page, as well as listing everything essential that a new person should know rather than relying on haphazard cultural transmission. We had rules like "no new debt" that were well-adopted and began to slip out of the culture as new people joined.
Yes, there are often a lot of organizational barriers that inhibit doing the right thing. Most organizations like to talk a big game about how innovative they are, but they really just want to keep doing the same things that they've always done.
I have been in the situation where over a period of time I inherited a lot of code that was written by my boss and absolutely critical to our business. Over time I began to see that they had made a number of poor design choices that made it very difficult to work with some of these internal frameworks. For example, there was a lot of global state passed around that kind of worked in production, but made it impossible to run tests in parallel (as well as making some of them flaky even when run serially). I introduced an internal dependency injection framework that leveraged some unusual host language features to remove the global state without painfully having to add a bunch of parameters to every function/class that would have to be passed around. This allowed the tests to run in parallel and completely removed the flaky failures. At the time I introduced the changes, they reduced the time to run all the tests from about 3 minutes to 20 seconds with no flaky failures. This was a major quality of life improvement not only for me, but for everyone else working on this part of the system.
Was this work rewarded? No. What I didn't consider when undertaking the refactoring was that I was implicitly making my boss look bad by fixing the systemic problems that they'd introduced. Even worse, I used techniques that were unfamiliar to them which was a further blow to their ego, though they wouldn't have admitted this at the time. Instead, they complained that they didn't understand some of my techniques rather than seriously try to learn what I'd done and appreciate the benefits that it brought. Ultimately, it became soul crushing to realize that they were more invested in doing things the way they'd always done them then to learn how to do things better (or at least offer constructive feedback beyond "I don't superficially understand this so it must be bad"). When you are in this situation, advancement becomes almost impossible because you have now become a threat to your boss, who will never make the mistake of allowing you to be promoted to their level or beyond.
I still think we should all strive to do the best work that we can because ultimately you should feel proud of the work that you've done. But this often comes with a major cost (which may ultimately be that you are forced to leave the organization).
"promotion is more about consistent level N+1 behavior, while one could get a high performance rating by solving many level N problems."
Not all companies are like this, and I find it common that new employees are not sufficiently taught what their company is like :
A) you have been performing well as N,promotion means "we feel / hope you're ready to work as N+1 in the future "
B) you've done great as N and have repeatedly performed N+1 tasks successfully. Promotion means "we recognize what you've already been doing "
First one is "proactive" and represents faith you'll do well at next band. In principle all you have to do is do your job well. There is risk though that you man not succeed after promotion if next band has radically different role or skillset.
Second one ensures you're prepared for your new band, but you cannot there just by doing your job well, and many people are unaware or don't have opportunity to do tasks of next band in their current role
The duties of level N and N+1 are often a little different. If they're different enough, then it can be difficult to demonstrate N+1 while also doing N work. You're option is to work two jobs at once or, as I more commonly see, do the bare minimum of N work and focusing on N+1 work. This can look like the teammate who's focusing on division-wide initiatives at the expense of implementing the things their team is actually assigned - leaving their teammates to pick up the slack.
This is certainly not inevitable, and I'm definitely not saying B is a bad way to do things - just that this is a failure case I've seen of that model. The other, of course, is "why spend a year doing N+1 work here at an N salary when it's easier to just get hired as an N+1 at a new company today?"
One successful way I've seen this down is to proactively give people ownership of sub-projects with collaboration, or leadership scope N+1. It is explicitly acknowledged (so no shadow work that your team is covering to support) and acts as a way to gauge maturity, while limiting risk.
Yep...we had a whole slew of people acting like N+1s on my team a couple years ago. Everyone trying to "lead" everyone else. Load of horseshit. Just get the work done. Meanwhile I feel like I'm penalized for not attempting to micromanage everyone else. My scores are great for N but I guess I'll never get N+1.
I've seen B cause issues for great devs. It seems like if you're an ok or good dev with management qualities you're much more likely to get promoted than if you're a fantastic dev with management qualities. You're too important to delegate management tasks to, so you never get the required experience to move up.
the idea that there's no path for ICs beyond management is insulting and degrading to the hard working creatives and engineers out there. Don't give up there's a place for you, you just need to look a LOT harder and in places you wouldn't expect. Play the numbers its a statistics game.
Oh this hasn't happened to me, I've been pretty proactive about interacting with both my direct manager and several levels above them, but I have seen it happen to other devs. Especially ones from east Asia who seem to culturally focus a lot more on doing a great job on N level tasks instead of focusing on N+1 level tasks.
Has anyone been stuck in a position where you seemingly cannot land a promotion because you've had to keep taking on more responsibilities, to the point where you're beyond qualified for the next level, but you're not yet able to excel past it to the next level beyond that?
- Must achieve the highest performance rating for current responsibilities to qualify for promotion.
- But current responsibilities are actually level N+1½.
- Sometimes struggle with the responsibilities from level N+2.
- Never get promoted due to inconsistent N+2 performance.
> They take the extra step to make sure the next person won't have to spend the same level of energy fixing the same issue, or eliminate the problem class altogether for their team.
Conversely, I find it frustrating when engineers do the absolute minimum, avoid refactors, and put the next person at a disadvantage ... all while their velocity is recognized by management as good.
I'm conflicted about this. Definitely am a fan of "leave it better than you found it", refactor as you go, etc.
But when it comes to bug fixes, the "absolute minimum" is often the best approach -- it can be explained to demonstrate that we know precisely the root cause of the problem, it can be reviewed for correctness readily, and we can feel good that we are not adding a lot of new behavior that can have downstream effects. The "defensive programming" approach.
A refactoring that makes the problem go away forever is an awesome tech debt project, but when you have a hot problem and someone puts up a 1k LOC PR to refactor the problem away it often introduces so much uncertainty that the improvements aren't worth the risk. You have to start asking questions like how long has this person been around, do they understand what they are doing here, have they really tested this fix or does it just have good "coverage", do they really understand the root cause of the problem or is this just a "refactor and pray it goes away" fix...
A small-as-possible fix relies much less on the credibility of the code author and much more on the fix being self-evidently correct.
"Absolute minimum" in practice usually means hard-coding a special case/workaround directly into a live prod instance. That is the quickest and 99% of the time won't cause any operational issues in the short-term. It happens all the time and contributes enormously to technical debt.
It's a judgement call. If you're maintaining software, it's true that every time you touch the code -- even to fix a bug -- you're taking a risk that you're introducing a new bug. Refactoring software is an even greater risk.
So, in general, the instinct is (and should be) to touch the code as little as possible. However, that's not always the right answer. The right answer is to do a risk/benefit calculation and choose the approach that is most likely to lead to an overall improvement.
Sometimes, that means change as little as possible to fix a bug. Sometimes, that means refactoring a major piece. It all depends.
> I find it frustrating when engineers do the absolute minimum, avoid refactors, and put the next person at a disadvantage
If such engineers don't care about other or future engineers who will have to touch their work, they should at least care about themselves. Perhaps they've never heard the maxim: work to make the life of the next engineer to touch the code easier, because the "next engineer" will probably be you.
There is some unfortunate tension between the two maxims of "cleanup as you go" and "keep PRs focused." An idealist will tell you, "why not submit the cleanup in a separate PR?" But in reality, there is always overhead to each new PR, and the code most amenable to "cleanup on the go" seems to be the code that sits just below the start of that overhead.
It's especially frustrating when "cleanup the code" means "change one line in a few dozen files," because this results in a PR with a high number of changed files. Even if each change is only one line, the high number in the "top line metric" (no pun intended) of "files changed" makes the PR look unfocused, and a lazy reviewer might waste everyone's time criticizing that. So the path of least resistance is to avoid even mentioning the cleanup. Repeat for every PR for the lifetime of the codebase, and your code quality will deteriorate from death by a thousand cuts.
> I find it frustrating when ... recognized by management as good.
... but you're not the one writing the checks, remember? Management is. They don't care about quality. They can't even define it, much less recognize it. They care about delivery dates. Delivery dates are easy to measure. So that's what matters.
> put the next person at a disadvantage
The first guy was at the same disadvantage - which was that he was under an unreasonable deadline that didn't leave enough time for quality improvements. So you expect him to sacrifice his reputation so that you don't have to sacrifice yours?
Doing these kinds of unnoticed forward-looking remedial things provide the doer with good conscience and inner satisfaction, but do not put the doer on a fast track to career success and riches. It's the nature of the economic beast.
I've grown wary of folks in senior spots that are completely oblivious to the problems they cause in the name of going beyond.
All too often, today's problems stem from yesterday's solutions. This does not mean that yesterday someone made a mistake. It just means progress moves answers. If you did make a choice that feels evergreen, it is just as likely that you are oblivious to work other people are doing.
To that end, maximal choices on how to fix things so that they don't come back are tough. And just as often, large and reaching refactor jobs cause work for the sake of the work. Which is not at all a positive. If you think you can predict what will work, good luck. But don't require good luck from people below you for survival. Celebrate good fortune without belittling the others.
Of course you don't want the next person to slip on the same stone, as it were. But realize that forcing people to walk next to you is a large portion of the reason they may be slipping. And leaving holes in the ground as an alternative is clearly off. As is not waiting for the work of a larger fix.
>I've grown wary of folks in senior spots that are completely oblivious to the problems they cause in the name of going beyond.
Like over-abstraction, adding layers of indirection, changing things to use generic code that has no clear relation to the problem domain (and complicates domain-related changes to business requirements), then making other parts of the system (or other systems) re-use those same generic abstractions, so that they're all coupled and/or have increased dependencies?
A good abstraction can be useful, and can even simplify things, but sometimes adding abstractions can be injecting second-system effect into an existing system. Whenever I think of doing something that could turn out like that, I try to get at least a couple of my teammates opinions on it before I even start.
That's the difference between programmers and engineers.
Programmers fix the code, engineers fix the underlying issue. Engineering is being able to spot patterns and know enough about a subject to be able to research it properly and efficiently. "Is this a state machine?", "can I represent this as a tree?", "is this a regular language or do I need a more sophisticated parser?".
I recall someone from a bootcamp writing a cascade of nested if-else statement, 6 level deep in some places. Then someone with a real engineering background told him that he was basically building a finite state machine, to which the other dev responded that "he didn't need anything fancy, just for the function to work". Eye opening.
Something I am thinking of similar to this. Firefighter vs Gardener.
The firefighter focuses on the high stakes issue at hand and make sure it is solved in time and without much damage. The Gartner ensues the fire doesn't happen in the first place by making sure he is cleaning up the dried leaves in the first place.
But in a company environment, the firefighters are more valued are praised because that's something you can see and quantify. where as the regular cleanup/refactoring and fine-tuning the api is invisible and boring. Just a thought.
Another way to put this is that more senior engineers recognize that the issues impacting an org are not a discrete set of problems, but are sometimes the same problem being reported by different people or teams.
I've worked with people that do this and nothing will put you on my never-work-with-again list faster. Hamstringing entire teams because you're doing something only for yourself is toxic.
I wouldn't say this is the 'best' path, but certainly one that is rewarded a bit too often. Ultimately this leads to some really poor org structures and power imbalances, at which point I hope all the good people have left.
Sad but true, I've grown to realize that it's important to highlight the applications that just work or were easy to extend because of a little foresight, otherwise it's too easy to forget. As I've done this more, I realize manager appreciate the reminder and reward the effort for these drama-free applications.
> Once they reached the other side, the general asked "aren't you afraid that you'll slip on the same stone on the way back?" "That's okay. I'll know which stone to watch out for," said the young soldier. To which the general replied "what about the rest of the infantry?"
The infantry is bound to reinvent whatever you come up with badly, and slip on the same stone.
Promotions are a function of the person(s) awarding the promotion and the person receiving them. Article is about the ideal behavior of the person receiving them. Unfortunately, often the people awarding promotions are looking for some other quality or are completely incapable of distinguishing level N+1 behaviours from the rest.
I was fired from a small ERP company that will remain nameless for fixing the broken math that caused an entire class of bugs instead of just the one bug as reported. It was a snapshot of an older version of the code base, so by giving known issues with known "correct" solutions to newer hires they could get them familiar with the code base quicker.
It was an interesting idea, but egos in the room weren't ready for the possibility that the "official" answers weren't actually correct.
I was very new there and my manager didn't understand the algebra when I showed them, and they were very uncomfortable when I showed how the general case manifested in production. I was sacked a day or two later.
It was obviously for the best, but still pretty funny that there are some places where this sort of "maturity" is frowned upon.
Really mature engineers will spend 3 months running a psyop campaign with upper management about the value of removing the stone, 3 months devising a new stoneless architecture, 15 months letting a team of underlings implement it, and a lifetime of failing upwards because they “led large scale projects.”
I like how Sarah Mei says, "Inline everything". You don't get to go on a refactoring crusade. You do it on a daily basis when you are working on the actual production issues. Easy to say but based on my experience, it is the only way to move things in the right direction and keep them that way.
Couldn't agree more, and one of the more frustrating aspects of the job. I especially for situations where I find out that someone fixed a problem for x, even though they knew at the time the problem existed for y and z, but "no one reported it." So much time and productivity is wasted.
One danger in the "what about the rest of the infantry?" mindset is that it's too easy to start generalising the problem - and, consequently, design solutions - to a bigger and bigger extent.
What about other stones inside the encampment that the soldiers can slip on? What about the stones outside of the encampment? Wait, isn't the wall around the encampment MADE OF stones?
And pretty soon you have soldiers cleaning up stones all over the landscape, replacing the wall with solid concrete and designating a stone-cleaning platoon that will go ahead and clean the stones preemptively.
Even bigger danger is convincing yourself that you are the hero that eliminated a "class" of problems without a concrete evidence that the class exists or worth eliminating. After all, everyone is biased to build a story in their mind in which they are the hero.
I've seen too many teams happily investing in stone-cleaning activities while forgetting there's a war out there.
You have a project that has both typescript and javascript. Someone makes a change in a JS file. They try to access a property of an object but there is a typo.
You could fix the typo and be done.
Or you could fix the typo, and convert the file to TS and make sure its typed.
And then you get another typo in another JS file. So you convert it too. Rinse and repeat. In the end, all your application is now written in typescript, hooray.
Only the number of defects stays the same. Puzzled, you start investigating, and eventually you realize that all of those bugs were shipped by the same person. Turns out your whole TS crusade was a distraction. Even worse: you were optimizing for poor performers at the cost of your best performers.
But when all you have is a hammer, everything looks like a nail.
On a product's multi-year death march, I look at the next bug assigned to me for the project-within-the-product I was on.
Over an hour of diligent debugging revealed the problem - the C++ code meant to do
X = Y;
but someone had typed and committed:
X == Y;
The destructive value assignment became an innocuous comparison, whose result is immediately ignored.
I decided to search the rest of the tens-of-KLOCs project for similar assignment-turned-comparison statements.
That's mindless and tedious work perfectly designed for a computer.
Several minutes later, after weeding through the false positives, I created bug entries for any offenders.
Did I stop there? No.
I connected to the source server for the entire product and kicked off the same search.
When the search finished, I separated the buggy wheat from the chaff and created bug entries accordingly.
This happened on the weekend.
When the product triage team met the following weekday morning, they saw all the bugs entered across the several projects in the product due to the same root cause - double equals instead of a single equals sign.
Management decided to take the next step. They bought a site license for a static source code analyzer. We integrated the analyzer into our project build process and ran the analyzer on each build and triaging accordingly.
Highest compliment I got for creating all this "extra work": "F*ck you!" said with a smile.
Did I stop there? No.
I kept my eyes and ears on the lookout for more possible typos.
I would read commit emails and resolutions to bugs to see if the cause/fix would fit the model of "easily-found-via-grep".
I expanded my batch file to cover new cases and created new bugs when the batch file found them.
Did I stop there? No.
Eventually I hit the wall of diminishing returns for source code analysis via regular-expression searching.
I looked for a tool that could go to the next level.
At that time, the one scripting language available in our project's build tools was perl.
Off to the bookstore and I bought O'Reilly's Pink Camel Perl book.
A few hours later, I had a rudimentary source code analyzer that looked at lines of source code for typos.
I added more cases as appropriate.
But C/C++ source code isn't rigidly formatted and programmers wouldn't play nice and limit their code to one code statement per source code line. Preprocessor macros also confused the line-at-a-time script analysis.
So I bit the bullet and expanded the perl script to "parse" C/C++ code.
I added checks to make sure memory allocations were checked for failure (no exception-handling for memory failure in our codebase).
Did I stop there? No.
I publicized the script within the company, answered questions, listened to suggestions, and offered help to other product groups in the company.
A couple of other product groups integrated the script into their build process. Obviously it'd be better if each programmer would run the script before they committed modified code.
Did I stop there? No.
I had worked on the company's new signature product before the current project's death march had started. I still had access to that signature product's source code. So I would run my perl script against the product's source code. That product is huge. A source code scan over the network (my work machine didn't have enough disk space to enlist in every project in the product) would take all night. I had access to the bug database but I didn't know which source code directory mapped to which project in that product.
So I did the next, most annoying thing - I sent a cleansed list of defects to all the developers in the product.
Did I stop there? No.
The company had bought a static source code analyzer product and proceeded to integrate it into most of the company's products' development process. They created a stripped down version of the source code analyzer suitable for developer use before committing code. My perl script was obsolete now.
But that didn't mean others couldn't benefit from it even though the company didn't need it anymore.
I noticed a "call for papers" notice for an open source conference.
I emailed the company's legal department and requested to open source my perl script. Their reply: "Permission granted."
I wrote my proposal, sent it into the conference organizers, who accepted the proposal, wrote a talk, and presented at the open source conference.
Did I stop there? No.
Talk is nice. Code is better. I published/uploaded the script on CPAN (the Comprehensive Perl Archive Network - https://www.cpan.org ).
Did I stop there? No.
Around the time Coverity started scanning open source projects, I had downloaded the source code to several prominent open source projects and scanned their code and sent emails with possible bugs where appropriate.
GCC has a warning for statements with no effect like x == foo();
$ echo "void f(int x) { x == g(); }" | gcc -Wall -W -x c -
<stdin>: In function ‘f’:
<stdin>:1:22: warning: implicit declaration of function ‘g’ [-Wimplicit-function-declaration]
<stdin>:1:19: warning: value computed is not used [-Wunused-value]
This is great. I have a person at my company who doesn't even "learn the stone" for his own stuff. This person isn't even a baseline engineer - he's a negative for the whole team. And the management who doesn't recognize this is dragging us down as well.
The article describes how it should be. But in experience, it's sadly the one who churns out most features that wins. It doesn't' matter if the leaves the codebase in ruins. That someone's else's problem.
Posts like this hit HN pretty regularly and are always popular, but they're essentially the equivalent of the perennial LinkedIn "I was already late for a job interview, but I stopped to help a stranded motorist change a flat tire, and then when I showed up for the interview, it turned out I was interviewing with the stranded motorist". For whatever reason, a lot of people seem to believe that if you post something often enough on the internet, you'll make it come true.
> the young soldier slipped on a stone. Feeling flustered in front of the general, the young soldier quickly put the stone back in place to catch up...
> the general asked "aren't you afraid that you'll slip on the same stone on the way back?"
That makes no sense. The general is an idiot if he thinks that the stone is the problem here. There are millions of stones in a river and they shift over time. Also, it's more likely a problem with the way he was walking; not feeling around with his foot to check that the stone is stable before shifting his weight. The message I get from this story is that bosses are often looking to invent ways to criticize their subordinates in order to bring down their self-esteem. Employees with low self-esteem will be more obedient, accept lower salaries, etc...
I guess. But as a millennial who's been in the place of that young soldier a bit too many times, it's difficult to see it from the general's point of view. The message of subordination stands out more to me.
I see cultural differences here. An American will get grumpy and just fix the damned thing. Someone from another culture might sigh that someone else didn't do their job and shrug shoulders and amble away.
But I have also been part of a team that was replaced by another because we weren't heroic enough, because we had no bugs in our software and there was no drama for the business to get what it wanted and needed. Management rarely values this type of engineering.