Blaming IT managers is akin to blaming a bus driver when the bridge collapses.
IT managers are required to implement AV for compliance. They'd lose their jobs if they refused to do it and either failed an audit or got hit by common malware.
It's ironic to see so much praise this morning for the macOS model for kernel extensions. The fact of the matter is Microsoft allows 3rd party code to do exactly what happened today, and the industry leader in endpoint security completely failed to QA their updates.
Bus drivers allowed themselves to be assigned routes over bridges that were not engineered or built well enough.
It's ridiculous to put the blame on IT managers for this. The fact that Microsoft Azure itself has been affected by this shows that this is not a problem that any middle manager at a small firm should have be held responsible for.
It's obvious that you have never served in corporate IT management, so this is a pointless conversation. I'll just leave this here for anyone else who may be interested.
In many/most orgs, the windows sysadmins and IT management will not have access to the crowdstrike console. The CISO and security teams are completely separate, and operate independently. They will always push for the business need to deploy policy updates at any time, as the threat landscape will not wait for the next patch cycle. And actually, this has been the right call and has objectively prevented compromise many times over.
You were so close to getting this right, too. It's crowdstrike that didn't follow proper software deployment practices when they pushed their update to the channel. Where was their QA? They know how many of their customers have auto-updates enabled (every one of them).
I once worked in an online cloud environment, and the security team had crowdstrike installed everywhere, with the expressed promise that in production it would always be in monitor mode only. Well, suddenly our monitoring pods began to get SIGKILL'ed before they even could init and it took days of investigation.
Root cause? infosec turned on enforcement in crowdstrike, and didn't tell anyone. We didn't have access to their splunk logs, nothing got logged locally, and naturally nobody on my side of the org even knew crowdstrike was running because we had to fight to get SSH access to the k8s nodes.
But sure, blame the IT manager who is now desperately trying to bring up their entire AD infra. That's helpful.
"IT managers" is an awfully big umbrella term. It can refer to anyone making decisions like this. The CIO, the CSO, the CFO (if over IT as in a lot of organizations) and even the CEO. Responsibility may also extend to other mid and senior level managers within the chain of decision making who didn't speak up, to a lesser extent. In my last company I was an individual contributor, and if some one proposed placing an autoupdating, closed source kernel changing program in my critical path, you can bet I would have loudly spoken out against it.
You don't like the term "IT managers"? Let's call them "company managers", suits better?
You as a company allowed and external entity to push a completely uncontrolled update on all your production envs. What if crowdstrike had been hackered and instead of a BSOD you get a cryptolocker?
If you don't realize how crazy is this, then I having nothing to add.
I am sorry but I don't really understand what you are arguing for.
I can see what you are arguing against: it's the unchecked autoupdate policy for updating critical software. The problem here is that almost nobody does that anymore, specially because of the overhead this caused across the industry. To replace the overhead, there are contracts in place that if a supplier messes up they will be held financially accountable. It's called SLA.
Now, as for virus protection: AFAIK nobody ever gated AV updates. OS updates, yes, OS upgrades, even more so. But AV? Not to my knowledge.
What you seem to be arguing for is unrealistic. Consider a 0-day exploit, being frantically pushed by AV vendors to fight against, but the IT fails to gate the update in time. Time and time again the autoupdate saved our collective a*es.
The IT managers are definitely held accountable, so they will definitely insist on NOT gating updates.
The Crowdstrike should be held accountable for this fiasco and not the individuals, that manage each and every company's IT infrastructure. If the said company survives this, that is then a failure of our companies' leadership.
AV is still software, special kind of software but still software.
If you look at how "normal" software updates are handled around the world, you will see a recurring pattern:
- updates are first done in test envs and then in production.
- large production envs are updated in "waves".
- critical updates may go directly into production, but when it happens they need extra authorization and awareness
Please tell me why this pattern cannot be applied to AV updates?
> Now, as for virus protection: AFAIK nobody ever gated AV updates.
What happened today tell us that it is a bad practice, and btw I'm aware of some customer of mine that didn't incur any issue in production because they first updated the test env and spotted the problem.
> The Crowdstrike should be held accountable for this fiasco and not the individuals
We agree that crowdstrike should be held accountable, but they are not the only one.
What happened today tell us that there is a big hole to be plugged. And it can be fixed only but single companies. Crowdstrike, if survives, can improve it's QA process, but who will guarantee you that it won't happen again? What about other vendors? You should always assume that everything can fail and adopt processes that help prevent and mitigate these failures.
Again, think just for a minute, what would be the consequences of the today fiasco if instead of a bad file they would have push a cryptolocker or troian?
Solarwinds tells us that this is not a hypothetical scenario but a real risk.
I personally think that IT managers should be blamed for the disaster, but in the collective imagination this will e a microsoft/windows problem.