Hospitals are incredibly important services that has under-invested into cybersecurity. A lot of medical devices are on very old systems (Windows XP) with no upgrade paths. When it comes to ransomware, you want to attack something that's important, and something with weak defenses.
Hospitals are why I don't "blame" underinvestment into cybersecurity. Their #1 goal is saving people's lives, not messing with IT issues. You want hospitals to be paying for important equipment, important people, important skills. The whole IT part is just supporting the administrative tasks.
But yes, it means that paying the ransom is the better move a lot of the time than to actually try to restore IT services.
--------
At some point, it becomes more efficient to go after the hackers, rather than trying to defend every single Hospital.
Ex: When REvil accidentally hacked an oil-pipeline (instead of a more passive target), the blowback was so severe that REvil disbanded and ran away. It caused an international incident, to the point where Russia has caught the attackers and is offering them up to the USA as a peace offering.
What is rather unfortunate, is that we put more importance to our oil-infrastructure than our hospital infrastructure. But these ransomware attacks on health care has been going on for years. Its not new.
> Their #1 goal is saving people's lives, not messing with IT issues
Technically, profit tends to be the #1 goal, at least in the US. Consequentially, this also drives a lack of investment in cybersecurity. Also, US hospitals have some of the most opaque pricing and billing processes of any industry that I can think of, which makes it much easier for them to recoup losses from patients that can't pay by shifting those costs onto the insurance provider and other patients who can pay. This is one of the reasons why basic things like bandages cost so much in an ER. Despite efforts to bring transparency to medical billing, hospitals are still resisting the push to publish pricing and explain their business models in more detail. We've become so culturally desensitized to the state of US healthcare that we're now just defending it as "we really can't expect hospitals to do any better than they are right now", and that kind of apathy really scares me.
As the healthcare sector continues to be consumed by private equity, I don't expect to see the situation to improve. Again, it's all about profit, saving lives is secondary.
> Technically, profit tends to be the #1 goal, at least in the US. Consequentially, this also drives a lack of investment in cybersecurity.
UK's hospitals fare no better in terms of cybersecurity. This is about the culture of nursing / doctors / hospital administrators, which is largely shared between USA and UK.
This isn't a systemic issue that is solved by nationalizing health care like UK did.
USA health care system, culturally, is about saving lives. Whether our system matches it is another story. But the underlying people largely do the right thing.
------
I think the systemic issues regarding health care / infrastructure / investments are wholly independent of this cybersecurity issue.
> USA health care system, culturally, is about saving lives.
With all respect, but for someone who had lived in the US after moving from EU, I'd say it's first and foremost about making money. It saves lives where saving is needed, but I'd argue vast majority of cases are outpatient and the culture is strikingly blunt about milking the patient.
Hospitals in the US are not especially profitable. Including federal relief, median hospital profit margin is 2%.
The whole market is wildly distorted- starting with doctor education up through private insurance and government programs like Medicare and Medicaid- that simple answers like this totally miss the mark.
Agreed. Any simplistic statement like "the problem with healthcare in the US is [blank]" is evidence of someone that doesn't know very much about the many complex and interlinked issues. Likewise, someone thinking the system can be fixed by "just doing X" is also being reductionist.
The pandemic showed a number of areas in healthcare where people were generally ignorant. For example, thinking that hospitals have tons of reserve capacity to handle extraordinary events. Even well before the current situation, hospitals (community) tended to run at about 80% occupancy. Far from being a profit-consideration, even the department of Health and Human Services mandated that hospitals had to run at least 55% occupancy, or they lost benefits.
The pandemic is a bit unusual in affecting everyone at once. For a local or regional problem, staffing wouldn't be as much of an issue because workers can travel. (For example, traveling nurses.)
> that simple answers like this totally miss the mark.
I am not providing my "answer" to the US problem, I am merely noticing how strikingly different approach the healthcare has here, so I reject your insinuation.
To be honest, I don't need to care who's making how much money to make a point – all I know that from my perspective, at the end of the day it is about milking the patient and it differs wildly from the general EU experience.
If profit were the primary motive, wouldn’t you expect non-profit institutions (both healthcare and otherwise) to be in much better shape from a cybersecurity standpoint? E.g., is there evidence that a large non-profit healthcare system like the VA is substantially better at cybersecurity?
While profit no doubt impacts the decisions, it doesn't appear to be the primary driver of cybersecurity lapses.
I wouldn't. Both goals of maximizing profit and achieving a goal on a minimal possible budget end up cutting costs in places that aren't immediate blockers, where security lies. In my experience, security is a focus at places, either non-profit or otherwise, in one of the following situations:
* The organization has one or more squeaky wheel employees that force everybody else to consider security where they wouldn't otherwise.
* The organization or another in the same industry has already had a very painful security breach.
* Security itself is part of the selling point.
Non profits are slightly different, but they still experience many of the same problems because the goal is still getting the most done on the budget you've got.
Yeah, I can see that. I think you’re right. But that feels more to the "cultural" point (i.e., different perspectives having different priorities) than the specific claim specifically that "profits" are the driver.
> If profit were the primary motive, wouldn’t you expect non-profit institutions (both healthcare and otherwise) to be in much better shape from a cybersecurity standpoint?
I will respond to that partially: where profit is not a primary motive, i.e. in countries where healthcare is public, it tends to be centralized on federal or regional level, and, as such, much of the IT and cybersecurity is a lower, shared cost incurred by the government.
So if I understand your point correctly, it’s not necessarily that removing the profit incentives directly improves the outcome but rather the improvement is attributable to a better economy of scale?
There is a lack of cybersecurity investments in almost every industry. The issue is that the executives making the decisions 1) Usually aren't knowledgable about CyberSec and 2) don't justify the investment because it's not something they can physically point at and take credit for. .
The "economist" proposed a solution: tire cyber-security incidents to the stock market. The approach proposed was something akin to "have someone count and display the incidents of each company and blast radius". I'm not sure if this would actually work.
The other capitalist option is to make cybersecurity insurance mandatory, and impose high fees both to reimburse victims and to some government watchdog/agency (yes, government watchdogs and capitalism can co-exist). Then, it will be in the insurer's best interest to have clients with adequate cybersecurity implementations, and the market can sort it out.
At the same time, we should make sure that any insurance company that chooses to pay the criminals instead loses their license to operate.
The US healthcare system cannot be primarily about saving jobs or the AMA would not have ever lobbied to restrict residencies to prevent a glut of doctors.
Since the AMA is an organization of medical professionals, one must conclude that it reflects their position: protectionism for their field.
In that case it is a culture of low salaries and tech being a support function. Governments aren't paying market salaries for tech and are not willing to have highly technical people in many leadership roles.
Many governments aren't willing to have highly technical people in any leadership roles. I've worked with government IT departments before where 100% of management (not an exaggeration) was non-technical, as in had never been a developer, sys admin, or any type of engineer. From the front line managers the whole way up to the "CIO."
I certainly agree with much of your attitude toward American hospitals. But I don't think dragontamer's point had anything to do with greedy American corporate hospitals. So mentally substitute "community-owned co-ops of small rural hospitals out in farm country" if you need to.
The point is, just like it says in the Preamble to the U.S. Constitution - "...insure domestic Tranquility, provide for the common defence..." - that protecting everyone from large-scale, organized, high-skill malicious activity is a bedrock function of any national government. NONE of the hospitals, water treatment plants, small corporations, city governments, ordinary citizens, etc. should need to worry about high-cost, high-skill self-protection against ransomware groups - any more than they should have to hire and equip private security forces to protect themselves against mafia enforcers, Russian paratroopers, or missiles launched from North Korea.
it's true that domestic orgs should be guaranteed peace and stability at home on American soil. But the us government only guarantees you and your property/business interests to a limited degree abroad and domestically as well.
Banks and stores in US routinely employ private security. There is no reason why US public should foot the entire security bill of Tiffany's or CVS.
US gov should have defensive and offensive cyber capabilities deployed strategically to assist and deter, but uncle sam can't babysit each and every it vendor or client.
US gov also needs to hammer shit IT practices and make it too expensive for bad guys to do harm and too expensive for "good guys" to be morons.
Para. 2 - Yes...but notice how few people or organizations need private non-cyber security. And even when they do, it's usually a tiny number of modest-training, modest-pay security guards. If half a dozen bandits with automatic weapons started robbing a Tiffany's - is Tiffany's expected to have (or foot the bill for) a security team armed & ready for that? Or would public (city / county / state) law officers be expected to arrive ASAP, and take over?
Para. 4 - Lordy, yes. Though that needs to be competently done. Starting with setting up a computer version of Underwriters Laboratories - that could drive the sellers of Internet of T*rds crap out of business (at least in the U.S.), revoke Experian's right to operate a database full of sensitive financial information, etc.
What about those that are non-profit? You can refuse to do business with for profit hospitals. Getting rid of the for profit does more harm than good, especially for underserved communities.
There are many "non-profit" billion-dollar hospital chains in the US.
A few examples;
Ascension Health - $5.7bn net income on $27bn revenue in fiscal 2021 [1]
Cleveland Clinic - $1.3bn net income on $6bn revenue in H1 2021 [2]
Mayo Clinic - $728mn net income on $14bn revenue in 2020 [3]
"Non-profit" doesn't mean they don't like profits just like corporations. It's a designation meaning no shareholders, as in money made by the organization stays within the organization.
I like the term "not-for-profit" rather than "nonprofit" as I think it more accurately captures that the while the primary goal is not profit (unlike a traditional corporation), it does not mean that they don't make money. Pedantic, perhaps.
You don't get all that much choice in hospitals. In an emergency, you go to the nearest. Otherwise, you go to the one your insurance/doctor are affiliated with.
Sure, some places have two (or more) hospitals in the region, but often there's only one and the choice is just go or don't go. It is one of those areas, like utilities, where it really is _not_ a free market.
> Their #1 goal is saving people's lives, not messing with IT issues.
Two years ago (October 2020) when COVID first started and hospitals became cyber-attack targets, all the government agencies put out guidelines for them to follow.
As part of this, the hospital I was a sysadmin at sent me to cyber-security training. I was excited at first, it was part of a big healthcare coalition, running out of the top university in the state...
And we get to the classes. Most of the people there were the CISO, VP of cyber security, etc. Our entire first day was wasted just getting people signed into the labs. Web-based VMware client, a mix of Windows and Linux virtual machines, depending on the excerise.
I realize these people aren't 'hackers'. I realize all of these people don't have VMware or Linux experience. But I felt like I was walking my grandmother through creating an Amazon account. And all of these people are making 6 figures and the head of something security related at the largest hospitals in the state. Insanity.
Hopefully these people have very capable staff under them. The second day, we only wasted half a day with getting people to be able to log into a VM and follow step-by-step commands. It was basic stuff, what you'd find in a 'Hacking for Dummies' book. You'd run Kali Linux and do a vulnerability 'attack', analyze some files, patch some software to it was no longer vulnerable...
When we got to part that was a short C program illustrating a buffer overflow, I realized the wrong people were attending the class. I think most others did as well as you never heard another peep from the 30 people on the Zoom meeting until the very last day, asking how they could get their continued earning credits or units or whatever they are called.
I don't find it remotely hard to believe that the skills needed to be a 'hacker' and the skills needed to run a security organization for a billion-dollar healthcare organization have zero overlap. You don't want a CISO or VP of CS to be playing around in VMware. That's got nothing to do with their job.
I attended CISO course run by Uni of Washington https://www.washington.edu/research/research-centers/center-... It was total garbage by clueless paper pushers. Main focus was on securing business agreements delegating responsibility, aka deflecting the blame. There was zero time spend on actually securing anything, it was all theatre or/and way of funneling funds to select companies from private sector (like Kronos).
I wouldn't say it's the main focus, but this is an extremely valuable risk treatment. I would say it's underrated.
Too many orgs go into do or die mode and try to do everything and do it poorly. Accepting risk is a liability no person in authority ever wants to sign, even if it's accepting the risk of an earth destroying comet. They will ask you to do what you can instead, when you don't have enough resources to even begin implementing the needed controls.
Often in info sec governance literally the only way for us to get leadership to accept a realistic scope is to do as much risk transference as possible.
I'm not even sure that hospitals are under-investing in security so much as that the current security paradigms are dysfunctional for hospitals (and a few other key industries).
The current security paradigm includes a lot of rapid adjustment. Upgrade this package immediately, ship a new binary using an upgraded library, firewall this off right now, etc.
I think that might be fundamentally incompatible with an environment where downtime can be counted in human lives. The risk calculations are a lot harder when death is a potential outcome of downtime caused by upgrades.
I don't have a magic bullet solution to that, but I do think that gets lost in a lot of the armchair security discussions around hospitals. They operate under very different expectations than the rest of us.
Pay for university professors to be experts in maintaining this civil infrastructure, for the maintenance of the commons. Reward bounties to students and volunteers who triage and resolve issues.
Have an expressly stated set of goals about the above as well as a core set of stable priority maintained software that gets extra security vetting. Formal analysis, whole classes of students in different locations scrutinizing and learning every line of code, function, and the overall design. Formal validation where possible.
> Ex: When REvil accidentally hacked an oil-pipeline (instead of a more passive target), the blowback was so severe that REvil disbanded and ran away. It caused an international incident, to the point where Russia has caught the attackers and is offering them up to the USA as a peace offering.
On a similar vein: Hackers Apologize to Arab Royal Families for Leaking Their Data
It's not the main problem, but one problem with Hospital IT is doctors making IT decisions. I hear from others there's a similar problem with IT around lawyers.
I can confirm the lawfirm side of things. Back when I cofounded an msp they were some of our best clients. Why? Because they all collude about pay stuff (illegal but what are you gonna do, sue all the best lawyers in town?) to the point where around 2008 they just started firing entire it departments and sysadmins thinking they could pay less for outsiders who could then be scapegoats if shit went wrong. The funny thing was that they not only spent more money on the msps and consultants, but got less work and machinery for it. Getting anything approved was like pulling teeth, especially in places where it all had to go to the partners first.
I appreciate my time working with some great lawyers because I learned so much and still have many useful contacts (do you know the best IP lawyer in your state?) but it really created a quiet seething distrust of lawyers and the legal system in general.
Ive never seen the worst people in society hailed as the paragons of the community as much as lawyers.
The biggest hospital gig I had was for the neurosurgeons and they got stuff done faster than any other hospital department because they had their own building, the pull, and the money to do so and due to stories I heard I just knew they were an outlier.
Hospitals have terrible budgets. There's never money to buy anything. Doctors make big salaries, but there's so much administrator bloat, it's similar to colleges.
There's a bit of responsibility from us IT / cybersecurity folks.
Our system is setup that we defend the networks we've been assigned to. The greater cultural problems are someone else's problem. We don't actually look outside of our own networks.
Hospitals getting hacked? Well, that's sad, but not our problem. Not until they pay us at least.
------
Granted, I'm not sure what we _should_ be doing about this issue. But at least acknowledging our current culture would be a step forward. Good IT security comes from the top, from a culture of security.
Some of that is due to being told to not touch them. There's strong cultural memory of safety, security, or just sound planning being thrown out by non-IT people, till even new hires quickly start getting instinct to bunker down.
People who are in highly educated fields but aren't IT adjacent somehow get that idea that computers are not that difficult. Doing IT for doctors and lawyers is usually frustrating.
Some of the best hospital clients I've had have been doctors who are smart enough to understand that info tech knowledge is almost as important to them as medical knowledge. If you spend the time with them to advise on how to best implement their ideas for new systems/processes then you can end up with the best possible win/win result.
That said, you have to know what you are talking about or admit you don't. Doctors are used to being mislead by the best, and if you try to mislead or bullshit them they will know and you've lost their trust. Admit when you don't have in depth knowledge about what they are asking for and they will respect you for it. In a lot of cases you may be able to learn from them, as there interest is specific when you will have to deal with the entire environment.
The other important thing is don't waste their time. A lot of Doctors are working 18 to 20 hour days, and don't have time for you to be disorganised. If something is going to take a long time to do, then tell them so they can plan around the job. If a quick task suddenly looks like it's going to take longer, let them know as soon as you can so they can plan.
Keep at the top of your mind that the clinical staff you are supporting are their to save the lives of real people, they are not there at your convivence. Remember one day you will be a patient and you don't want your doctor to have to stuff around with a unhelpful IT specialist.
>What is rather unfortunate, is that we put more importance to our oil-infrastructure than our hospital infrastructure.
Just conjecture here, but this may be because the healthcare system is more resilient. Even as bad as it is, disruption to the healthcare system in these attacks is more local and more easily addressed by load shifting. Contrast that to oil infrastructure which may have more single points of failure as well as being more interconnected to the economy as a whole.
> Hospitals are why I don't "blame" underinvestment into cybersecurity. Their #1 goal is saving people's lives, not messing with IT issues. You want hospitals to be paying for important equipment, important people, important skills. The whole IT part is just supporting the administrative tasks.
And yet everything crumbles and collapses when there's an IT outage. How interesting.
These organizations might not be culturally accustomed to have IT at the core of their business/mission, but it very much is. They might not value engineering skills and people in IT, but they have evolved an absolute dependency on those over the years.
The issue here is cultural, not technical. These randsomware attacks, breaches and outages are completely self-imposed. They can end anytime as soon as the hospital wants it. All they have to do is value and acknowledge IT as a fundamental pillar of their organization. Else the cycle will endlessly repeat itself.
> Hospitals are incredibly important services that has under-invested into cybersecurity
You can't "invest" yourself to become secure.
In theory, perhaps, but it's going to cost you through the nose and beyond.
Reasonable security is reall very cheap, but involves saying "no" a lot. That's not something most organizations are very good at, so there's this whole cottage industry thriving on the promise that you don't have to if you buy a lot of expensive products instead.
This comment may seem flippant, but beneath that thin veneer it's completely serious. It's not primarily a question of money. No amount of money can upgrade an infrastructure in place to be secure.
I worked in hospital IT for a short time. A lot of the Windows XP machines are connected to that life-saving equipment and that is why there is no upgrade path. Putting administrators on a modern Windows is easy.
> At some point, it becomes more efficient to go after the hackers, rather than trying to defend every single Hospital.
I'm sorry, but this is completely backwards. It implies some global authority over communications, which is complete opposite of the Internet environment of communication in spite of hostile noise. Yeah sure it seems mighty cool that the US can pressure Russia to go after a notable group and shut them down. But thinking that can scale up to eliminating "Internet crime" is hopelessly naive. Unless we want to end up with a globally surveilled permission-required network where every node needs some associated identity, as well as making people even more liable for security failings (when their identity gets used as a proxy to attack others), it's a non-starter.
What needs to happy is that hospitals, every business, and really every individual needs to develop a small sense of network security. This is akin to how everybody has developed a basic intuition about electricity - ie don't touch it unless you know what you're doing or you will get shocked, start a fire, and/or die. The Internet is a multi-actor environment and connecting your stuff to a multi-actor environment is not free. If you want to avoid increasing the cost, knowing what you're doing can be simply consist of avoiding networked devices, getting explicitly security support and indemnification from the manufacturer, etc. The current culture of just plugging whatever in, proclaiming "works for me!", and then promptly forgetting there could be other implications is what's not sustainable.
> Their #1 goal is saving people's lives, not messing with IT issues.
If you're going to adopt a new tool, you maintain it. They seem to sterilize scalpels just fine, so they should be able to maintain second-order tools, too.
Medical devices running windows 7 or earlier are not allowed on networks anywhere. These devices connect through serial and are accessed over terminal servers. The terminal servers are the vulnerable point.
Our hospital system is actively collapsing right now. They have chronically underinvested in “run the business” activities and taken any capital out of the system through buybacks and large capital expense budgets for building new facilities. I don’t see any way out of this crisis other than public ownership of hospitals — or a lot fewer hospitals and a lot more people dying at home or in the street because the system was too broken to care for them.
Knowing this country, I’m sad that this choice is likely a foregone conclusion.
We don't have enough nurses/doctors to open new hospitals. It doesn't matter how much money is in the business if there's simply not enough nurses/doctors to go around.
We only don’t have enough doctors and nurses because many have simply left healthcare entirely due to the low pay, impossible conditions and how the administrative tasks of the hospital were being placed on doctors and nurses in addition to their existing jobs with no additional pay. This is all while hospital systems were doing large dividends and share buybacks to extract capital and return it to shareholders.
That’s a perfectly fine business model for a manufacturing plant, but we have to ask if that for-profit model makes sense for health care. With a public system, you can just decide to pay doctors and nurses more until you actually have enough of them to run the system. You can make cost / service level trade-offs intentionally rather than “how much capital can we extract before the whole thing collapses”?
Not to mention the almost impossible system of schooling, training, credentials, and general hoop-jumping to become a practicing doctor (and to reach that very desired $200k+ salary) means the supply of new doctors is incredibly constrained.
The amount of friends and peers of mine who gave up a career medicine because the ridiculousness of this whole system turned them off completely is really saddening.
I understand that we should be diligent about making sure the people we entrust our lives to are trained and trustworthy, but do we really need:
- 4 years of undergraduate studies that have ZERO medical treatment curricula
- 2-3 years of work experience if you don't get into medical school right away
- Studying for the MCAT concurrently and trying to get a high score
- 4 years of medical school
- A high stakes test that determines if you will receive the residency you want
- A lottery system that "matches" you with hospitals for residency
- 3-5 years of this residency in hopefully the specialization of your choice (depending on if you passed that test), hopefully in a location you desired. You will be paid very little and work 80+ hour weeks
If you track this entire system perfectly, you will become a full fledged doctor that makes the 6-figure salary at around 32 - 35 years old. And every step of the way is a huge filter that break and washout many promising potential doctors.
And then there is the medical school debt that you will be saddled with even if you washout.
This system is madness and we need something more efficient to both incentivize more people becoming doctors and less people washing out.
> And every step of the way is a huge filter that break and washout many promising potential doctors.
> And then there is the medical school debt that you will be saddled with even if you washout.
> This system is madness and we need something more efficient to both incentivize more people becoming doctors and less people washing out.
Who has the control over this? A legalized monopoly here in America (the AMA) that also famously restrict the number of available residency spots. This creates an artificial scarcity and props up the price of care for the public. Same organization that lobbied and got the government to create laws mandating “certificates of need” [0] to make sure they wouldn’t have to compete in a fair market.
This can end at any time. But it won’t because this would go against their interests.
I’ve dated a lot of doctors actually. The only one who made more than me was a plastic surgeon. Even the guy who was a cardiothoracic surgeon was barely matching my middle-management salary.
Side note, a lot of doctors in their late 30s are single for a reason. They literally do not know how to turn work off. Not a bad quality to have in a doctor, but also not a good one to have in a prospective partner.
I don’t know. Doctors work 80+ hour weeks, calls and holidays making $50,000 a year in their twenties and early 30s after paying for the privilege of grueling med school. Then they come out, maybe work 60-hours but still have the call, weekends and holidays. Then people kind of smugly think salaries like$300,000 a year is a lot. Well yeah; it’s-a lot but it took aheluva time getting to that point making about minimum wage and once they’re finished with training is not like they’re working 40 hour weeks with weekends and holidays off like most of us.
Residency programs are straight up exploitive and the crazy hours they work young doctors is dangerous.
Pilots have mandatory crew rest. Truck drivers have mandatory rest. Doctors need mandatory rest too. Any job that involves real risk to life should have rest procedures.
* Each duty period must begin with at least 10 hours off-duty.
* Drivers may work no more than 60 hours on-duty over seven consecutive days or 70 hours over eight days....
* Drivers may be on duty for up to 14 hours following 10 hours off duty, but they are limited to 11 hours of driving time.
I mostly agree, but I'll point out that the $300,000 / year part isn't the problem.
If we just raise the doctor's salaries to $500,000/year, it won't really solve those other, more important issues.
------
Similarly, if we lower the cost of creating Doctors, I don't think we'll necessarily see a drop in their salary. We're in too much of a doctor shortage for that to happen, at least immediately. (Of course, the market / supply+demand will shift things in the long run, but that's over a 20+ year cycle and not over a short one)
I have no doubt this is hard work, but is it smart work?
From having been around residents, I can tell there's a lot of work getting done 2-3 times because of poor communication or sleep deprived professionals making mistakes. And there’s absolutely no automation in the field!
Do I understand what you’re saying… doctors are working hard because they’re not working smart? Maybe the answer is they just need a sufficiently smart person to tell them how to work?
> doctors are working hard because they’re not working smart? Maybe the answer is they just need a sufficiently smart person to tell them how to work?
Correct. Duplicated work, endless shifts because it's believed that switching doctors is more dangerous than having one work for 24 hours at a time are all accepted practices in modern medicine.
NASA (and the military) figured out how to solve both issues back in the 60’s as it was impossible for manned flight to work (or nuclear subs). The medical field still has not, and there’s no incentives to (it’s not result-based, unlike NASA).
Crazy but true anecdote. I interviewed with this company for a DevOps-type position. My would-be manager spent his time with me talking about "the birds and the bees", and quipping that prisoners smuggle cell phones in their butt. The only question of his that I specifically remember is what video games do I play. I held it in, went home, and declined their offer. The recruiter told me that I was too sensitive, and ignored me about travel expense reimbursement for months, until I contacted their head of HR (guessing the address). I just checked and that would-be boss is now a senior manager of security engineering there. Not blaming him, but I do feel like Trinity dodging ten agents right now.
There was a flagged comment here that thought this manager's interviewing strategy was a good idea.
In this world, bosses end up holding power over their underlings. A boss who uses this power capriciously is tyrannical. A leader should use their power to achieve the shared mission of him and his followers, not arbitrarily.
There's a time and a place for ribaldry, and an interview isn't one. Generally those are optional situations that people can avoid if they don't want to hear it. This, and other sorts of 'tests' that some bosses use in interviews to test for 'thick-skinnedness' is equivalent to seeing if an underling will tolerate arbitrary abuses of power. It's equivalent to a test for absolute loyalty and servility, to see if the underling will be a yes man.
If a leader crosses a line for no good reason, perhaps by cracking a too risque joke, he or she should apologize and tone it down. It's about using your power responsibly and respecting your employees as you will have them respect you.
As a current prisoner, I'm not sure this guy had it right. It's usually the guards that supply cellphones and they just bring them in pockets or bags. Or nuns. I had a nun smuggle in a C# book with accompanying CD-ROM.
Drugs often travel in your "prison pocket", but I've never heard of anyone jamming a cellphone up there.
Luckily, before I went to jail, I had been given this book, the knowledge of which proved useful inside:
Reminds me of the Panera executive who got upset (seemed to be confused) when a security researcher wanted to exchange keys…. dude thought it was a scam / sales tactic.
Their stock looks fine. You'd never know their business is inoperable.
I know customers of theirs that just said 'screw it' and wrote their own payroll/timeclock systems. They don't have a 100% replacement yet (not a small project) but at least they can use cards to clock in and track hours.
I'm surprised every employee who uses the system hasn't had their personal information posted to the dark web yet.
Management pays Kronos because they feel like there's no options, even though their product is terrible and can be easily duplicated. I know this because I attempted to do it years ago. I was tasked with working with a consultant from Kronos to implement their "time management" system. I was surprised/not-surprised when I saw how lame it was. But I couldn't keep quiet after I heard we were paying Kronos something like $250K (early 2000s; can't remember exact $$$ amount, but it was a lot). I told my bosses I could make the same system over a weekend, and support it myself. Management resisted at first (Kronos has certifications!), but then they told Kronos we're thinking about rolling our own, thinking Kronos would drop the price by a few bucks. Nope. Kronos threatened to sue, saying we were attempting to steal IP.
> I told my bosses I could make the same system over a weekend, and support it myself... Kronos threatened to sue, saying we were attempting to steal IP.
If you were going to steal IP, stealing from Kronos would probably be pretty low on your list, because you're trying to build something that works, right?
I gather there's a strong tie-in with SAP payroll stuff (at least in sunny NZ): several of the organisations I've worked with who moved to SAP payroll/employee management have also taken kronos for timesheeting. And since you can't get fired for choosing SAP, Kronos keeps getting customers
The org I work for transitioned their Kronos from onsite to their multi-tenant cloud system. And it's been an absolute nightmare. Both software suites are a mess but transitioning to their cloud suite is like downgrading at least 10 years of upgrades.
Could it be because large investors in Kronos are also large investors in the companies that use Kronos, thereby having a vested interest in keeping the money flowing? Sincere question.
>A month-old ransomware attack is still causing administrative chaos for millions of people, including 20,000 public transit workers in the New York City metro area, public service workers in Cleveland, employees of FedEx and Whole Foods, and medical workers across the country who were already dealing with an omicron surge that has filled hospitals and exacerbated worker shortages.
I was surprised when it first happened there wasn't more publicity. To find out it is still going on a month later is jaw dropping.
Why don’t all critical applications at the very least not have immutable backups to a different account. This is easy to do in AWS. Ideally, these apps would also have better security posture, but doing this alone would go a long way. On my product, backup lambdas only have write access to our other account for backups. We’re backing up dynamodb and s3 continuously.
For a long time I used to work at one of Kronos's competitors. This space is so incredibly behind the times that it doesn't surprise me ar all. Up until recently time capture/entry software was still on premise (or even via paper time sheets!) For most large enterprises.
Calling something behind the times because it's on prem is exactly what caused these large corps to put their trust in this crappy vendor's 'cloud' which is what made it such a lucrative target in the first place. If every company we're running their infrastructure on prem this wouldn't have happened in quite the same way. So no, the 'cloud' is not always better just because it's the 'cloud'
If it were 'in the cloud', now the hacker has to interface with the time management / payroll service as if it were a web browser client trying to access it, assuming the network entry was via the hospital itself or some unsecure medical device physically present in the hospital. In absence of a properly-segmented LAN, it's better to have a segmented-by-design WAN in the form of SaaS and cloud vendor-based solutions.
Sort of, but for any reasonable enterprise the amount of human hours to work on and process them is a huge waste of money when payroll rules are supposed to be the same for all your employees (or at least the same for various groups)
I always wondered why. One of my assumptions is that since payroll/timekeeping does not really change, the incentive to update these systems is not there.
Also I bet these systems have lots of little moving parts under the surface no one really considers, but these little parts prevent an upgrade.
Kronos is very very good at account management, so baseball tickets and steak dinners. Everyone agrees their product is awful but it’s very hard to break that hold. (Disclosure: competitor)
I worked for a medium size payroll software company.
Payroll is highly regionalized problem - every state and city has different rules/taxes and very unique ones as well. Its often not so simple to generically describe a payroll tax and plug in different configs per region. Much hidden complexity that's grown organically over time as laws/taxes change (which they do!).
A rewrite would be an archaeological dig. I would have to be paid a lot to take that problem on.
It's also not trivial software to manage, so often it's outsourced as a service (run by humans) on top of software that cut the checks for your employees. Makes me think that the margins are low? I dunno.
My only question: Did Ultimate Kronos Group (UKG) pay the ransom? If UKG chose not to pay the ransom (the morally right thing to do), then I think we should cut them some slack. However, if UKG did pay the ransom, I hope they fail and go under because of this hack.
Maybe we need certifications for systems along with audits that would allow us to rate companies like this.
I know for our company to do work with DOD we had to meet a bunch of criteria and make changes to our systems to comply. But it wasn't a standardized process at all.
Nobody is objecting oversight, inspection and certification when in comes to physical infrastructure like bridges, tunnels, etc. I think we reached the point when critical software infrastructure and services should be treated the same way. Are there lessons to be learned from how they do it for DOD?
Tesla was impacted by this as well. Here is what Elon Musk wrote to all:
> Unfortunately, our payroll processor, Kronos, has been hit
with a ransomware attack, making them temporarily
unavailable. We are tracking things manually for now and
will issue pay manually, if they are unable to get back
online.
We are doing everything we can from our side. Sorry for
the trouble.
Elon
When you depend on third parties for critical but not directly business related tech you are just as vulnerable to disruption as if you directly got hacked. Even huge companies with ridiculous valuations can fail to audit indirect suppliers like payroll (i.e. Kronos) or air conditioning contractors, like in the famous Target hack.
The job of Tesla employees is to deliver the goods, even when it takes super-human efforts and creative miracles. The job of Musk is to pay them, even when it takes super-human efforts and creative miracles. No excuses.
While Musk deserves a huge amount of criticism, what part of "We are tracking things manually for now and will issue pay manually, if they are unable to get back online." sounds like an excuse to avoid the job of paying them?
Maybe I'm in the minority, but my issues with apologies from CEOs or companies is that they are generally lacking action or avoiding accountability. In this case, the apology like any somewhat genuine one adds to the note. It's the difference between "Sorry we lost your data." and "We have taken the following significant actions to make sure this never happens again, and have provided the following services and/or compensation to make it right to you. We are sorry."
And I'm someone who generally finds Musk hard to like.
It would be nice if there was a Linux alternative to whatever hospital infrastructure is still running windows xp. I mean it would be lucrative, secure, and even help to pay for things like salaries for programmer's wages, support foundations, and so on. And it'd be kept up to date unlike windows xp. Just a thought.
How much is the ransom? Is it being demanded of Kronos, their clients, or both? Is the demand believable, in that payment would have restored service?
We can talk a lot about the morality of paying ransoms, but should Kronos be putting their clients through so much to avoid it? Should they make that decision on their behalf?
Kronos attack timing correlates with log4j vulnerability wide announcement, but from what I've read, log4j vulnerability was not the attack vector used for the hack. At least it was not concerned as the attack vector for the hack.
Interesting observation that Kronos stock haven't dropped, but even have grown during the month since the hack. Either market is irrational, or it is perceived that impacted customers has no altetnstive. And it could be the later, since switching to a new payroll provider without the previous records is a huge undertaking.
TLDR: For anybody wondering why "hospitals" appears in the subject line, it's purely click-bait. Hospital services were not targeted or particularly affected more than any other industry.
The only statistic in the article that gives a clue how many hospitals were affected appears in the statement:
"In Montana, more than 250 nurses at Missoula's Community Medical Center have missed out on pay due to the hospital's decision to pay employees by duplicating an early December paycheck"
So in this particular company, some nurses were forced to accept their expected regular+OT pay and will have to wait a couple more weeks for any extra overtime they might be entitled to.
How many healthcare workers were affected? No more than any other industry. I couldn't find any news on the internet actually revealing how many workers in general were affected other than "up to thousands". So how many might be health care workers? Up to hundreds? So maybe 0.005% of healthcare workers have been inconvienced?
So my question is, why has NPR specifically addressed the impact to "hospitals"? Why is the impact to healthcare workers more important and news-worthy than to the impact to everybody else?
> "The outage is an unneeded administrative nightmare timed precisely as the omicron surge is hitting hospitals, Riggi said."
Ah! The outage was "timed"!!!!
The evil hackers intentionally timed the attack to threaten COVID victims!!!! My god! They're MONSTERS! attacking and murdering the weakest of us! It's outrageous!
What should be done? Is it's time to fire up the gas chambers for these inhuman hacker terrorists? Or maybe it just time to click on NPR's clickbait title?
> The evil hackers intentionally timed the attack to threaten COVID victims!!!! My god! They're MONSTERS! attacking and murdering the weakest of us! It's outrageous!
The hackers are criminals and extortionists. No more. The impact of the crime is embarrassment to businesses, time and money recovering from data loss, and an inconvenience to workers across all industries.
NPR played up an angle that doesn't exist in any meaningful or significant way. Why? More clicks.
I expect criminals to be assholes.
I expect more from NPR. They used to have more integrity and objectivity than they do today.
> I don't agree
> The hackers are criminals and extortionists.
I think where we disagree is the impact of the crime. I'm pretty sure the UKG hit is going to do a lot more damage to UKG's customer's customers (patients not getting good care because the hospital couldn't write a paycheck) that "embarrassment" and "data loss".
I personally know of at least one other hospital that was also affected and took the same actions of duplicating previous checks. So it's much more widespread than just one hospital at the very least.
> So how many might be health care workers? Up to hundreds?
Try tens of thousands of employees, easily. This outage affects more than just nurses trying to clock in. It's used for fire, police, dispatch, EMS, city utilities, auxiliary staff like the hospital cafeteria workers, and so on.
one reason i wouldn’t trust any metrics coming from hospitals and labs. they have terrible security and run lims systems from a handful of vendors . a small team of hackers could manipulate test results
Terrorism from within the country seems like it falls under the FBI’s purview. Malware from people residing in other countries seems like a job for a government agency that can operate outside US borders.
Western militaries have bombed schools, hospitals, and weddings across the world, and by your stated logic in this post you are personally morally responsible for this. Maybe getting a late paycheck and W-2 can be called tit for tat!
I don’t think GP’s logic extends to what you claimed. I took their post as saying all those who are being personally enriched by cryptocurrencies have to acknowledge and take responsibility that the one of most widely adopted, global use cases for crypto is allowing the ransomware industry to mature.
Your remark also assumes the poster is American. Assuming the morally dubious personal enrichment claim from GP, your statement would be true for anyone holding stock of Raytheon, Northrop Grumman, or Lockheed Martin, however.
By the username I assumed he was French, whose military frequently helps oppress and murder people across the MENA region.
Bitcoin is a currency, so no, any holder of any US or NATO allied regime's currency should be equally culpable as the currencies have their value rooted in military-enforced petrochemical trading monopolies. Dollars and francs both.
For a large part of the population (though probably not yet the majority) the reaction to this would be "...yes? That's why we are calling for policy changes and changing our consumption patterns"
Apples and oranges are both fruit but one can differentiate them.
A beggar does have the same impact on the climate as the CEO of Exxon. Asking someone not to support crypto is not the same as asking them to cease living.
If we're going down rabbit holes, then the internet is to blame. Oh wait, the internet is made up of a bunch of computers. Computers are to blame. Oh wait, computers require electricity. Electricity is to blame.
I believe they’re saying that crypto has enabled ransomware to become lucrative, and therefore all supporters of a decentralized payment method are also supporting digital piracy.
It’s like investing in a company that does bad things. Some people invest in oil and some don’t. People find ways to justify it to themselves but it is what it is. It’s not something to be proud of yet many clearly feel no shame. That will change as the problem gets bigger, which will happen if crypto continues to enjoy success.
With time it turned out the main use of cryptocurrencies (apart from speculation) is for illegal transactions, as they don’t normally manage to compete with legal transactions, but provide a way to avoid law enforcement while staying pseudonymous or anonymous when receiving or sending money.
By supporting cryptocurrency infrastructure you are indirectly supporting those illegal transactions. Now you could say the same e.g. for bakers that they also feed war criminals or whatever, however bread’s main use isn’t feeding criminals. It’s much more akin to providing money laundering services.
Hospitals are why I don't "blame" underinvestment into cybersecurity. Their #1 goal is saving people's lives, not messing with IT issues. You want hospitals to be paying for important equipment, important people, important skills. The whole IT part is just supporting the administrative tasks.
But yes, it means that paying the ransom is the better move a lot of the time than to actually try to restore IT services.
--------
At some point, it becomes more efficient to go after the hackers, rather than trying to defend every single Hospital.
Ex: When REvil accidentally hacked an oil-pipeline (instead of a more passive target), the blowback was so severe that REvil disbanded and ran away. It caused an international incident, to the point where Russia has caught the attackers and is offering them up to the USA as a peace offering.
What is rather unfortunate, is that we put more importance to our oil-infrastructure than our hospital infrastructure. But these ransomware attacks on health care has been going on for years. Its not new.