The consulting firm PricewaterhouseCoopers recently published lessons learned from the disruptive and costly ransomware attack in May 2021 on Ireland’s public health system. The unusually candid post-mortem found that nearly two months elapsed between the initial intrusion and the launching of the ransomware. It also found affected hospitals had tens of thousands of outdated Windows 7 systems, and that the health system’s IT administrators failed to respond to multiple warning signs that a massive attack was imminent.
PWC’s timeline of the days leading up to the deployment of Conti ransomware on May 14.
Ireland’s Health Service Executive (HSE), which operates the country’s public health system, got hit with Conti ransomware on May 14, 2021. A timeline in the report (above) says the initial infection of the “patient zero” workstation happened on Mar. 18, 2021, when an employee on a Windows computer opened a booby-trapped Microsoft Excel document in a phishing email that had been sent two days earlier.
Less than a week later, the attacker had established a reliable backdoor connection to the employee’s infected workstation. After infecting the system, “the attacker continued to operate in the environment over an eight week period until the detonation of the Conti ransomware on May 14, 2021,” the report states.
According to PWC’s report (PDF), there were multiple warnings about a serious network intrusion, but those red flags were either misidentified or not acted on quickly enough:
- On Mar. 31, 2021, the HSE’s antivirus software detected the execution of two software tools commonly used by ransomware groups — Cobalt Strike and Mimikatz — on the Patient Zero Workstation. But the antivirus software was set to monitor mode, so it did not block the malicious commands.”
- On May 7, the attacker compromised the HSE’s servers for the first time, and over the next five days the intruder would compromise six HSE hospitals. On May 10, one of the hospitals detected malicious activity on its Microsoft Windows Domain Controller, a critical “keys to the kingdom” component of any Windows enterprise network that manages user authentication and network access.
- On 10 May 2021, security auditors first identified evidence of the attacker compromising systems within Hospital C and Hospital L. Hospital C’s antivirus software detected Cobalt Strike on two systems but failed to quarantine the malicious files.
- On May 13, the HSE’s antivirus security provider emailed the HSE’s security operations team, highlighting unhandled threat events dating back to May 7 on at least 16 systems. The HSE Security Operations team requested that the Server team restart servers.
By then it was too late. At just after midnight Ireland time on May 14, the attacker executed the Conti ransomware within the HSE. The attack disrupted services at several Irish hospitals and resulted in the near complete shutdown of the HSE’s national and local networks, forcing the cancellation of many outpatient clinics and healthcare services. The number of appointments in some areas dropped by up to 80 percent.”
Conti initially demanded USD $20 million worth of virtual currency in exchange for a digital key to unlock HSE servers compromised by the group. But perhaps in response to the public outcry over the HSE disruption, Conti reversed course and gave the HSE the decryption keys without requiring payment.
Still, the work to restore infected systems would take months. The HSE ultimately enlisted members of the Irish military to bring in laptops and PCs to help restore computer systems by hand. It wasn’t until September 21, 2021 that the HSE declared 100 percent of its servers were decrypted.
As bad as the HSE ransomware attack was, the PWC report emphasizes that it could have been far worse. For example, it is unclear how much data would have been unrecoverable if a decryption key had not become available as the HSE’s backup infrastructure was only periodically backed up to offline tape.
The attack also could have been worse, the report found:
- if there had been intent by the Attacker to target specific devices within the HSE environment (e.g. medical devices);
- if the ransomware took actions to destroy data at scale;
- if the ransomware had auto-propagation and persistence capabilities, for example by using an exploit to propagate across domains and trust-boundaries to medical devices (e.g. the EternalBlue exploit used by the WannaCry and NotPetya15 attacks);
- if cloud systems had also been encrypted such as the COVID-19 vaccination system
The PWC report contains numerous recommendations, most of which center around hiring new personnel to lead the organization’s redoubled security efforts. But it is clear that the HSE has an enormous amount of work ahead to grow in security maturity. For example, the report notes the HSE’s hospital network had over 30,000 Windows 7 workstations that were deemed end of life by the vendor.
“The HSE assessed its cybersecurity maturity rating as low,” PWC wrote. “For example, they do not have a CISO or a Security Operations Center established.”
PWC also estimates that efforts to build up the HSE’s cybersecurity program to the point where it can rapidly detect and respond to intrusions are likely to cost “a multiple of the HSE’s current capital and operation expenditure in these areas over several years.”
One idea of a “security maturity” model.
In June 2021, the HSE’s director general said the recovery costs for the May ransomware attack were likely to exceed USD $600 million.
What’s remarkable about this incident is that the HSE is publicly funded by the Irish government, and so in theory it has the money to spend (or raise) to pay for all these ambitious recommendations for increasing their security maturity.
That stands in stark contrast to the healthcare system here in the United States, where the single biggest impediment to doing security well continues to be lack of making it a real budget priority. Also, most healthcare organizations in the United States are private companies that operate on razor-thin profit margins.
I know this because in 2018 I was asked to give the keynote at an annual gathering of the Healthcare Information Sharing and Analysis Group (H-ISAC), an industry group centered on sharing information about cybersecurity threats. I almost didn’t accept the invitation: I’d written very little about healthcare security, which seemed to be dominated by coverage of whether healthcare organizations complied with the letter of the law in the United States. That compliance centered on the Health Insurance Portability and Accountability Act (HIPAA), which prioritizes protecting the integrity and privacy of patient data.
To get up to speed, I interviewed over a dozen of the healthcare security industry’s best and brightest minds. A common refrain I heard from those interviewed was that if it was security-related but didn’t have to do with compliance, there probably wasn’t much chance it would get any budget.
Those sources unanimously said that however well-intentioned, it’s not clear that the “protect the data” regulatory approach of HIPPA was working from an overall threat perspective. According to HealthcareIT News, more than 40 million patient records have been compromised in incidents reported to the federal government in 2021 so far alone.
During my 2018 talk, I tried to emphasize the primary importance of being able to respond quickly to intrusions. Here’s a snippet of what I told that H-ISAC audience:
“The term ‘Security Maturity’ refers to the street smarts of an individual or organization, and this maturity generally comes from making plenty of mistakes, getting hacked a lot, and hopefully learning from each incident, measuring response times, and improving.
Let me say up front that all organizations get hacked. Even ones that are doing everything right from a security perspective get hacked probably every day if they’re big enough. By hacked I mean someone within the organization falls for a phishing scam, or clicks a malicious link and downloads malware. Because let’s face it, it only takes one screw up for the hackers to get a foothold in the network.
Now this is in itself isn’t bad. Unless you don’t have the capability to detect it and respond quickly. And if you can’t do that, you run the serious risk of having a small incident metastasize into a much larger problem.
Think of it like the medical concept of the ‘Golden Hour:’ That short window of time directly following a traumatic injury like a stroke or heart attack in which life-saving medicine and attention is likely to be most effective. The same concept holds true in cybersecurity, and it’s exactly why so many organizations these days are placing more of their resources into incident response, instead of just prevention.”
The United States’ somewhat decentralized healthcare system means that many ransomware outbreaks tend to be limited to regional or local healthcare facilities. But a well-placed ransomware attack or series of attacks could inflict serious damage on the sector: A December 2020 report from Deloitte says the top 10 health systems now control a 24 percent market share and their revenue grew at twice the rate of the rest of the market.
In October 2020, KrebsOnSecurity broke the story that the FBI and U.S. Department of Homeland Security had obtained chatter from a top ransomware group which warned of an “imminent cybercrime threat to U.S. hospitals and healthcare providers.” Members associated with the Russian-speaking ransomware group known as Ryuk had discussed plans to deploy ransomware at more than 400 healthcare facilities in the United States.
Hours after that piece ran, I heard from a respected H-ISAC security professional who questioned whether it was worth getting the public so riled up. The story had been updated multiple times throughout the day, and there were at least five healthcare organizations hit with ransomware within the span of 24 hours.
“I guess it would help if I understood what the baseline is, like how many healthcare organizations get hit with ransomware on average in one week?” I asked the source.
“It’s more like one a day,” the source confided.
In all likelihood, the HSE will get the money it needs to implement the programs recommended by PWC, however long that takes. I wonder how many U.S.-based healthcare organizations could say the same.
How interesting so you are indicating that most Hospitals still have their heads in the sand about the Ransomware / Malware threat? How is that possible?
They don’t have the money for it.
Perhaps the “tl;dr version” of this story reads something like this:
Clueless employee opens document attached to email from a source they may or may not recognize. Said document was allowed into the organization’s network due to lax email security; failure to scan all incoming & outgoing emails & their attachments. Additionally, said organization may have failed to employ any controls on the use of documents containing embedded scripting languages.
The fact that this organization uses an older operating system may or may not be of any consequence; the article does not show how it is implicated in this matter despite calling it “outdated”.
So Sleepy, I guess your recommendations would be
a) Clueless employee: Better education and training
b) Lax email security: Improve email security
b.1) failure to scan: Scan all emails and attachments
b.2) no control of documents containing embedded scripts: Block scripted content
Okay, we will strictly follow your recommendations. And than we get hit again! The employee double checked the attached document, but it was artificially crafted not to arouse suspicion. The mail was scanned, but no scanner detected the attachment as malicious. A macro milter was active, but unfortunately the malware author used some very clever techniques to mask the scripts.
We followed all your recommendations – but nevertheless, our organisation got hit, our documents encrypted and we can choose between paying ransom or rebuild our organisation from the scratch!
What went wrong?
It is possible that outdated systems such as Windows XP and Windows 7 can still safely operate, provided that they are isolated to a private VLAN and can communicate on only the expected port(s). The hospitals in the story above don’t sound as though they set up this type of network.
It was MS Excel on desktop + phishing success and “maybe” 10 would have blocked it? Maybe?
Either way, GET RID of the MS office suite already! That’s a bigger phishing hole than 7 a la carte.
Get rid of the MS office suite, and replace it with…what? LibreOffice or OpenOffice which just 2 years ago had the mouseover remote code execution vulnerability. Who’s to say there aren’t more there, just as with MS Office?
Of course there are “more than 1” but you aren’t making a serious comparison to any.
The Word/Excel macro surprise ingress has been going on for decades unabated.
Literally any other option is better than MS and getting locked into that is pointless.
OSS all the way. Far, far fewer bugs and millions of lines less code and feature creep.
But you do you, lol.
You are missing a key element of security in an organization. You must balance user experience and security, and by switching to open office or Libre you are alienating 95% of your user base to a product they know nothing about. In theory it sounds simple, but MS has much better support and patches on a more regular basis. This reminds me of the Mac’s cant get malware crap that was being spewed in the early 2000’s. Its not that Mac’s weren’t vulnerable its just people did not use them as much as they covered like 1% of all devices in use, so it just wasn’t enticing enough for bad actors to target them.
Government resources aren’t the issue. Let’s just look back at every security review of US Federal departments, not least Defense (which I guess has a bigger budget than the whole Irish government), to see that will and competence are the relevant factors.
@Moike
Makes sense. I’m just a dummy, but real experts I read have been saying for a long time that not everything needs to be connected to the web/internet(yeah I know they’re different). It seems to be a no brainer, but I’m just a dummy.
It’s long been a mantra that “If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.” (Weinberg’s Law, circa 1975)
What we have done since this law was formulated is to put all of our sensitive data, necessary processes, and critical infrastructure on systems and software subject to this law: We have willfully put our own lives at the mercy of any hacker (or woodpecker) who comes along.
Why? Greed. To save money, we don’t bother with security at any level. “Dave” clicks on any link and doesn’t care. The middle management pushes aside security to “get the job done”. The C-Suite parrots the mantra that “everyone gets hacked eventually” so they can avoid responsibility by claiming that it’s “the industry standard” to throw up their hands in defeat when asked to implement security. EVERYONE figures that security is “Someone Else’s Problem”.
Right now, we know beyond any doubt that authoritarian regimes are hacking into anything that they can get into, with the obvious motive to cause massive disruption if democratic nations object or interfere when the authoritarian regimes invade Taiwan/Ukraine/South Korea/whatever.
Disconnect your sensitive data from the Internet NOW.
I wish the simple assessment offered by Sleepy and the simple non-solution offered by schorsch were sufficient, but the defenses are only as strong as the weakest link, and we all have weaknesses somewhere. A mature and comprehensive Information Security Program that addresses the entire set of InfoSec requirements, meeting all parts of a good framework, ISO or NIST, is needed to reduce the risk of material harm. Granted, that is expensive, but so are hacks!
Pick a framework, assess the maturity of all your practices, and prioritize areas for improvement, and up your game!
*HIPAA
(don’t feel bad, Brian — there are plenty of healthcare-focused professionals that make the same error re: that acronym)
HIPAA provides high-level/non-specific requirements for security safeguards. The US gov’t is currently pushing out more specific guidance to healthcare entities more focused on practical threat management via the 405d workgroup “healthcare industry cybersecurity practices” which may be of interest to some of your readers
The main message I take away is that in the US, healthcare companies aren’t prioritizing the PII of their customers, staff and the security and infrastructure of their own organizations. This messages resonates with my experience as a consumer and “security nerd” that I have become.
I appreciate your report Brian and the openness of all involved in sharing this with a larger audience.
The fact that they’re only paying attention to HIPAA compliance tells you that the only way to have cybersecurity taken seriously is to make companies liable for lax security practices (and software suppliers liable for code vulnerabilities). That is already happening with insurance companies reacting to the growing prevalence of ransomware, but it would be nice if we didn’t concede the power of regulation to the insurance industry, since there is inevitably going to be collateral damage (i.e., hospitals closing), while the incompetent managers walk away to another gig.
Who in the world still even relies upon a machine running Windows 7? I had a beloved Windows 8 laptop that I had purchased used for just a few dollars on eBay and when I was done with it many years later, someone introduced me to a mentally disabled ex-convict who needed a computer. He’s still using it a few years later, and one of his caretakers contacted me recently because they don’t know what to do about upgrading the operating system from Windows 8, because now the most basic of tasks is becoming impossible for him to complete. And that’s Windows 8, not even Windows 7. My point with this rambling statement is to inquire how is it possible for a national government-sponsored healthcare system to run anything on a network of Windows 7 computers?
Rather a bizarre post. Windows 7 was Microsoft’s last decent OS. You offer no evidence at all that a later Microsoft OS would have been more secure. Windows 8 was catastrophically poor from UI point of view an 10 wasn’t much better. I, like many, stuck w Windows 7 out of choice until I switched to Linux. A publicly funded health care system has many calls on its resources and money spent on IT is money not spent on treating the sick. It’s only when the cost of not spending is sufficiently clear that it is possible to do it. In any case, Ireland was and remains a country where the nr of people who go bankrupt annually because of medical bills is 0, and where the US approach to health care is reviled.
Makes me wonder if they had legacy software that doesn’t run on more modern software. I’ve seen that in the past but can’t speak for the 30k Win 7 computers they have/had in use.
You’re not considering legacy/hw support issues. There are Xp machines running for a reason.
Some may say the reason isn’t a “good” one, but if there aren’t funds to completely replace an
already bought system (and there aren’t) if you want those (big $) to work you need legacy OS.
There’s nothing too wrong with that, so long as you keep them hygienic and off the www.
It’s unclear if 10 would have even stopped this phishing + macro payload either, probably not.
You may be surprised to know Windows 10 doesn’t keep you that much safer by default.
Anyone expecting a vastly improved security paradigm from 11? Buy my magic beans too.
Fundamental failure of system design and security architecture. By refusing to modernize (to the 1990s, at least!), organizations make their environment subject to these rudimentary attacks. EXPECT users to infect their systems! Build your defenses so their doing so can’t impact real data or systems. Fix it!
https://youtu.be/Xi7XmXEO1hw
So where is all of the money going from the people that were bankrupted by unreimbursed health expenses, since hospitals claim that they do not have the money to upgrade their IT infrastructure? This seems to be a common theme when something goes wrong–no money. No money to repair and rebuild bridges and road, no money to train employees, no money to upgrade factories to compete with the advanced factories in China and elsewhere, no money to establish broadband throughout the US, etc. So where is all of the money going? CEO and Board of Director and shareholder, i.e., pension and hedge fund, pockets? It seems that organizations do not have money for ongoing maintenance, but they have money for everything else. Hackers are doing companies and us favor, since they putting the spotlight on bad practices. I just hope that the government does not give even more money to corporations to improve their security practices, since that is something that should be factored into their operating costs to begin with. I guess hospitals will be instituting a “IT security surcharge” as a line item on doctor and hospital bills pretty soon instead of adjusting their overall operating costs and increasing overall prices.
The better one would be to reduce the financial waste and keep the price the same or even reduce
They are using Windows 7 because they are unable to upgrade the proprietary medical software which was written for Windows 7 machines and will not operate on later versions of the Microsoft Operating System. This custom medical software cost over fifty million euros but is completely monolithic and cannot be easily ported onto modern MS versions without more consultants fees of tens of millions of euro.
Also the HSE received a decryption key because ofthe influence of Irish diplomats who raised the problem with Russian delegates on the UN Security Council of which Ireland had recently beome a temporary member. According to the irish Times, Irish Foreign Minister Simon Coveney spoke to Sergey Lavrov during a UN Security Council meeting about the ransomware. Within days the decryption key was handed over. https://www.irishtimes.com/news/health/coveney-discusses-health-service-cyberattack-with-russian-foreign-minister-1.4568264
Not all organizations get hacked. Some chose to not to be connected to the Internet. In the case of HSE, they could have chosen to have dedicated lines interconnecting their hospitals, and have no Internet connection. Dedicated lines would have been a heck of a lot cheaper than the cost of this one hack.
One thing I see frequently is that there’s another dimension/gap between “Cybersecurity is a necessary evil” and “Cybersecurity is part of the culture”: that is, whether corporate actions in service of cybersecurity are actually meaningful. As part of a security-focused product group in a much larger company, we frequently see “cybersecurity” actions that we know are not useful. This tends to discount the value of actual security by making us take corporate’s actions less seriously, including finding ways to work around them. That’s not good in a number of obvious ways.
If windows of all flavors is a security risk from executing embedded macro languages then switch operating systems to NOT EXECUTE embedded macro languages. Cut the exoloit surface. https://ghostbsd.org. or
https://pop.system76.com/
Need to run windows legacy programs? Use WINE.
Need active domain? Use Samba.
How much money in ransome or lost availability, before you switch from a faulty or crumbling foundation operating system windows to BSD or Linux.
Prevent Dan , from being a priblem security risk by opening a phishy email.
Yes, this takes work to move forward from a security failure.