National Veterinary Associates (NVA), a California company that owns more than 700 animal care facilities around the globe, is still working to recover from a ransomware attack late last month that affected more than half of those properties, separating many veterinary practices from their patient records, payment systems and practice management software. NVA says it expects to have all facilities fully back up and running normally within the next week.
Agoura Hills, Calif.-based NVA bills itself as is the largest private owner of freestanding veterinary hospitals in the United States. The company’s Web site says it currently owns roughly 700 veterinary hospitals and animal boarding facilities in the United States, Canada, Australia and New Zealand.
NVA said it discovered the ransomware outbreak on the morning of Sunday, Oct. 27, and soon after hired two outside security firms to investigate and remediate the attack. A source close to the investigation told KrebsOnSecurity that NVA was hit with Ryuk, a ransomware strain first spotted in August 2018 that targets mostly large organizations for a high-ransom return.
NVA declined to answer questions about the malware, or whether the NVA paid the ransom demand.
“It was ransomware, but we’ve been referring to it as a malware incident,” said Laura Koester, NVA’s chief marketing officer.
Koester said because every NVA hospital runs their IT operations as they see fit, not all were affected. More importantly, she said, all of the NVA’s hospitals have remained open and able to see clients (animals in need of care), and access to patient records has been fully restored to all affected hospitals.
“For a few days, some [pet owners] couldn’t do online bookings, and some hospitals had to look at different records for their patients,” Koester said. “But throughout this whole thing, if there was a sick animal, we saw them. No one closed their doors.”
The source close to the investigation painted a slight less rosy picture of the situation at NVA, and said the company’s response has been complicated by the effects of wildfires surrounding its headquarters in Los Angeles County: A year ago, a destructive wildfire in Los Angeles and Ventura Counties burned almost 100,00 acres, destroyed more than 1,600 structures, killed three people and prompted the evacuation of nearly 300,000 people — including all residents of Agoura Hills.
“The support center was scheduled to be closed on Friday Oct 25, 2019 due to poor air quality caused by wildfires to the north,” said the source, who asked to remain anonymous. “Around 2 am PT [Oct. 27], the Ryuk virus was unleashed at NVA. Approximately 400 locations were infected. [Microsoft] Active Directory and Exchange servers were infected. Many of the infected locations immediately lost access to their Patient Information Management systems (PIMs). These locations were immediately unable to provide care.”
The source shared internal communications from different NVA executives to their hospitals about the extent of the remediation efforts and possible source of the compromise, which seemed to suggest that at least some NVA properties have been struggling to accommodate patients.
A missive from NVA’s Director of Operations Robert Hill on Oct. 30 acknowledged that “we continue to be faced with a monumental effort to restore IT service [to] nearly 400 of our hospitals.”
“This really hit home for me Saturday,” Hill wrote. “One of my best friends had to take his Yellow Lab into Conejo Valley for urgent care. Thankfully CV was able to provide care as their [systems] were up and running, but many of our hospitals are not in as good shape.”
In an update sent to NVA hospitals on Nov. 6, the company’s new head of technology Greg Hartmann said its security system successfully blocked the ransomware from infiltrating its systems — at least at first.
“Because of the scale of the attack, the virus eventually found three smaller points of entry through accounts that were unaffiliated with NVA, but unfortunately opened within our network,” Hartmann said. “Upon discovery of the incident, our technology team immediately implemented procedures to prevent the malware from spreading; however, many local systems were affected. Still, we have many hospitals whose systems are not recovered. The technology team continues to set up interim workstations at each affected hospital while they prepare to rebuild servers.”
The source told KrebsOnSecurity that NVA suffered a separate ransomware infestation earlier this summer that also involved Ryuk, and they expressed concern that the first incident may not have been fully remediated — potentially letting the attackers maintain a foothold within the organization.
“This is the second time this year Ryuk struck NVA,” the source said. “The first time, NVA was rather open to all facilities about what happened. This time, however, they are simply referring to it as a ‘system outage.'”
A set of talking points NVA distributed to staff on Oct. 27, the day some 400 veterinary hospitals were hit with the Ryuk ransomware.
Koester said some NVA facilities did get hit with a malware incident earlier this year, but that she did not believe ransomware was involved in that intrusion.
The Ryuk ransomware has made a name for itself going after businesses that supply services to other companies — particularly cloud-data firms — with the ransom demands set according to the victim’s perceived ability to pay. In February, payroll software provider Apex Human Capital Management chose to pay the ransom demand after a Ryuk infection severed payroll management services for hundreds of the company’s customers. And on Christmas Eve 2018, cloud hosting provider Dataresolution.net suffered a multi-week outage after a Ryuk attack.
According to a bulletin released by the FBI in May, cybercriminals had targeted over 100 U.S. and international businesses with Ryuk since August 2018. Security firm CrowdStrike estimated that attackers deploying Ryuk had netted over $3.7 million in bitcoin ransom payments between Aug. 2018 and January 2019.
Many people and organizations may be under the impression that ransomware attacks like Ryuk can appear at a moment’s notice merely from someone clicking a malicious link or opening a booby-trapped email attachment. While the latter appears to be the most common vector for ransomware infestations, an advisory released in September by the U.K’s National Cyber Security Centre suggests most Ryuk victims are compromised weeks or months before the ransomware is actually deployed inside the victim’s network.
“The Ryuk ransomware is often not observed until a period of time after the initial infection – ranging from days to months – which allows the actor time to carry out
reconnaissance inside an infected network, identifying and targeting critical network systems and therefore maximizing the impact of the attack,” reads the NCSC advisory, which includes tips on spotting signs of a Ryuk infection. “But it may also offer the potential to mitigate against a ransomware attack before it occurs, if the initial infection is detected and remedied.”
As for what changes NVA will be making to prevent yet another ransomware outbreak, an internal update on Nov. 7 from NVA’s chief information officer Joe Leggio said NVA was investing in software from Carbon Black, a cloud-based security solution that will be installed on all NVA property computers.
“Throughout my career, I have witnessed incredible advances in technology making our lives better,” Leggio wrote. “At nearly the same rate, the bad guys have been increasing the aggressiveness and sophistication of their attacks. As we rebuild, we are also thinking of the future. That is why we are investing in cybersecurity talent, new infrastructure, and better software.”
And now the state of Louisiana…
I’m glad NVA is “investing in cybersecurity talent, new infrastructure, and better software.” A little late, aren’t they!
As is too often the case, companies make those investments AFTER they’ve been hit. I call it the Target Syndrome — where Target failed (in 2013) to take some of the most basic steps to secure its systems BEFORE something happened.
Every time a press release says, “We take security seriously,” the reality is: they didn’t take it seriously enough.
only way to get a suit to spend money…. never trust a suit.
That and these companies trying to ‘one size fits all’ crap. Decentralize your software. Win. If there is a solution out there that magically fills all your bills you should be very, very skeptical.
Somewhere you lost your grasp on whats involved, when it comes to malware/ransomware infiltration. When a person is off the mark on how things can be handled, its definitely a red flag they don’t know much about internal security matters. Oh well…
Sony comes to mind. Hilarious.
They blamed NK.
The state of Louisiana , now dealing with this issue.
SNAFU vs. FUBAR (with a dash of BOHICA) is the conundrum faced by the unlucky or unwary in the modern commercial cyberworld.
It appears to me that they should have done some pen testing right after they cleared the first infection. With all those vet hospitals/facilities it would have been a good idea.
Regards,
Well at least it was only animals that were the most adversely affected, but to some, it may feel even worse.
If you don’t do something about it the first time you get hit, you deserve to get hit again. Not that I’m taking any responsibility away from the criminals; but when are they going to learn?
I’d wager it isn’t necessarily a software problem, as they seem to be focused on that, but an operating system wide problem. When are they going to realize you can set Active Directory and the whole MS universe to resist this malware? Or – maybe MS needs to come up with a way to lock encryption down and make it more difficult to use? I realize the malware can probably bring its own encryption with it, but that is not what I’m talking about.
Free data
And what payment ransome attackers accepting?
Its a Still Bitcoin like Old times? Or You Can now pay your ransome bill with other crypto currencies too?
It sounds like someone at NVA likes opening emails.
For many organizations the first vector of attack is their email system. Second – block, or better whitelist, web sites that staff can go to. Almost all ransomware and phishing attacks happen that way.
Additionally it wouldn’t hurt to train Susan in accounting what exactly to watch out for when she answers customer emails. But again, an automated filter in the email program that would not allow clicking links (other than the whitelisted ones) and would prevent opening any attachments (other than maybe .pdf) would be paramount. Also obviously keep it up to date and try to stay away from Adobe products.
This should cover 95% of cases.
firewall or AV would block those attachments 99% of the time… trust me my users (small %) still click even though they have been trained.
“Because of the scale of the attack, the virus eventually found three smaller points of entry through accounts that were unaffiliated with NVA, but unfortunately opened within our network,”
sounds more like the actors had admin and then deployed the “xware” to clients/servers… prob a 3rd party VPN that had full ad rights…. but who knows…
Agree up to a point. There are two levels (at least) of attack sophistication. Embedded links or malicious attachments is one level. Even here, I’ve seen some brilliantly constructed phishing emails, but we should have systems capable of handling these rather than relying on busy people doing the forensic examination that should be automated.
The second type are targeted and ‘clever’. The ANU published a good analysis of their attack. The malicious email was never ‘opened’. No problem was detected by scans. As part of their email triage, it was previewed, determined to be not valid, and deleted. No links were clicked and no attachments opened. In reality the preview action does ‘open’ the email – and that was enough. The attack described above is targeted at a particular type of services organisation. It is not part of the 95% of standard scripted attacks. Therefore well-meaning advice about training staff not to click links or open attachments is probably a non-sequitur for this level of attack.
Stop blaming the women in accounting. Last time you blamed Carol!
The fault is entirely with the idiots who put the practices/hospitals online, where it doesn’t belong.
I like the talking point:
“You can direct people to me via email not a phone call”
So you aren’t sure of the severity, may have been compromised multiple times, and the suggestion is to use email? How about we try some kind of out of band communications?
The goal is to ensure asynchronous communication. They don’t want to be caught in the spotlight…
guessing they used a gpo or sccm to hit all the clients since they had control of AD?
If they weren’t running Windows they wouldn’t have been infected. No sympathy.
“If they weren’t running Windows they wouldn’t have been infected”
I hope you wake up before you become infected –
https://www.zdnet.com/article/thousands-of-servers-infected-with-new-lilocked-lilu-ransomware/
Thousands of servers infected with new Lilocked (Lilu) ransomware
Researchers spot new ransomware targeting Linux-based servers.
Lulz @ the keyboard warriors pontificating about teaching users to avoid these attacks. Obvious who has never been tasked with making that work.
About a year ago I ran a phishing exercise testing our users. Well written bait designed to entice response. Caught a lot of people. As part of it I reviewed the so-called training provided to teach correct skepticism to users. Three of the four points recommended didn’t work if I was reading the email from a mobile device. But who reads email from their phone these days, eh? And every point could be automated to check and inform users instead of expecting them to manually inspect.
So yeah blame the users. Much easier than actually fixing the technology.
The fault is not the users. You’re right.
The fault is with those who sell, design and administer ONLINE appointment and veterinary practice management systems.
It’s sold to them with the promise of efficiency and convenience. Instead it raises risks to their clients, records, smooth practice flow, and time.
The cloud model is broken.
During containment and even the recovery phases of this incident, the NVA was forcing all of their clinics to begin cutting the hours of full-time workers, and forcing some to go home since they “were not doing anything”. The NVA did not have any server or VM backups which is the number one resolution to a ransomware attack.
On top of this, a local clinic that was recently purchased by the NVA had to resort to buying their old computers back from their old IT admin to restore their backups from ~3 months ago. The NVA’s cybersecurity implementation is ridiculous.
This sounds more like a press release than a news article. This company seems inept to its core, and their explaination of the basics doesn’t pass the most basic of fact checking.
All domestic practices are either on their network (and locked down in a vm for local network computers), or were in the process of converting to their cms system. Sounds to me like their backup and update hygiene was non-existent.
If they had anything like an IT team on staff… oh never mind.
I think I will back up my Computer tonight.
If you are looking for affordable vet services in Canada I would recommend the website http://www.staplesanimalhospital.com/
There is a time and place for hawking your wares.
Advertising for coffins during a funeral isn’t one of them.
My vet is one of those that is part of NVA, and they are STILL affected by this. I tried to make a well visit appointment today (got a new pet) and was told that they don’t have access to their patient records (and I guess no way to add new records), so they are only taking appointments for sick pets. They said they had been told it was supposed to get fixed over Thanksgiving weekend, so we’ll see what happens when I call them on Monday.
A few months back, I work in Dallas at a 24 hour veterinary hospital. Our system was hijacked by ransomware. Took a week to get back and running.