Organizations in the throes of cleaning up after a ransomware outbreak typically will change passwords for all user accounts that have access to any email systems, servers and desktop workstations within their network. But all too often, ransomware victims fail to grasp that the crooks behind these attacks can and frequently do siphon every single password stored on each infected endpoint. The result of this oversight may offer attackers a way back into the affected organization, access to financial and healthcare accounts, or — worse yet — key tools for attacking the victim’s various business partners and clients.
In mid-November 2019, Wisconsin-based Virtual Care Provider Inc. (VCPI) was hit by the Ryuk ransomware strain. VCPI manages the IT systems for some 110 clients that serve approximately 2,400 nursing homes in 45 U.S. states. VCPI declined to pay the multi-million dollar ransom demanded by their extortionists, and the attack cut off many of those elder care facilities from their patient records, email and telephone service for days or weeks while VCPI rebuilt its network.
Just hours after that story was published, VCPI chief executive and owner Karen Christianson reached out to say she hoped I would write a follow-up piece about how they recovered from the incident. My reply was that I’d consider doing so if there was something in their experience that I thought others could learn from their handling of the incident.
I had no inkling at the time of how much I would learn in the days ahead.
EERIE EMAILS
On December 3, I contacted Christianson to schedule a follow-up interview for the next day. On the morning of Dec. 4 (less than two hours before my scheduled call with VCPI and more than two weeks after the start of their ransomware attack) I heard via email from someone claiming to be part of the criminal group that launched the Ryuk ransomware inside VCPI.
That email was unsettling because its timing suggested that whoever sent it somehow knew I was going to speak with VCPI later that day. This person said they wanted me to reiterate a message they’d just sent to the owner of VCPI stating that their offer of a greatly reduced price for a digital key needed to unlock servers and workstations seized by the malware would expire soon if the company continued to ignore them.
“Maybe you chat to them lets see if that works,” the email suggested.
The anonymous individual behind that communication declined to provide proof that they were part of the group that held VPCI’s network for ransom, and after an increasingly combative and personally threatening exchange of messages soon stopped responding to requests for more information.
“We were bitten with releasing evidence before hence we have stopped this even in our ransoms,” the anonymous person wrote. “If you want proof we have hacked T-Systems as well. You may confirm this with them. We havent [sic] seen any Media articles on this and as such you should be the first to report it, we are sure they are just keeping it under wraps.” Security news site Bleeping Computer reported on the T-Systems Ryuk ransomware attack on Dec. 3.
In our Dec. 4 interview, VCPI’s acting chief information security officer — Mark Schafer, CISO at Wisconsin-based SVA Consulting — confirmed that the company received a nearly identical message that same morning, and that the wording seemed “very similar” to the original extortion demand the company received.
However, Schafer assured me that VCPI had indeed rebuilt its email network following the intrusion and strictly used a third-party service to discuss remediation efforts and other sensitive topics.
‘LIKE A COMPANY BATTLING A COUNTRY’
Christianson said several factors stopped the painful Ryuk ransomware attack from morphing into a company-ending event. For starters, she said, an employee spotted suspicious activity on their network in the early morning hours of Saturday, Nov. 16. She said that employee then immediately alerted higher-ups within VCPI, who ordered a complete and immediate shutdown of the entire network.
“The bottom line is at 2 a.m. on a Saturday, it was still a human being who saw a bunch of lights and had enough presence of mind to say someone else might want to take a look at this,” she said. “The other guy he called said he didn’t like it either and called the [chief information officer] at 2:30 a.m., who picked up his cell phone and said shut it off from the Internet.”
Schafer said another mitigating factor was that VCPI had contracted with a third-party roughly six months prior to the attack to establish off-site data backups that were not directly connected to the company’s infrastructure.
“The authentication for that was entirely separate, so the lateral movement [of the intruders] didn’t allow them to touch that,” Schafer said.
Schafer said the move to third-party data backups coincided with a comprehensive internal review that identified multiple areas where VCPI could harden its security, but that the attack hit before the company could complete work on some of those action items.
“We did a risk assessment which was pretty much spot-on, we just needed more time to work on it before we got hit,” he said. “We were doing the right things, just not fast enough. If we’d had more time to prepare, it would have gone better. I feel like we were a company battling a country. It’s not a fair fight, and once you’re targeted it’s pretty tough to defend.”
WHOLESALE PASSWORD THEFT
Just after receiving a tip from a reader about the ongoing Ryuk infestation at VCPI, KrebsOnSecurity contacted Milwaukee-based Hold Security to see if its owner Alex Holden had any more information about the attack. Holden and his team had previously intercepted online traffic between and among multiple ransomware gangs and their victims, and I was curious to know if that held true in the VCPI attack as well.
Sure enough, Holden quickly sent over several logs of data suggesting the attackers had breached VCPI’s network on multiple occasions over the previous 14 months.
“While it is clear that the initial breach occurred 14 months ago, the escalation of the compromise didn’t start until around November 15th of this year,” Holden said at the time. “When we looked at this in retrospect, during these three days the cybercriminals slowly compromised the entire network, disabling antivirus, running customized scripts, and deploying ransomware. They didn’t even succeed at first, but they kept trying.”
Holden said it appears the intruders laid the groundwork for the VPCI using Emotet, a powerful malware tool typically disseminated via spam.
“Emotet continues to be among the most costly and destructive malware,” reads a July 2018 alert on the malware from the U.S. Department of Homeland Security. “Its worm-like features result in rapidly spreading network-wide infection, which are difficult to combat.”
According to Holden, after using Emotet to prime VCPI’s servers and endpoints for the ransomware attack, the intruders deployed a module of Emotet called Trickbot, which is a banking trojan often used to download other malware and harvest passwords from infected systems.
Indeed, Holden shared records of communications from VCPI’s tormentors suggesting they’d unleashed Trickbot to steal passwords from infected VCPI endpoints that the company used to log in at more than 300 Web sites and services, including:
-Identity and password management platforms Auth0 and LastPass
-Multiple personal and business banking portals;
-Microsoft Office365 accounts
-Direct deposit and Medicaid billing portals
-Cloud-based health insurance management portals
-Numerous online payment processing services
-Cloud-based payroll management services
-Prescription management services
-Commercial phone, Internet and power services
-Medical supply services
-State and local government competitive bidding portals
-Online content distribution networks
-Shipping and postage accounts
-Amazon, Facebook, LinkedIn, Microsoft, Twitter accounts
Toward the end of my follow-up interview with Schafer and VCPI’s Christianson, I shared Holden’s list of sites for which the attackers had apparently stolen internal company credentials. At that point, Christianson abruptly ended the interview and got off the line, saying she had personal matters to attend to. Schafer thanked me for sharing the list, noting that it looked like VCPI probably now had a “few more notifications to do.”
Moral of the story: Companies that experience a ransomware attack — or for that matter any type of equally invasive malware infestation — should assume that all credentials stored anywhere on the local network (including those saved inside Web browsers and password managers) are compromised and need to be changed.
Out of an abundance of caution, this process should be done from a pristine (preferably non-Windows-based) system that does not reside within the network compromised by the attackers. In addition, full use should be made of the strongest method available for securing these passwords with multi-factor authentication.
The thing that I think people don’t understand.
They see ransomware as a -=SOLITARY=- issue.
Ransomware gets in, encrypts and then you either pay the ransom or go to your backups.
But this is “IT” (eye-tee) thinking.
What they are failing to understand is HOW the ransomware got there and what it’s doing.
If it wrote itself to the system, what would make you think they are stopping there?
Listen, please don’t think like an IT shop!
If you had a ransomware infection, ASSUME the cyber-actor has accessed every system/file/process in your organization.
And this should include passwords and their stores.
Start encrypting….EVERYTHING!!!!
This frustrates me to no end
IT shops pretending to do security and taking their clients down with them.
Attention IT shops….partner with a MSSP….PLEASE!!!
I’m not sure if encryption would be a good defense technique against ransomware, but I’m honestly not 100% sure. My thinking is because, say, it is drive-level ransomware. The ransomware will just encrypt the whole drive and not care about your files. Or vice-versa, if you’re encrypting your whole drive from your end, the ransomware is still going to do its file-level encryption without ever knowing or caring that you use encryption yourself.
Your right the ransomware just encrypts encrypted files so encryption will not protect against ransomware. The only viable solution at present as far as I am aware is protected folders but this is a windows technology and not yet available on other platforms to my knowledge.
If they have compromised all of your accounts and passwords, then encryption is most likely irrelevant. Applications still decrypt the data to make it human readable. That is why the attacks happen at the human layer.
And look at the timeline. First entry point 14 months before. They then sit and let the clock tick down. Total horizontal penetration four weeks before. Then Dec 3 event. Restore from backup – sure for data – but your server backups are all compromised server images. The servers need to be rebuilt from scratch. Even if you pay the ransom, the massive clean-up is still required, because they own every privileged account.
Can we change your term to legacy IT shops, Frank. Connecting to that naughty internet has been a game changer but many practices haven’t changed, and additional tools are required. Nevertheless good IT shops are aware.
“I had no inkling at the time of how much I would learn in the days ahead.”
…that promoted nostalgia – I remember in the 80’s when we worked toward a wonderful new age when all our computer systems would be linked together! We had no inkling it would lead to this level of crime and fraud.
And in so many ways seems “we” still have not learned. It seems the entire industry has still failed to get on the wagon of building secure code, using tool sets that require more secure data structures, and implementing some form of user/process isolation that would help reduce the propagation.
The other aspect is that most security seems to be developed in the context of security experts and assumes a type of behavior and thinking by the general user community that simply does not exist. Something akin to designing security in an ideal world and then trying to force the less than ideal world to conform. That solution cannot be the best.
. . . Not to mention giving rise to all the “evils” of social networks that allow disinformation to be spread to people who get their news and “reality” from FaceCrook and sites like that — sites that are just interested money making — damn the facts.
If only we had the foresight to not connect our computers . . .
So, Bryan, what about your point of the attackers somehow knowing about your upcoming conversation with the owner of that company? It seems like they can read her emails too.
Or, perhaps they are reading Bryan’s email…..just sayin’
O365 credentials were among those stolen.
“While it is clear that the initial breach occurred 14 months ago, the escalation of the compromise didn’t start until around November 15th of this year,”
Prevention is key, but detection is a must. If we spend more efforts on detecting rather than placing a false trust in our prevention security software’s “block list”, I would be willing to bet that these times taken to discover a breach would be drastically lower. Obviously, preventive based security solutions are essential, but we need to place a stronger emphasis on detecting threats.
The hijacking of data and disruption of access to data in health care sector where lives depend upon the information is nothing short of domestic terrorism (assuming they are within the continental US) or foreign terrorism if located elsewhere.
“a company battling a country” – this is what I’ve been telling others. It’s not a fair fight at all. Companies are out-gunned. The hacking tools used by the bad actors far exceed the sophistication of anything at the disposal of an ordinary US company.
Very comprehensive article, @Brian!
Thanks for sharing as we need to educate ourselves with the potential scope of this risk.
“Holden said it appears the intruders laid the groundwork for the VPCI using Emotet, a powerful malware tool typically disseminated via spam”.
Same or similar malware at an IP address used by the mail server for Complete Technology Solutions (CTS)–which the article of December 7 is based on–to this day some “good” entity has mal-communication with the IP address.
Unfortunately my original message didn’t get posted.
Can you clarify what you mean by “password”?
Most OS and systems usually do not store the actual password on a system, only the hash of the password. For example on Linux systems in the shadow file you store the algorithm, hash and the salt value.
Getting the original password from a hash can be a very lengthy process if the original password was complex.
Obviously once a network is compromised key loggers and other malware can be installed and obtain keystrokes from users. For example it could obtain the last pass master password as the user enters it.
And sadly 14 months is a long time for an intruder to have access to your network. I think I recall seeing that often an intruder has access to a network for 3-6 months before anyone notices it.
Thanks
Almost all of those listed above were probably stored in browsers, where they can be easily read
Yeah,agree. Thank goodness there are some good browsers available (Mozilla example) you have to get into the settings…..can’t use them out of the box. Tweak and Tweak some more. If you don’t know how, ask someone or go read some Youtube tutorials. I came online back in the Windows XP days, boy have I learned a lot since then- almost like a college education! I like to stop in now and then to read Brian’s news it’s always interesting.
Is Google Chrome one of those “easy targets”?
Vilo
https://www.literatlik.sk
And If the Blockchain BTC transactions are Public why ransome attackers not traced down then?
Encrypting Your Files will protect against Your Data been stolen but will not Protect against Ransomware and other Malware.
Valt.X Cyber Security (www.valtx.com) has developed Cyber Attack Disrupt and Resilience Solutions – Our users simply reboot their systems to Disrupt and Instantly recover from Ransomware, Malware or Hacker attacks.
That doesn’t make much sense to me. The vast majority of malware’s first step is to enable persistence by injecting itself into system startup processes, the registry, etc. By the time you know you’re infected, persistence has already long been established. Rebooting will have no effect or delay on the malware.
Dennis is probably doing some form of file system freezing, like Faronics Labs Deep Freeze that I use on public-use Mac and Windows PCs that I manage in a higher ed environment. Freezing the image on disk is relatively robust to typical malware if properly executed, since it’s not an expected protection level. If using the Windows protected write features, much less robust…
Deep Freeze!! That was the name I was trying to recall. Yeah, that technology is 20+ years old. I ran into it in the K-12 environment.
Mangle a machine all you want and on reboot, it looked as if you hadn’t touched it.
So it’s basically like a VM, but a true image. I guess rebooting is like rolling back a snapshot then. That’s really interesting!
Hold on just a moment, there. It appears Mr. Meharchand and Valt.X were found to be violating Canadian Securities legislation in October 2018 by dealing in Securities without a license. Their Facebook page hasn’t been updated since their December 2018 ‘CrowdBuy’ post, but here is what the Ontario Securities Commission had to say about their CrowdBuy: “The CrowdBuy program: The Respondents initiated the CrowdBuy program in early 2016, after the OSC issued a temporary order that, among other things, cease-traded securities in Valt.X Holdings and prohibited the Respondents from trading in securities generally. Under the CrowdBuy program, the Respondents solicited people through the Valt.X website and via email to purchase Valt.X software licenses at a discount, claiming that purchasers could earn guaranteed returns of 20-50% in the first year through resales of the licenses. Participants were also offered the opportunity to convert their CrowdBuy subscriptions into Valt.X Holdings common shares. The CrowdBuy program also contemplated that the Respondents would hire sales agents to resell the licenses on behalf of the CrowdBuy program purchasers.”
Be careful and do your research on anything someone suggests in a comment thread as a near magic solution.
Excellent article! This highlights how useful hardware keys such as Yubikey could be , but clearly even they would be just a part of the overall strategy to protect and secure.
Hardware keys are most useful for companies where lost hardware keys are a simple matter of re-issuing in person.
Only remedy, is to have hot and cold backups. Hot backups which backup at least once a day or more and our connected and accessible. Then you have cold backups which might be daily or weekly and are disconnected once the backup is complete.
It’s incredible that it’s 2020 and we are still in a position where a single user can click on the wrong thing and cause an entire company to get taken down. This is a fundamental computing architecture issue that won’t get solved by throwing a heap of point security solutions at the problem. And it’s stymied by the fact that workers have only ever known open computing environments where stuff just (mostly) works and there are few or no obstacles to them getting to the systems and data they need to do their jobs. Start locking things down and everyone complains, including the executives who have the power to undercut the security professional’s efforts.
Well, there are only so many ways…let (the employees) complain.It (systems) HAS to be locked down….same as using limited access on a home computer…not running administrator….
Good article. Note also that keys and important access data that persists across active/backup can result in non-obvious (new) avenues of future attack once there is restore from backup even with nullification of passwords. The original holes the attackers used are closed, but the attacker obtained information to allow access via another path.
I made an article 2 weeks before that. After a couple delays requested by Microsoft.
https(colon)//securityintheenterprise.blogspot.com/2019/11/microsoft-azuread-and-office365-not.html
For enterprise customers, we might get single-signon. Vector can be anything that points to Microsoftonline.com – Or web services that employees actually trusts. Like draw.io as an example.
Microsoft office store apps are getting something wrong. Microsoft says app needs one permission, but on first run, it needs consent for way more.
In Europe, aka GDPR land, we have the big issue that if an employee gives (as an example) draw.io access to his OneDrive, and he has personal data there, then it is a potential breach, as they would then be able to process personal data without a data processor agreement in place.
We have shutdown user consent. We had to to ensure we are not running into issues with GDPR. And I think Microsoft has an issue here. This is clearly not security by default.
Brian,
It’s getting more and more difficult to have any sympathy for companies that get hit by Emotet/Trickbot and its cohorts.
How hard is this?
Remove-WindowsFeature FS-SMB1 -Restart
There is no excuse for the EternalBlue vulnerability to be alive and well on _any_ network today. It’s been two freaking years or more!
The patches are available. Why are people not installing them and/or updating their images?
Removing SMBv1 on all Windows servers eliminates that attack vector effectively shutting down the lot.
*sigh*
Ransomware can spread over the Internet without specific targets. However, the nature of this file-encrypted malware means that cybercriminals can also choose their targets.
This positioning capability allows cybercriminals to catch up with those who can pay more ransoms and is more likely to pay more.
I read an interesting article on cyber attacks -https://www.loginradius.com/blog/2019/10/cybersecurity-attacks-business/ which talks about ransomware impact on businesses.
Thanks for sharing the info krebs.
I clicked on that login radius link above and the first think that popped up was that I had 2 new messages. Closed it right away
It is T-system, and not T-systems!!!
T-systems have nothing to do with T-systems. :facepalm:
T-system is about Hospitality!
What if the author were part of ransomware team?
What if Hold Security, the author’s consultant for this article, also was part of the ransomware team? It isn’t mysterious that he’s got logs of the attack… ?
{dramatic sound effects}
Maybe they both are hackers for the Ukraine!
80% of these infections are coming form email links and attachments, as well as social engineering techniques. And that is probably a conservative number.