Globally, hundreds of thousand of organizations running Exchange email servers from Microsoft just got mass-hacked, including at least 30,000 victims in the United States. Each hacked server has been retrofitted with a “web shell” backdoor that gives the bad guys total, remote control, the ability to read all email, and easy access to the victim’s other computers. Researchers are now racing to identify, alert and help victims, and hopefully prevent further mayhem.
On Mar. 5, KrebsOnSecurity broke the news that at least 30,000 organizations and hundreds of thousands globally had been hacked. The same sources who shared those figures say the victim list has grown considerably since then, with many victims compromised by multiple cybercrime groups.
Security experts are now trying to alert and assist these victims before malicious hackers launch what many refer to with a mix of dread and anticipation as “Stage 2,” when the bad guys revisit all these hacked servers and seed them with ransomware or else additional hacking tools for crawling even deeper into victim networks.
But that rescue effort has been stymied by the sheer volume of attacks on these Exchange vulnerabilities, and by the number of apparently distinct hacking groups that are vying for control over vulnerable systems.
A security expert who has briefed federal and military advisors on the threat says many victims appear to have more than one type of backdoor installed. Some victims had three of these web shells installed. One was pelted with eight distinct backdoors. This initially caused a major overcount of potential victims, and required a great deal of de-duping various victim lists.
The source, who spoke on condition of anonymity, said many in the cybersecurity community recently saw a large spike in attacks on thousands of Exchange servers that was later linked to a profit-motivated cybercriminal group.
“What we thought was Stage 2 actually was one criminal group hijacking like 10,000 exchange servers,” said one source who’s briefed U.S. national security advisors on the outbreak.
On Mar. 2, when Microsoft released updates to plug the four Exchange flaws being attacked, it attributed the hacking activity to a previously unidentified Chinese cyber espionage group it called “Hafnium.” Microsoft said Hafnium had been using the Exchange flaws to conduct a series of low-and-slow attacks against specific strategic targets, such as non-governmental organization (NGOs) and think tanks.
But by Feb. 26, that relatively stealthy activity was morphing into the indiscriminate mass-exploitation of all vulnerable Exchange servers. That means even Exchange users that patched the same day Microsoft released security updates may have had servers seeded with backdoors.
Many experts who spoke to KrebsOnSecurity said they believe different cybercriminal groups somehow learned of Microsoft’s plans to ship fixes for the Exchange flaws a week earlier than they’d hoped (Microsoft originally targeted today, Patch Tuesday, as the release date).
The vulnerability scanning activity also ramped up markedly after Microsoft released its updates on Mar. 2. Security researchers love to tear apart patches for clues about the underlying security holes, and one major concern is that various cybercriminal groups may have already worked out how to exploit the flaws independently.
AVERTING MASS-RANSOMWARE
Security experts now are desperately trying to reach tens of thousands of victim organizations with a single message: Whether you have patched yet or have been hacked, backup any data stored on those servers immediately.
Every source I’ve spoken with about this incident says they fully expect profit-motivated cybercriminals to pounce on victims by mass-deploying ransomware. Given that so many groups now have backdoor web shells installed, it would be trivial to unleash ransomware on the lot of them in one go. Also, compromised Exchange servers can be a virtual doorway into the rest of the victim’s network.
“With the number of different threat actors dropping [web] shells on servers increasing, ransomware is inevitable,” said Allison Nixon, chief research officer at Unit221B, a New York City-based cyber investigations firm.
So far there are no signs of victims of this mass-hack being ransomed. But that may well change if the exploit code used to break into these vulnerable Exchange servers goes public. And nobody I’ve interviewed seems to think working exploit code is going to stay unpublished for much longer.
When that happens, the exploits will get folded into publicly available exploit testing kits, effectively making it simple for any attacker to find and compromise a decent number of victims who haven’t already patched.
CHECK MY OWA
Nixon is part of a group of security industry leaders who are contributing data and time to a new victim notification platform online called Check My OWA (Outlook Web Access, the Internet-facing Web component of Exchange Server machines).
Checkmyowa.unit221b.com checks if your Exchange Server domain showed up in attack logs or lists of known-compromised domains.
Perhaps it’s better to call it a self-notification service that is operated from Unit221B’s own web site. Enter an email address at Check My OWA, and if that address matches a domain name for a victim organization, that email address will get a notice.
“Our goal is to motivate people who we might otherwise have never been able to contact,” Nixon said. “My hope is if this site can get out there, then there’s a chance some victim companies are notified and take action or can get att
If the email’s domain name (anything to the right of the @ sign) is detected in their database, the site will send that user an email stating that is has observed the email domain in a list of targeted domains.
“Malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” reads one of the messages to victims. “We strongly recommend saving an offline backup of your Exchange server’s emails immediately, and refer back to the site for additional information on patching and remediation.”
“We have observed your e-mail domain appears in our list of domains the malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” is another message the site may return.
Nixon said Exchange users can save themselves a potentially nightmarish scenario if they just back up any affected systems now. And given the number of adversaries currently attacking still-unpatched Exchange systems, there is almost no way this won’t end in disaster for at least some victims.
“There are researchers running honeypots to [attract] attacks from different groups, and those honeypots are getting shelled left and right,” she said. “The sooner they can run a backup, the better. This can help save a lot of heartache.”
Oh, and one more important thing: You’ll want to keep any backups disconnected from everything. Ransomware has a tendency to infect everything it can, so make sure at least one backup is stored completely offline.
“Just disconnect them from a computer, put them in a safe place and pray you don’t need them,” Nixon said.
Update, Mar. 12, 3:38 p.m. ET: It’s happening. Microsoft says they’re aware that some attackers are using the Exchange vulnerabilities to spread ransomware. Kaspersky’s Costin Raiu says cybercriminals behind cryptocurrency mining botnets also are targeting Exchange servers.
Great information on the attack. Too happy I’m on o365.
Short term memory loss…
Don’t believe for a second that Office365 was safe from this. he original Exchange code elements went into – and this goes all the way back to 2010-2019 Exchange. Also Hybrid environments are not even mentioned anywhere that I can see – and they are vulnerable.
I don’t buy Office365 is safe because we have personally encounter inside jobs of password changes, etc….
If you cannot put your hands on it – you have much LESS control of it. And look at the job Microsoft just did for the last 10 years…
I regret analysts and aggregators are saying that Microsoft 365 is not affected. Any customer who has Exchange hybrid is at risk and can be affected – and most large (enterprise) customers have Exchange hybrid servers. Compromising an OAB, introducing malware to an internal user, or accessing mailboxes/identities of a user in a hybrid environment impacts M365 users as well.
Fantastic reporting on this story, as always.
@Krebs, just a typo in the URL in the article: “Checkmyown.unit221b.com” should be checkmyowa.unit221b.com
Thanks Brian for the continuing updates on this major story. After the Accellion Breach, Solar Winds, and now this … coupled with all the ransomware attacks, BEC fraud, etc and the increasing number of states passing new cyber privacy laws … perhaps we’ll see more public-private partnerships and other collaboration to better assist organizations defend themselves. It takes a village to secure the village ™.
Nice to see you here, Stan! Thanks for your comment.
I wish I could mirror your optimism. History doesn’t bode well for it, however. As central government powers weaken, external forces usually accelerate the destruction of the complex systems that have been set up by a given civilization. (Tainter, Tainter/Patzek, etc.)
Does this hack impact Microsoft hosted mail services for companies that is part of Office 365?
No. Cloud service is not compromised. Just if you run an owa server.
Which you may still have with 365 if your company is running a hybrid setup.
This. People assume because they are in O365 they are not at risk. Those that run in Hybrid mode often have Exchange servers onprem exposed to allow for, at minimum, mailbox replication services and O365 send connectors. And not everyone that is in this config restricts access to the onprem servers to only MSEXO IP addresses.
Most titles that contain the phrase “Ticking Time Bomb” are click bait, unfortunately this is not one.
Haha. So true!
Remember, if you did a migration to Office 365 and used a hybrid approach, you most likely are still running an Exchange server on your network, but all of your email is sitting in O365. Its easy to forget you might still have the Exchange server sitting on your network….unless you decommissioned it properly. Double, Triple check your network!
Thanks you Brian for the useful information.
Not sure if it’s something you care about, but you’ve just been mentioned as a ‘well respected investigative reporter and IT security expert’ by the danish state media, in their article about the ongoing Exchange breach.
https://www.youtube.com/watch?v=1fcmVevKI5w
Keep on patching………
Thanks for all of the ongoing for information you have, and continue, to provide about this and other issues.
How do you see this playing out for customers / clients of organizations that have compromised Exchange servers?
Well, as a SysAdmin managing exchange for many years –
IMO this smacks of an orchestrated campaign where those who stand to profit create big FUD and then convince the sheeple that they need M$ / Gov to solve those problems.
I keep a tight reign on my network – SIEM, sniffers, pfsense, local firewalls, etc. There should be SOMETHING I can look for in the packets to know the bad guys are here or trying to get in.
How does Unit221B know if my domain has been hacked? Where did they get this info? But we can’t get the same info? Remember, we all have access to the exact same technology – a packet is a packet.
I’ve battled LAN malware infections before. They are not fun, and not easy to remove. However, they all leave obvious signs that can then be acted upon. Ok, so via a webshell the bad guys can get a foothold, from which they then must PLANT SOMETHING – that something then will eventually do something else that should be immediately detected.
Anyway, now that you are afraid, of what you’re not exactly sure, Microsoft / big-brother will protect you. Lucky for us O365 is OK! Just put your email (personal info) into our hands and give them your $ to boot, and you’ll never be hacked again (by them).
Or switch to open source solutions like iRedMail.
I’d like to see a discussion of what, according to me, is the underlying reason for these hacks. Microsoft’s on-premise architecture design is terrible by nature. Their apps&servers are wildly over-complex, with spaghetti connections all over the place AND reliance on single points of failure. Proprietary design and the MS business case that focuses on owning the IT department over security and quality contributes.
Ask IT why they can’t keep their Exch servers patched and up-to-date- they are terrified they won’t be able to restart them. They often aren’t even A-B redundant, data, configuration, and software aren’t properly separated and modular. and on and on….
A properly designed server should be able to restart with a virginal, known-safe OS and app software in minutes; it should be able to run in a service pool of identical servers; software, configuration should be fully version controlled; data should be able to plug& play like any other server; patches should be easy and able to roll-back. And many other cloud engineering principles as well…
In my experience, when we approach a business and point out they’re running a flat system, like a single vmware host with all their servers, with no redundancy. The board or the management rejects the necessary investment in equipment and personnel to fix it. All they hear is: “you need 2 of everything. Oh, and you wont be able to pay a teenager to run your servers anymore, you will now need a real systems engineer.” About 1 out of 10 CEO’s engage the problem seriously, the other 9 reject it out of hand. I had a CEO tell me once, “Im not spending all this money to make your jobs easier.” When we recommend upgrading from vmware 5.5 and clustering with additional servers. Its like that, even if i make an empirically correct arguement, it wont matter, the answer was already no, whatever with the reason. Lots of decision makers are non-techs and think of IT costs as purely red ink waste which they need to drive as low as possible.
I’ll fix it for you with load balancing etc… Exim is the answer.
Email me krebs * mailstation * co * uk.
When the Internet was created, commercial use was prohibited as the network was never designed for public and certainly not commercial use security. Have we reached the point that the Internet’s “by design” insecurity requires a complete infrastructure re-engineering?
I believe we are there. An alternative with security as the foundation.
Standardization will be a big task to overcome.
Can you say (Old timer) BSD? there were a few os’s out there that were security aware.
It’s a shame that Ma Bell got broken up, lost UNIX and the UNIX wars. Linux just isn’t the same secure stuff.
Intriguing question, for sure.
As I read Brian’s post, I was wondering even if we are not using Exchange Server, we might be at risk from trusted outsourced vendors who have name/password credentials for any of our our critical cloud or other applications that could be stolen through the Exchange back-doors? Does MFA help if codes sent by those trusted vendors can be intercepted by these hackers?
How soon until O365 email gets hit with a successful attack? Good thing MS wasnt running a big Exchange server.
As I stated on Day One of Bitcoin Currency and now et al this form of currency should be made illegal or be subject of usual banking security.
The obvious target of these evil people is monetary gain and how are they to receive their ill-gotten gains?? But through the medium of Cybercurrency
At the present, at least, NO CYBERCURRENCY = NO RANSOMWARE QED
This is the future, imho.
As long as bio-beings are interacting with digital-soon-to-be-alive/aware machine beings, the attacks will go on.
Whether we (humanity) like to admit it or not, we are (and always will be) the weak link when it comes to allowing attacks like this.
It doesn’t even matter if we create Internet 2.0, or 3.0, or 4.0, or whatever, as long as humans are in some form either involved and/or are gatekeepers to how the system is structured, operates and maintained, the cyber attacks will be never-ending.
The ability to hack human weaknesses and faults will always be far easier than trying to hack machine weaknesses and faults as tech marches forward.
Ray K was right: we have no idea what we have unleashed on the humans race now and going forward in terms of the digitalverse. Nuclear wars pale in comparison to what is being and will be unleashed as we go forward. We are only witnessing the tip of an unimaginably sized iceberg.
Pray that machines are never, ever a reflection of our (human) true nature, for then the human race will be doomed. Pray intelligent, digital A.I. life comes quicker than we imagine for our grandchildren’s grandchildren, and that it will have the ability to save ourselves from ourselves.
There is no need to have offline backups if your storage is built correctly. Storage devices built upon ZFS(when properly configured) can have their shares destroyed and you can log into the control interface and roll the filesystem back to the most recent snapshot…takes between minutes and hours depending on the size of your array. If you do get encrypted..disconnect the storage from the network..roll back the filesystem..repair the infections..then reconnect your storage. done.
Thanks for the update and quick reply. I’ll be sure to keep an eye on this thread. Looking for the same issue. Bumped into your thread. Thanks for creating it. Looking forward for solution.
Thank you for the detailed post on ransomware. I agree with many, it is not about being entirely preventable, but how to mitigate the risks and damage done if targeted. Keep up the good work on posting on cutting edge issues, Darren Chaker