At least 30,000 organizations across the United States — including a significant number of small businesses, towns, cities and local governments — have over the past few days been hacked by an unusually aggressive Chinese cyber espionage unit that’s focused on stealing email from victim organizations, multiple sources tell KrebsOnSecurity. The espionage group is exploiting four newly-discovered flaws in Microsoft Exchange Server email software, and has seeded hundreds of thousands of victim organizations worldwide with tools that give the attackers total, remote control over affected systems.
On March 2, Microsoft released emergency security updates to plug four security holes in Exchange Server versions 2013 through 2019 that hackers were actively using to siphon email communications from Internet-facing systems running Exchange.
Microsoft said the Exchange flaws are being targeted by a previously unidentified Chinese hacking crew it dubbed “Hafnium,” and said the group had been conducting targeted attacks on email systems used by a range of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs.
In the three days since then, security experts say the same Chinese cyber espionage group has dramatically stepped up attacks on any vulnerable, unpatched Exchange servers worldwide.
In each incident, the intruders have left behind a “web shell,” an easy-to-use, password-protected hacking tool that can be accessed over the Internet from any browser. The web shell gives the attackers administrative access to the victim’s computer servers.
Speaking on condition of anonymity, two cybersecurity experts who’ve briefed U.S. national security advisors on the attack told KrebsOnSecurity the Chinese hacking group thought to be responsible has seized control over “hundreds of thousands” of Microsoft Exchange Servers worldwide — with each victim system representing approximately one organization that uses Exchange to process email.
Microsoft’s initial advisory about the Exchange flaws credited Reston, Va. based Volexity for reporting the vulnerabilities. Volexity President Steven Adair said the company first saw attackers quietly exploiting the Exchange bugs on Jan. 6, 2021, a day when most of the world was glued to television coverage of the riot at the U.S. Capitol.
But Adair said that over the past few days the hacking group has shifted into high gear, moving quickly to scan the Internet for Exchange servers that weren’t yet protected by the security updates Microsoft released Tuesday.
“We’ve worked on dozens of cases so far where web shells were put on the victim system back on Feb. 28 [before Microsoft announced its patches], all the way up to today,” Adair said. “Even if you patched the same day Microsoft published its patches, there’s still a high chance there is a web shell on your server. The truth is, if you’re running Exchange and you haven’t patched this yet, there’s a very high chance that your organization is already compromised.”
Reached for comment, Microsoft said it is working closely with the U.S. Cybersecurity & Infrastructure Security Agency (CISA), other government agencies, and security companies, to ensure it is providing the best possible guidance and mitigation for its customers.
“The best protection is to apply updates as soon as possible across all impacted systems,” a Microsoft spokesperson said in a written statement. “We continue to help customers by providing additional investigation and mitigation guidance. Impacted customers should contact our support teams for additional help and resources.”
Meanwhile, CISA has issued an emergency directive ordering all federal civilian departments and agencies running vulnerable Microsoft Exchange servers to either update the software or disconnect the products from their networks.
Adair said he’s fielded dozens of calls today from state and local government agencies that have identified the backdoors in their Exchange servers and are pleading for help. The trouble is, patching the flaws only blocks the four different ways the hackers are using to get in. But it does nothing to undo the damage that may already have been done.
A tweet from Chris Krebs, former director of the Cybersecurity & Infrastructure Security Agency, responding to a tweet from White House National Security Advisor Jake Sullivan.
White House press secretary Jen Psaki told reporters today the vulnerabilities found in Microsoft’s widely used Exchange servers were “significant,” and “could have far-reaching impacts.”
“We’re concerned that there are a large number of victims,” Psaki said.
By all accounts, rooting out these intruders is going to require an unprecedented and urgent nationwide clean-up effort. Adair and others say they’re worried that the longer it takes for victims to remove the backdoors, the more likely it is that the intruders will follow up by installing additional backdoors, and perhaps broadening the attack to include other portions of the victim’s network infrastructure.
Security researchers have published several tools for detecting vulnerable servers. One of those tools, a script from Microsoft’s Kevin Beaumont, is available from Github.
KrebsOnSecurity has seen portions of a victim list compiled by running such a tool, and it is not a pretty picture. The backdoor web shell is verifiably present on the networks of thousands of U.S. organizations, including banks, credit unions, non-profits, telecommunications providers, public utilities and police, fire and rescue units.
“It’s police departments, hospitals, tons of city and state governments and credit unions,” said one source who’s working closely with federal officials on the matter. “Just about everyone who’s running self-hosted Outlook Web Access and wasn’t patched as of a few days ago got hit with a zero-day attack.”
Another government cybersecurity expert who participated in a recent call with multiple stakeholders impacted by this hacking spree worries the cleanup effort required is going to be Herculean.
“On the call, many questions were from school districts or local governments that all need help,” the source said, speaking on condition they were not identified by name. “If these numbers are in the tens of thousands, how does incident response get done? There are just not enough incident response teams out there to do that quickly.”
When it released patches for the four Exchange Server flaws on Tuesday, Microsoft emphasized that the vulnerability did not affect customers running its Exchange Online service (Microsoft’s cloud-hosted email for businesses). But sources say the vast majority of the organizations victimized so far are running some form of Internet-facing Microsoft Outlook Web Access (OWA) email systems in tandem with Exchange servers internally.
“It’s a question worth asking, what’s Microsoft’s recommendation going to be?,” the government cybersecurity expert said. “They’ll say ‘Patch, but it’s better to go to the cloud.’ But how are they securing their non-cloud products? Letting them wither on the vine.”
The government cybersecurity expert said this most recent round of attacks is uncharacteristic of the kinds of nation-state level hacking typically attributed to China, which tends to be fairly focused on compromising specific strategic targets.
“Its reckless,” the source said. “It seems out of character for Chinese state actors to be this indiscriminate.”
Microsoft has said the incursions by Hafnium on vulnerable Exchange servers are in no way connected to the separate SolarWinds-related attacks, in which a suspected Russian intelligence group installed backdoors in network management software used by more than 18,000 organizations.
“We continue to see no evidence that the actor behind SolarWinds discovered or exploited any vulnerability in Microsoft products and services,” the company said.
Nevertheless, the events of the past few days may well end up far eclipsing the damage done by the SolarWinds intruders.
This is a fast-moving story, and likely will be updated multiple times throughout the day. Stay tuned.
Update, 8:27 p.m. ET: Wired cybersecurity reporter Andy Greenberg has confirmed hearing the same number of victim numbers cited in this report: “It’s massive. Absolutely massive,” one former national security official with knowledge of the investigation told WIRED. “We’re talking thousands of servers compromised per hour, globally.” Read Greenberg’s account here.
Also, the first and former director of CISA, Chris Krebs (no relation) seems to be suggesting on Twitter that the victim numbers cited here are conservative (or just outdated already):
Update 8:49 p.m. ET: Included a link to one of the more recommended tools for finding systems vulnerable to this attack.
Update, 10:17 p.m. ET: Added mention from Reuters story, which said White House officials are concerned about “a large number of victims.”
Update, March 6, 10:56 a.m. ET: CISA’s Twitter account says the agency “is aware of widespread domestic and international exploitation of Microsoft Exchange Server vulnerabilities and urges scanning Exchange Server logs with Microsoft’s detection tool to help determine compromise.”
A tweet today from the U.S. Cybersecurity and Infrastructure Security Agency (CISA).
Further Reading:
Thanks to you and one other site, we ensured our safety the day this was announced. Confirmed this morning by tools we were not effected. Patching has gone from waiting a few days to see if it breaks things to, tying to patch before it’s even been announced lol. Thanks for the heads up.
US is filled with such reporters, attorneys, useless executives/CISOs who have no idea what Security Engineering is but with just bunch of non-sense certifications.
Send them to Russia and China to get trained.
It’s better to learn the hard way.
This is why I got out of the IR space. Too many b.s. artist certification hounds running around both on IR teams and in the companies that we tried to help out.
A completely saturated field with many players. The only real differentiator for companies in this space are the ones with data (e.g., Mandiant) or the ones with industry contacts who get the first call (e.g., PwC and VISA, or Cyber Insurance Provider and their preferred vendor).
Too many jabber jaw, b.s. artist CIO/CISO/whatevers who got to those roles by virtue of their charisma and not much of anything else.
Too many companies who see security as a cost center and who really do not give a sh*t until something bad happens. Even then, no real accountability for poor decision making.
I mean, GlobalPayments was a textbook example and the Italian is still over there running things despite the company being re-breached after we left in 2012. We had to rebuild their entire PCI enclave because everything they had in it was EOL and yet, the CIO walked around like some kind of tech wizard leader. Please.
Microsoft also released a patch for Exchange 2010.
Posted @ 4:07PM on a Friday afternoon. Thank you Brian.
-admins and security engineers everywhere
Great and timely reporting. Thank you Brian
I would like to say a lot about Micro$%#*>>>>>?!<
…note that volexity first reported this in January…
“Security researchers have published a tool on Microsoft’s Github code repository that lets anyone scan the Internet for Exchange servers that have been infected with the backdoor shell”
Is this tool generally available? Could you provide a link so we can scan our own servers?
Presumably the nmap scanner at the bottom of https://github.com/microsoft/CSS-Exchange/tree/main/Security
I am not certain nor an expert on this particular issue but I think this github repo might be it:
https://github.com/cert-lv/exchange_webshell_detection/
WARNING! Carry on reading or your soul will be taken, even if you read the word “warning”!
There once was a man named Duke Hunapon. He was lazy, and very uncaring. He always wore a jacket, no matter how warm it was outside. He had an older brother named Micheal who always bossed him around. One day, Micheal was killed, and it affected Duke greatly. He went insane and started murdering people. Soon, he got into a fight with someone and got killed. Now, he roams around as a tall skeleton with a red shirt, and the exact same hoodie that Duke wore. This skeleton was known as “Swapfell Papyrus”, and he will kill you if you don’t post this on 15 comment sections of any website before your bedtime. If you fail, and you wake up when he’s in your room, your death will be slow and very painful. A girl named Lily Lilupanin read this, and didn’t listen. She was raped and killed in her sleep. If you copy and paste this on 15 comment sections of any website before your bedtime, Swapfell Papyrus will make you feel safe.
Thank you for the added detail that you brought in this piece. I wonder why Exchange Online was not affected? Maybe MS had already applied the necessary patches or a more sophisticated firewall in that environment.
AFAIK the Exchange Online runs almost the same code as on-premises edition (sometimes the cloud is a few builds in advance, this can be detected from headers leaking build numbers). Neverthless, architecture in the cloud is incomparable to classic on-premises Exchange deployment, at least due to the need of authentication outside of the Exchange world (forced redirect to IdP), which effectively hides anonymously accessible parts of OWA website. Of course similar setup can be done in on-premises environment as well – personally I did this in lab with F5 APM as a preauthentication reverse proxy and ADFS as an IdP, no part of Exchange is then accessible without authentication, but no customer wanted this, because it is complex to deploy and more complicated to troubleshoot.
It’s certainly possible that the attackers chose not to target MS Online.
If they had, Microsoft would have been more likely to spot the attack sooner (better IDS).
Once the attack was discovered, if MS Online was vulnerable, you can assume it was patched.
It’s also possible the file systems for the MS Online offering are more read-only than your average system.
Best practices for cloud deployments these days involves spinning up read-only images and rotating them into place instead of modifying a live mutable computer. (It’s definitely possible to have stricter ACLs on NTFS than the defaults, it wouldn’t shock me if MS does.)
Nothing in your article about mitigating the alleged installed script(s). Yes, I have the patches installed on my Exchange server, but would love to hear about hunting down/removing the alleged scripts. And yes, I’ve read the MS articles on these four covered issues.
This is just classic MS all around I mean I don’t want to say if you’re running Outlook Web Access you deserve to have your email sever owned but… Just the title of that product should be a good hint it’s extraordinarily insecure and bound to get it eventually. When did everyone start assuming anything having to do with email was secure anyway? That was a real bad assumption, probably some time around when we put the bean counters and suits in charge and let the plebs on the internet and let them all run wild so I also don’t want to say we had it coming but…
Sorry doubt my last comment will pass moderation but if it does I really didn’t mean to reply directly to you just a general comment on the state of things in re information technology.
In penance I will answer your question, back everything up and get a new email server preferably not Exchange but if it has to be don’t let anyone connect to the server from outside your private network and make sure that’s secure too but might as well switch to cloud exchange anyway since this will very possibly happen to self hosted exchange again and ms will make you at some point too.
The problem with mitigating attacks is that the answer will vary. Once you’ve been attacked, you have to figure out what the attacker has done, and that can vary per instance. There’s no “one answer fits all”, unless you’re actually hit by a one-trick-pony-botnet and not someone’s knock-off of that botnet which has additional tricks.
Thus, the advice is “get a team to investigate”.
In the interim, take the computers off the network, and if at all possible, take all other computers they could reach (or whose credentials could be manipulated based on the mailboxes they manage) off the network.
Especially if there’s any chance that people w/ admin credentials could have had those credentials compromised by abusing their mailboxes.
And yes, the downtime suggestion here is pretty significant.
I told Klint about this vuln back in December 2020 when it was still a Zero Day
Hello lmao
How about linking the depicted Tweet and the Github repo with the scanning tool?
Better scoop — USG sent Microsoft a sample of Hanium, which is how they identified more victims.
As was said, we now say: Open biden
Come on USA what are you gonna do?
Get off social media and get your sh*t together. Unless you want to continue getting hacked left and right hahah
Brian and Chris and everyone else whose shoulders we all stand on, thank you.
Why is it MS can get away calling this a ZDE when it’s been around for two months? (I beg the question of why they can’t write good software, because that’s been around since before Win3.0.)
E
0-day = no current patch for it. You’re right though, it’s been 0-day for a looong time!
Ironic because I have been squabbling with inept Microsoft support staff this week about them twice black-listing my SMTP server for no discernible reason such that we cannot send passwords to users with hotmail, outlook, msn, and related email address domains. It’s not a new IP, we do not spam, have SPF, DKIM, etc. Keep up the good work Brownie. #$@#$%$@#.
Do you think all of this amped up Russia/China hacking is part of a larger campaign to lay groundwork for a potential war many years away? Every expert/book on the topic I’ve read seems to agree that a war between China and/or Russia would almost certainly have a massive cyber component, which would require access to any number of different agencies/companies to be effective.
Just curious your thoughts, if you even read the comments.
Nah, they do it because they can. Privately, Russian/Chinese hackers consider Americans simpletons who deserve to be made fun of and hacked for their arrogance, “they are rich anyway, it wont make much harm”
Correction, please:
“the riot at the U.S. Capitol.”
Is actually:
‘the failed insurrection at the U.S. Capitol.’
Thank you.
LOL, an insurrection, lol. Far from it. It was a small riot compared to this past 10 months in primarily dem cities. Looting, burning, riots, over 100 dead, 500 billion property damage, and I’m no longer a dumbsh#tdem or into politics because I woke up. I have seen videos of people caught putting on maga gear to make it “look like” it was a P45 act when in fact it was more dems in disguise, blm and antifa all of which are leftists. And lets not forget the dem that actually did major damage and death in DC she now runs the head of donations intake for blm. She’s a convicted terrorist. Wake up.
Correction to the correction. It was a “mostly peaceful protest,” just like Portland, Seattle, Chicago, etc.
It was also a “Summer Of Love”….The Love of breaking things….
Correction please:
‘the failed insurrection at the U.S. Capitol.’
Is actually:
“the mostly peaceful protest at the U.S. Capitol.”
What I love about these equivalence arguments is the implication that the safety and security of Congress is no more important than the safety and security of a liquor store in Oakland.
How do you know its the chinese doing this?
“The numbers I’ve heard dwarf what’s reported here & by my brother from another mother (@briankrebs).”
Touché, sir!
“Some subjects are so serious that one can only joke about them.”
— Niels Bohr
(As quoted in “The Genius of Science: A Portrait
Gallery,” 2000, by Abraham Pais, p. 24.)
Just like a lot of global intellectual property was copied and made “Shenzhen genuine” driving economic and consumer revolutions for decades; have we entered the era of the Shenzhen 0-day?
– I literally hear “oops” and think this every single time CISA warn of a China borne oh-day emergency.
If that was a hack and my hand fits my face then how do you spell the question reference word in object reference all the while being honest with ‘yourself?’ Wrong again dick-nose. Sell-out to the tin-man today! Now recruiting chin-nut-hackers who know how to push a mouse ‘clit’ one time at a 1000. Snark!
Get the heck in here. RHRBIII->LSBT$$
Is there any way to know if domains hosted by Microsoft’s mail.office365.com were affected?
“When it released patches for the four Exchange Server flaws on Tuesday, Microsoft emphasized that the vulnerability did not affect customers running its Exchange Online service (Microsoft’s cloud-hosted email for businesses).”
Office 365 was not affected.
Why would you run your own email server in 21st century? If you are doing that because it is “cheaper”, you will never put the effort to maintain and secure it properly.
Are/Were Exchange Servers vulnerable when OWA is protected by two-factor authentication. Did someone analyze this?
Thanks!
In this attack, two-factor is largely irrelevant. This is a “hey is your Exchange server vulnerable, okay give me all your email and direct access to your internal network” attack. It’s very bad.
Yes they are still vulnerable since the attack did not require authentication.
As long as there are parts in OWA (and possibly ECP) websites accessible anonymously, compromising unpatched server is possible without authentication. 2FA itself does not change this.
On the other hand, if 2FA is part of more robust deployment with preauthenticating reverse proxy and thus no one can reach any single part of Exchange web without authenticating first at the proxy level, Exchange exploitation without authentication is then infeasible.
Used to be that it was more important to wait and test patches, but nowadays with the solarwinds and this hack, your screwed if you patch early (solarwinds) and screwed if you wait (owa hack)…We all need to start taking security more seriously…especially with this garbage administration.
Don’t you mean “these garbage administrations”?
I checked the owner of the web shell left behind by this hack. The file owner is system. Why in the world does Microsoft run Exchange IIS web services using the system account not a least-privileged user account? I added IP Address and Domains Restrictions to IIS to block ECP and OWA from the internet until this hack has been mitigated. When I tested ECP access with a web browser, IIS ignores the IP restrictions and allows the DLL servicing ECP to service the web request. The browser clearly shows a 302 redirect to OWA for authentication instead of a 404 error. OWA access returns a 404 error but not ECP. More bugs. This is just nuts.
Agreed. It’s crazy.
What about certificate based authentication in Exchange ? Is it vulnerable? Without proper certificate client got always HTTP ERROR 403.