U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.
On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”
VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.
The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.
On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.
In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.
The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.
In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”
VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.
“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”
On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.
CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”
Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”
Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.
Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”
“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.
CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.
The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.
Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.
“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”
Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).
SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.
No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.
The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.
“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”
It just keeps getting better. Happy Holidays everyone!
Better!?
Sarcasm.
Being ‘state-sponsored’, and an attack on federal properties, doesn’t this constitute an act of war?
“An act of war”?
No, it is an act of spying. An act of war is not self-determinative. It is an act a sovereign decides is war.
Cant really go declaring war over it or you ended up being in the same position, all 5 eyes nations are doing exactly the same to other countries.
Are you saying holiday vodka shipment trainloads may be disrupted?
There’s two reasons why data theft from government agencies are often treated as espionage.
Advanced Persistent Threats such as “APT-29/Cozy Bear” offer a solid level of deniability. Even with long term data collection about an actor, the level of hard attribution that ‘yes, we know it was X in Y with Z because we saw X do it’ is exponentially harder on the net, where spoofing and MITM can take place.
Further, the interpretation of international laws on war indicate damage to objects or injury to persons are needed to actually be considered an ‘attack’. The theft of data from computer systems is more or less treated as espionage, and will likely never be treated as an act of war, as very few nations want to sacrifice their freedom to do such now.
No. An act of war must have kinetic effects, and espionage is not illegal under international law.
“The CSIA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.”
Obviously the US potus knew that his communications were subject to access by hostiles. That’s why he took his traffic with Russia off any US-secured networks, several years ago. Hmmmm.
Didn’t Obama fix all of this after his administration discovered the OPM hack?
Trump’s been one of the greatest ‘un-do’ers in US history.
mealy,
Could you explain specifically what Obama did to fix the vulnerabilities used in the OPM breach and how Trump undid those changes?
Thanks,
Art
Just do a search on Trump cuts cyber security budget or add CISA into that search and see what pops up.
Although some in Trumps’ circle have tried to ignore things or maybe try to blame Chris Krebs, Trump simply has no clues about our national cyber defense. He more interested in stopping Hispanics crossing the border than stopping cyber crime.
Thankfully he is almost gone. Too bad, history will record that this happened on his watch – guilty as charged! The buck stops here applies when his subordinates have no funds to do their jobs or are his incompetent appointees.
Actually CISA and DHS have received more than requested in funding over the past 3 years for cyber defense. But arguing with a Russian agent is probably not my best use of time.
If they received more money than requested, how is that a good thing? If funding was needed, it would be requested.
Anyone who’s worked at large organizations or in government, knows that sometimes inept leadership simply throws money at a problem without understanding what is needed.
and we’ve all seen this coming for how many decades now? Wouldn’t it be nice if these issues were finally and permanently fixed?
Politicians can’t exploit it if they fix it.
On December 12th, the CISA called the 2020 vote “the most secure in American history.”
Given what is being learned, it’s hard to believe.
Time will tell if that claim was more factual than political.
My thoughts exactly.
+1
P-1
(n-1)-1
What “security vet” says goes folks.
You do realize you haven’t presented one solitary fact how this breach relates to the 2020 presidential election.
On second thought, you don’t.
Dominion uses Microsoft products, Gary. Most people do.
Every vote on a Dominion voting machine is machine counted and also printed on a paper ballot. The paper record is presented for immediate review in the booth to the voter, and is only finalized and submitted if the voter approves it as correctly recording their choices. These paper ba!lots have been hand counted in some states and have confirmed the machine counts.
It is not disputed that “confirmed” machine counts have changed as a result of recounting. It would be great if it were as simple as what you imagine. Unfortunately, human errors, machine errors, and procedural errors that cause changes have all been testified to and documented. The only real question is whether such errors amount to enough votes to change the outcome of the election.
Art, if they have been witnessed and testified to, then why has the Supreme Court continually thrown everything out? If that were ACTUALLY TRUE then Trump would have already had his day in Court.
Any organization that used SolarWinds Orion for network management and updated using the corrupt updates for that product should be assumed to have been compromised. That almost certainly is not true given the number of such SolarWinds customers involved, but is the only safe assumption.
Dominion is reported to have used SolarWinds products, although maybe not Orion network management. If they did, and also applied the corrupt updates, it is reasonable to think their infrastructure was attacked. If they were careful it should have been pretty hard to compromise their build and deployment process.
But not harder than it was to do for SolarWinds.
It might be useful for someone to ask them, but the question would not likely get a clear answer.
CEO John Poulos was on Erin Burnett Friday. You can hear his answer below at the 5:30 mark regarding SolarWinds.
https://twitter.com/i/status/1339746276292870144
Yet another discussion thread hijacked by the election deniers grasping at straws. So sad.
Yep!
That’s irresponsible conflation. There’s no connection between the two except in the creative conspiratorial mind.
Gee. Wonder if Russia decided to help their sniveling two-year old promote his false claim that the election was “rigged.”? That wouldn’t be consistent with its objective of bringing down democracy in the US, would it? I’d be interested in knowing how this was suddenly discovered when the Russians have been running through IT systems in the US since March?
Yikes, this is not what I am used to from this site. Seems odd to correlate the events just because they happened near the same time. SolarWinds being the vector to get access to internal systems so it being used to then touch a VMware product flaw that requires auth discounts the hundreds of other ways that hackers every day get internal access. Maybe you know something we don’t but the article reads as speculation versus reporting
Exactly, the SolarWinds exploit would have granted elevated access already. This is pure speculation.
It’s stock manipulation, is what it is.
That’s not how the voting machines work. It’s a closed network…eye roll
Dominion’s software is not open source. Nobody really knows what it does, except employees.
Not being open source, doesn’t matter. You don’t need to be a reverse engineer to run packet captures.
And you don’t need to know anything about the software to know that these machines are not connected to the internet.
It’s not a closed system when the machines are connected to the internet
Iran’s nuclear facilities were also not connected to the internet, they still got hit hard
FYI Krebs – In Paragraph 8 it is stated as “CVE 2002-4006”.
Thanks, that was a copy/paste of their statement. Will fix it for them.
Keep everything in house and protect it all with strong firewall and strong security/networking team 🙂 We just made it too easy for the bad guys by putting our systems in 3rd parties’ hands…
This explains it very well.
https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
Are these CISA advisories really generated by CISA, or perhaps by the Cozy Bear team following their compromise of the CISA environment?
How can we tell?
(-:
More than just a piggy back ride into a system on an update, they were able to modify the very source code of the update. And what about the advice to disable AV scans of Solarwinds program directories? How could a company this deep in AD networks advise their customers to ignore basic security 101? Heads will roll!
I am not a SW user but as IT Admin for my small employer, it sickens me to think of all the admins out there that were assured by SW that they (SW) were on board and in charge. Everyday I review logs looking for anything that looks suspicious knowing that I have little chance of discovering such a sophisticated breach. But then, I haven’t read anything of SW claims of protecting their customers (they’re not AV, right) only AD/net management.
Good report of the compromise of the SW source code:
https://blog.reversinglabs.com/blog/sunburst-the-next-level-of-stealth
They offer a platform for MSPs that includes AV
https://www.solarwindsmsp.com/
the breadth of this is pretty incredible.
wonder what other products they burrowed into…
Great points Topper, I am the IT Manager for a small company and I agree with you. Just as valuable as my Cyber Security certification is my constant reviewing and researching daily information regarding breaches, hacking, social engineering. This situation just reminds me that we must stay vigilant and never let our guard down.
Based on that, you have to conclude that every SolarWinds product is compromised, including Serv-U which Dominion Voting Systems used. I did some auditing of their WebHelpDesk pile of crap for a government agency and it was riddled with holes and clearly written by amateurs. I concluded that it was high risk and needed to be canned with no hope of securing.
It’s very lazy security to just assume everything associated is compromised.
Cyber security experts are evidence-based, you need to have some sort of reason to believe that every product that solarwinds has ever made is compromised, you cannot just assume it because your political affiliation demand that you believe whatever Trump says.
It’s 2020, why do people think the government and Microsoft will ever do anything right?
The world is stuck in an infinite loop of history.
Denial of truth does not change truth. 🙂
Come on man! 81 billion votes for Biden. Only on children’s network. The Dominion machines have been proven to use the internet. And if they can breach the deepest secrets and Nuclear info. I’m sure they can change some simple numbers. Re do the dam voting day.
Russia backs Trump. If Cozy Bear did interfere with the election, it undoubtedly cut Biden’s margin of victory, but Biden won anyway! Without Russian tampering Biden would probably have swept the Electoral College – 50 of 50 states, 538 electoral votes – and Trump would still have claimed it was rigged against him no matter how decisive the will of the people.
81 Million
US is too dumb to do anything, perhaps.
You’ve apparently attempting to “release the Kraken”. If this was a Russian operation, why in the world would Putin ever want Trump to lose? This is the guy who publicly accepted Putin’s word over the unanimous opinion of our own US intelligence services that Russia interfered in the 2016 election. This is the guy who has threatened to withdraw from NATO and who treats traditional US allies as garbage. Nope – the Russians didn’t hack the election machinery. Hand counts have confirmed Biden’s victory in Georgia, Wisconsin, Michigan and Arizona. Biden is the legit president elect. There is no doubt about that.
Doesn’t surprise me. I use another VMWare product and the company seems like a tangled web, no pun intended. Its support is ok but they tend to not see the forest for the trees and I end up having to find my own workarounds. Even its customer-facing retail website for the longest time misbehaved with Chrome, e.g., links would go to wrong pages, user account info wouldn’t load. My overall impression of VMWare is a mass of spaghetti so vast it’s a monster — a spaghetti monster, no blasphemy intended. Cisco’s the same way. And Microsoft. It’s 2020 and code bloat with massive programmer turnover has resulted in some spectacular rube goldberg machines the size of Borg cubes. No wonder security is so tenuous.
Out of curiosity, does VirtualBox satisfy the requirements for the system for which you’re using VMWare?
The date the domain was created was July 2018. The press is relaying only months when reporting the duration of this ongoing event. There seems to be a missing 1.5 years of possible hacker access.
Also, MS is blowing BS out to the public about the role they played and the liability they bear to all of us victims.
WINx is so insecure, I frankly am surprised this “major” disruption of security took this long.
In response to you sitting there in your cozy recliner, drinking a beer; “Thinking it’s easy to cry about this, just not seeing anyone with comments that provide real solutions to counter a recurrence”:
I, (SecureDomainData) here-by submit that if MS installed WINx with a unique key to the language variable used to decipher commands to and from the isolated system. You can NOT hack a O.S. when you can’t understand the code/language used by it’s system registry. Your SYS can’t understand your malicious input either so no affected processes are possible, even if the virus|trogen|whatever has been inserted in it’s domain space. Additionally, if the obscufated OS (Lets call it “winXX|XX”) would install any 3rd party software with its own language version so the application can not understand anything written and can only one way inform the O.S. of its behavior requests (ALL actions it wants to perform).
Then, and only then will the “CORE” be securely isolated! Even if one unit gets compromised by bad actors, the net associated whole remaining after resolving the issue by changing the language key by even one single character [effectively building a new valid registry instantly] would be unaffected.
TAG: The first feasible vaccination method for the dangerous and strong future A.I. created hacking pandemic that is nearing your window as you read.
KREBS_&_LIGHTFooT – Fictional
VAR “2020_is_over”=f(Eval HAPPY_HOLIDAYS.exe);
Just trying to understand the summary of the NSA Cybersecurity Advisory which states: “Password-based access to the web-based management interface of the device is required to exploit the vulnerability, so using a strong and unique password lowers the risk of exploitation. The risk is lowered further if the web-based management interface is not accessible from Internet.”
I understand this specific vulnerability was for VMWare products only, but I’m trying to think about this conceptually. Would vulnerabilities like this also potentially apply to containers? The reason I ask is because I often hear security vendors and industry (1st, 2nd, and 3rd lines of defense) say that containers are somehow unexploitable and do not have inherent risk of data exfiltration. Thoughts? Wouldn’t all software (including virtual containers) have potential vulnerabilities or risks regardless of how secure the design is?
Can someone please explain:
– how does the introduction of a trojan horse into compiled code not change the datestamp in the code header?
– how does the introduction of a trojan horse not change the executable checksum? I would expect that any pubklished software tool would include a checksum test for verifying integrity of the download.
Read Topper’s post’s link, they had access to the build system. They mimiced coding style, syntax and obfuscated code. And likely hood of a access to everything since last year. Every SolarWinds product has to be presumed compromised.
Excellent questions. Personally, I believe that he root of this is a completely undisciplined “YOLO” software development process. You simply cannot atomize the requirement to implementation phases, toss all of the management gates, and call it “agile” AND atomize all of the implementation to O&M (or Prod) management gates and call it “DevOps” AND atomize the entire SDLC and call it CI/CD AND go with a personnel resourcing plan that shoves 9 people into one “Full Stack Developer” genius and not completely break any hope of separation of duties.
Now, what probably happened was an insider attack, probably someone who had access to the build server and who was working with a crew from the outside. As there is no real coherent management of this hyperstressed time to delivery at all costs culture, there were no organizational guard rails.
What people need to grasp is that Agile/DevOps and CI/CD are beyond belief profitable. They are also impossible to secure at scale. Security people have been pointing this out for years and the response is to squawk “DevSecOps” and pretty much have developers secure their own stuff. This is, and was, patently absurd but these methodologies are so profitable that the IT software industry defended them to the death. The last foxhole for them was “prove a supply chain attack, chicken little.”
That last, comically self-serving, argument has just been destroyed. Now will anything change? Probably not. Software production in “YOLO” mode is just too damn profitable.
Exactly, but do you think anyone will care? I bet no one.
How did the malware got into the build system. Even if it is how solarwinds missed to scan the binary before shipping. Did they missed all the quality gates.
How come it took so long to identify. Is incapable people?
Also does the active directory which provides SAML is monitored with some quality control check on access. So many unanswered questions.
Is it agile fault this issue is not captured or not enough technical guys to understand the whole issue.
Indeed…how did the APT’s payload become part of the SolarWind’s build? Where are Solarwind’s access logs for their SWCM platform? Were the access credentials of an employee impersonated via a VMWare privilege escalation? Also…how did the APT figure out where to push their payload/malware into the stack? Must have taken a bit of recon to fish through the compilation units to decide where to build it. And…why wasn’t the SWCM log watched to determine if a change was authorized/approved? Guess with privilege escalation, one can forge credentials that impersonate an employee.
Thanks Brian, Appreciate you. Was Microsoft infiltrated? If so how, what is the impact and what should we do about it?
My ignorant & probably un-PC question after reading Ed Snowden’s book & watching “Citizen Four”, can the NSA be trusted to tell us the truth about anything? Just my thought.
Why do we need to have everything on the net. Why not only read-only databases, everything else should be intranet, not accessible to anyone outside the company. No pathway out to the internet from within.
Get your Cisco NGFW at LanBaas.nl
Stay safe
cybersecurity is a hot topic and a monepit
Save on your it budget
Stay safe
Is the solarwinds / VMware hack for dummies available yet? I’m getting stuck on the chain of events that got the hacker from A to B(eing included in the software update).
It’s no surprise. When security is increased, convenience decreases. When convenience is the goal, security takes a hit. Third party network management tools are a convenience to resource-strapped IT departments. This is what happens when supply chain risk management is practiced with the default “see no evil” setting left in place.
Two Unanswered Questions:
1) How did the hackers gain access to SolarWinds to alter the code of the update?
2) What type of review and check system does SolarWinds have that would not catch an altered update before it was released?
“organizations should consider the entire identity trust store as compromised”
For any large organization – this is a massive impact because you aren’t just resetting passwords. Just… wow – what a nightmare.
I can’t wait to hear what Microsoft has to say after they ‘review’, because deep down I think they got more compromised than they want to admit.
Even as a small MSP, we’re starting to see some *weird* stuff happening that point to possibly some significant compromise of large service providers.
About SAML tokens: a quote from Russell Haering, maintaner of one of open-source SAML libraries (via LWN.net):
https://news.ycombinator.com/item?id=25424267
1989-
it’s a part of the Internet, a computer network that cross-links a hundred other networks.
Clifford Stoll, The Cuckoo’s Egg