December 18, 2020

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”

“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.

CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”


142 thoughts on “VMware Flaw a Vector in SolarWinds Breach?

  1. Daniel Chien

    We can stop all hackers. In addition to user ID, PW, auth server needs to verify login device as well. So hackers need to steal your PC in order to break in. There are many ways to verify device like checking CPU/motherboard serial number. The better way is using timestamp-based algorithm to verify login device.

    1. JamminJ

      Device fingerprinting is used in many places, and it’s not a panacea. A Man in the browser attack can get all of the unique information.
      And an attacker will often go through the new device enrollment process. That’s a weak spot considering real admins need to enroll remotely all the time.

  2. Make China Great Again

    oh come on. everyone knows it was china. trump said so.

      1. JamminJ

        You think a supercomputer did this?

        Wow, Trump cultists really get defensive for Russia. This is completely insane. There was a time when patriotic Americans would never bow down to Putin like this.

        Remember, Trump literally asked Russia to hack a political opponent. “Russia, if you’re listening”.

        1. Hillarys Mind

          Your delusional, the largest investigation in history proved Trump had nothing to do with Russia collusion, however Joe and Hunter have taken H U G E payments from the Chinese government, Russia and Ukraine. And the facts are there to support it. But deep state dems will not allow them to EVER be investigated for their cimes.

          1. JamminJ

            The Benghazi investigation cost more, took longer, and came up with nothing.

            The Russian investigation resulted in the indictments of 34 individuals and three Russian businesses on charges ranging from computer hacking to conspiracy and financial crimes.
            Those indictments have led to 7 guilty pleas and 5 people sentenced to prison.
            It concluded that Trump could not be touched since he’s president. Far cry from innocent.

          2. Sergi Tolstoy

            Thought this was about the USA being hacked, the Sovereignity of its Citizenery, anyway….

            Jared, the demon has negotiated every Middle East, China, Russian slush fund deal for the Trump companies since 2014 , just waiting for the Deustche Bank release.

            1. William Kemmler

              And Hunter has negotiated every back channel deal with the Ukraine and China for the Biden family. Big Daddy Joe even had to step in to protect Hunter when the Ukrainian state prosector was getting to close for comfort in his Burisma corruption investigation. So what’s your point, exactly?

              1. JamminJ

                And Hillary Clinton ran a child trafficking ring the basement of a pizzeria. Riiiight?

                There are no limits to the imagination of conspiracy theorist.

                1. William Kemmler

                  “And Hillary Clinton ran a child trafficking ring the basement of a pizzeria. Riiiight?

                  There are no limits to the imagination of conspiracy theorist.”

                  Well, if you want to believe that kind of lunacy then I won’t stop you. But that’s all it is. The lunatic fantasies of the extreme far right that can be easily and safely ignored as such. And that’s what I do. Ignore such idiotic lunacy.

                  1. JamminJ

                    That’s only one example of conspiracy theories used for political character assassination.

                    Pee pee tapes, Benghazi and Burisma are all fantasies and wishful thinking by political opponents to demonize with accusation. None of it has substance.

                    Most Qanon people don’t believe the entire pantheon of theories. But cherry pick the theories they like. And so did you.

        2. Aaron

          Trump has nothing to do with this. The point I made is China has 100x the cyber capability of Russia. And it’s even more if you are considering personnel.

          1. JamminJ

            So you think number of supercomputers equals cyber capability? Do you even know anything about cybersecurity?

            Sorry, but bringing up China is completely political. It’s only because Trump is trying to protect his relationship with Putin, that he suggested China, and because Trump supporters still have delusions of 4 more years, that they believe his claims.

            Experts in this field know attribution… And it’s all pointing to Russia.
            And as enemies of the US go… Russia has always been the biggest threat in cyberspace. By a good margin.

          2. nikehat

            Someone else already pointed out why your comment about supercomputers is asinine, but to your point about your comment having nothing to do with Trump: China wouldn’t even be in any of these conversations unless Trump once again contradicted his own experts and agencies, including even Mike Pompeo, and pointed the finger away from Russia to China. You guys are way too eager to eat up anything that comes out of that giant idiot’s mouth. What you’re doing is all about politics, it has nothing to do with any real fact finding.

            1. Aaron

              Russia is a backward country. They can’t even make anything interesting. Nothing in the average person’s home was made in Russia. If there is anything, it’s a novelty item, useless. Putin rides around on a horse shirtless, it’s a total clown country, and must not be taken seriously. They can’t even test nukes anymore. Their planes “solve” the problem that they don’t have low signature engines by having IR jamming, but everyone already knows “secret” missile systems don’t use IR, but other parts of the spectrum. Russia’s finest anti-air rockets are routinely defeated, in Syria, by US planes.

              https://www.reuters.com/article/syria-crisis-israel/syrian-army-says-israel-hits-southern-damascus-in-second-strike-in-week-idUSKBN2843DE

              This is a Russia-USA proxy war, and Russia is losing at every turn. Their best tech is garbage. A joke. Russia’s response to US high-speed nuclear rockets is to fly around old junk prop planes until the West decides to pretend they’ve only just noticed (they’re followed continuously from takeoff).

              1. Red Robin

                Aaron,
                You are correct to point out that China is far more technologically advanced than Russia. This is partly because of the all of the business deals they are doing with the US and because of their rampant IP theft. The CCP have both the means and the motive to pull of this attack, and there is no reason to rule them out at this point.
                JamminJ,
                There is nothing political about calling attention to the fact that China may have been involved about this. You sound like a crybaby butthurt liberal. Why don’t you take your “rule of thumbs” and uneducated political grieving’s someplace else.

                1. A. Bosch

                  “Crybaby liberal” – well that’s rich coming from a sycophant of the big crybaby, sore loser man Donald Trump. And how’s that Kraken doing? Unleashed yet?

                  1. JamminJ

                    Aaron, Robin,
                    You honestly think that commercial household items not being made in Russia is an indication of cyber warfare capability?
                    Is this the same nonsense that leads you to believe that the number of supercomputers has anything to do with offensive cyber-operations?

                    You seem to have a grade school understanding of Russia. Yes their economy collapsed at the end of the Cold war and the vast majority of their conventional military capability it’s gone.
                    But that is the unique thing about cyber warfare. It is low cost and can evolve independently of large military spending. It doesn’t cost much to utilize a criminal network of hackers for nation-state purposes.

                    China is a huge threat as well, but for different reasons and with different geopolitical motivations. Yes they steal our intellectual property and it should tell you something about this attack that IP was not targeted. Rather conventional cyber-espionage of government and military.

                    China is perfectly capable of carrying out this type of attack, but they would want to simply gain leverage over our economy in this ongoing trade war. They have little interest in military or federal government agencies. TTPs show us more about who is behind this, than your political instincts.

                    Since this trade war began, you and other Trump supporters have become pawns in another red scare against China. Sheep to be manipulated.
                    Just a couple of years ago, they lied about an alleged supply chain attack with physical chips being embedded onto motherboards. All of the alleged victims denied this happened, and no evidence was ever found.
                    China is certainly a foe, but these high-profile attacks being blamed on China are lies that are being fed to you.

                2. JamminJ

                  You honestly think that commercial household items not being made in Russia is an indication of cyber warfare capability?
                  Is this the same nonsense that leads you to believe that the number of supercomputers has anything to do with offensive cyber-operations?

                  You seem to have a grade school understanding of Russia. Yes their economy collapsed at the end of the Cold war and the vast majority of their conventional military capability it’s gone.
                  But that is the unique thing about cyber warfare. It is low cost and can evolve independently of large military spending. It doesn’t cost much to utilize a criminal network of hackers for nation-state purposes.

                3. JamminJ

                  China is a huge threat as well, but for different reasons and with different geopolitical motivations. Yes they steal our intellectual property and it should tell you something about this attack that IP was not targeted. Rather conventional cyber-espionage of government and military.

                  China is perfectly capable of carrying out this type of attack, but they would want to simply gain leverage over our economy in this ongoing trade war. They have little interest in military or federal government agencies. TTPs show us more about who is behind this, than your political instincts.

                  Since this trade war began, you and other Trump supporters have become pawns in another red scare against China. Sheep to be manipulated.
                  Just a couple of years ago, they blamed China for an alleged supply chain attack with physical chips being embedded onto motherboards. All of the alleged victims denied this happened, and no evidence was ever found.

                  China is certainly a foe, but these high-profile attacks being blamed on China are lies that are being fed to you.

        3. Joy Peterson

          Well I’m so happy to see someone managed to find a way to make this Trump-related and at the same time manage to be offensive to “patriots” and Americans in general. I actually made it almost the whole way through my morning without being offended. I might be naive but I was reading this article for insight into worldwide security issues. Glad to see so many others have no interest in the “big picture”. I stopped chanting “Trump is bad, everyone else is good” and therefore failed to recognize this must also be Trump’s fault no matter what any evidence may or may not show.

    1. tu quoque

      Words we’re watching – Whataboutism

      “If everyone is guilty of something, is no one guilty of anything?”

      https://www.merriam-webster.com/words-at-play/whataboutism-origin-meaning

      I also enjoy this one as it dovetails so naturally with ‘espionage’

      psyops: plural noun, often attributive – : military operations usually aimed at influencing the enemy’s state of mind through noncombative means (such as distribution of leaflets)

      https://www.merriam-webster.com/dictionary/psyops

      Cheers!

    2. William

      Why do you think Trump has said it was China?

      I say FOLLOW THE MONEY!

      1. JamminJ

        Yep. China is an easy target. We are in an ongoing trade war with China, so it benefits us to accuse them of wrongdoing, regardless of the facts.

        Remember, just 2 years ago we accused China of planting spy chips onto the motherboards of equipment used in the US. This allegation was denied by the supposed victims, and should have been easy to prove. Not a single shred of proof has been found.

        But the moral of the story is… allegations, even without proof, can sometimes work to produce an effect.

  3. Brazen Zebra

    Do you think APT29 aka Cozy Bear could do such a sophisticated hack in the US on their own? Do you think maybe Edward Snowden gave them some help? I hear Snowden was just granted permanent residency in Russia as of October 2020. Was that a reward? Did the Russians twist Snowden’s arm by reminding him he could be deported at any time? Snowden is a proven genius in computer security and hacking. I would not be surprised if his name pops up in the coming months as one of the key players in this highly successful espionage operation. If true, I guess that would make Edward Snowden the new Kim Philby.

    1. Hydroplane

      Anything is possible, but you should have some facts to support your wild speculations before making them. I think Julian Assange said, ‘That which is asserted without evidence, can be dismissed without argument.’

    2. Gov_Lover

      Brazen Zebra, sounds like an NSA codename for an operation that spies on domestic innocent americans. Combine that with the fact that you’re baselessly accusing snowden of helping the ruskies, its 100% confirmed, we know who you work for.

    3. Alex

      edward snowden a genius hacker?
      HAHAHAHAHAHHAHAHA

      You people are so brainwashed its just sad. this comment section is a sad reflection of the state of our country. My God have mercy on all of us and save our republic.

    4. Prolly not

      Unlikely Snowden had anything to do with the hack. Not saying that Snowden isn’t a talented hacker, but he has been in Russia for 7 years already. He’s far removed from having any inside information.

  4. PattiM

    More and more it’s looking like the Western path to final collapse will occur along the lines of damage to crucial control infrastructure caused by bad cyber-actors. All the regular stuff, though – shortages, disease, etc. [Meadows et al., Tainter, Tainter/Patzek, etc.]

  5. Shawn

    Personally, I’ve had it with information from “anonymous government sources”. There seems to be a few news outlets that rely on this quite a bit.

    An organization that has the ability to create a hack of this sophistication would certainly have the ability to remove any indication of their involvement or conversely add characteristics which would point to someone else as “the bad guy”. So if it looks like it came from Russia, then perhaps that’s the one state that can be eliminated.

    1. Mac Namun

      Why the Russia focus? I’ll bet dollars to donuts it was the CCP. Find the incentive, find the actors.

      1. JamminJ

        Methods of attribution are mostly about TTPs (Tactics, Techniques and Procedures). There is a lot of sharing of code once it’s out in the public space, which is why leaks like Vault7 are so bad for intelligence gathering since it becomes hard to attribute an attack.
        But for the most part, a lot of these APT groups are still very much siloed and like to do things a specific way that, when combined, can profile them in a fairly accurate way.

        Then their is the higher level targeting and objectives…
        Rule of Thumb, if its attacking finance, business or intellectual property… China is indeed the first suspect. If it is military, infrastructure or energy… we looking at you, Russia.

        Russia and China are both waging war. China’s is a trade war for the most part. Russia’s is something entirely different. They both want us weak, but in different ways.

        Incentive alone, still points to Russia, not China.
        The fact that the bulk of the malware lies dormant when infecting corporations (unless can be used to get deeper into government), tells us it isn’t likely the Chinese.
        And the better indicator of attribution, is TTP… which also points to Russia.

  6. Hasan Hussein

    Thanks Brian,
    l have been a subscriber for years.
    l am writing because of a radio commercial.
    The commercial advertised Raccoon as a business support.
    l now looked it up and it is an “infostealer” at $75-$200 per week.
    I googled and got this sketchy info. 0ne place was cyberark.com . . .threat-reserch-blog/raccoon …infostealer.
    l have heard this commercial twice in the last week. l am sure these two items are the same. The commercial is in Cincinnati.
    Thought you may like to know.
    Please do not publicize this.
    H.

  7. Robert Waters

    Once we determine attribution for the cybercrime
    Once we have a vaccine
    Once we have our stimulus checks
    Once the US can get back to normal
    Once Biden is in office…..

    Once there was a time when America believed in God.

  8. Bob Dobs

    These kids don’t even know who the SVR or GRU are.

    Russian troll pawns, and they get all their info from a chick in a bikini who’s photo got ganked by “One Africa, One Success”.

    1. Dfjo

      Since the ascension of Putin, I refer to whole lot as “Okhrana”.
      Look back at the pre-revolutionary secret police.

  9. Catwhisperer

    Speaking of CISA, I just read the Vulnerability Summary issued in the 21st of December by CERT. You’ve got to be f*****g kiddig me! Throughout all the 10’s, there are more than a few BLANK ROOT PASSWORDS hardcoded in. Really, after all we have been through?

    At work, I have people that actually put the credential “password” on an internet facing account, in 2020. Thankfully it is a third party app that they use that is not on our hardware. But this is an adult that is living in IT Lala land and is why the wetware will always do you in.

    This has lead me to the idea of retiring. Sheep herding in the Colorado Rockies (not WY, btw, so the talk doesn’t start), or maybe Blacksmithing, to get the stress out…

  10. quixote

    Let’s assume the US government used open source software, would any of these hacks have been less likely to succeed?

    I keep seeing “proprietary” security software, proprietary this, proprietary that, plus they seem to be talking about all Windows machines. That’s what has me wondering whether maybe the “proprietary” aspect is part of the problem. There are an awful lot more eyes looking for vulnerabilities in open source.

    The amount I know about cybersecurity would fit on a postage stamp, a small one, obviously.

    1. Benny

      As it was an attack on the software vendor, who secured their repository with a leaking password, that contained the update package for the software, it would not matter if it was closed source or open source.

      This was a case of vulnerability in the daily physical security practices of those responsible for development and maintenance of SolarWinds.

      It is also a failure of management to enforce and audit basic security practices within the business, especially one that it is providing a security product to government to ensure security of sensitive systems and information.

      SolarWinds also failed to react when the leaking password was reported to them, which unfortunately is a common problem among many large developers of closed source software.

  11. deklen zuzu

    When are you going to connect the AT&T Nashville,TN Bombing with the solarwinds hack and realize this was done to cover up something on those servers. Given the 15min warning it looks like it could of been … not russia or CCP

  12. Question

    My question is why the usa own people dont clean up their country? How come they let happened all those scams and frauds amd cyber stupidness ?
    How come ? If i would be in usa and it will be my country i will ask right to way from authorites are you guys sleeping?
    Clean all the sht . Funny country how people just watch how the country is going down

    1. Benny

      US politicians only just passed legislation that IT products supplied to government must meet minimum security requirements. The legislation only applies to products and services supplied to government.

      It is not just the law which allows many security vulnerabilities, but also daily human behavior.
      Humans are the easiest entry point into any system, as they are careless, naive and predictable, especially the politicians (the simplest hack).

  13. pol bel

    Very interesting that roughly a month after Chris Krebs, head of the Cybersecurity and Infrastructure Security Agency get fired, officially for his work on the elections, news is leaked about a major sophisticated cyber-invasion of numerous governmental entities, going on for months or possibly years under his watch…
    He probably has a NDA over his head preventing him to comment publicly.

    1. JamminJ

      Lots of things happening this year, unrelated.
      The NSA and Cyber Command are more responsible for protection of these types of attacks, than CISA which is a brand new agency under DHS.

      Lots of failure to go around.

  14. buzzd

    Thank you Brian, and a happy and healthy New Year. I ran an NGO website, answered queries, updated, etc. for 15 years. It even kick-started my divorce. It’s a slog and I salute you for getting a good start on it. LOL

  15. Freeman

    I hold in my hand more proof that I am the President of the United States than all of the NSA papers, Corporate Media stories, and various sources have provided that this hack has anything to do with Russia.

    1. JamminJ

      What evidence points to Russia?

      Attorney General William Barr and Secretary of State Mike Pompeo both pointed the finger at Russia as the culprit. The attack, which showed patience and sophistication, is consistent with Russian tactics and hacking techniques, according to cybersecurity experts. A prime suspect is APT 29, a notorious group of hackers tied to the Russian government. The Kremlin denies involvement. President Donald Trump, who has previously contradicted U.S. assessments of Russian cyber activity — including a Russian hacking and disinformation campaign in the lead up to the 2016 presidential election — has downplayed the hack and Russia’s role in it. He suggested in a tweet that China was involved. Members of his party disagree. Marco Rubio, acting chairman of the Senate Judiciary Committee, said the hack — which he characterized as “the gravest cyber intrusion in our history,” was conducted by “Russian intelligence.”

  16. iMessage

    Thanks for the information keep sharing such informative post keep suggesting such post.

  17. iMessage

    Glad you like it. We have been using this for several weeks now and it seems to be going nicely. I also have this same question and i cannot find any proper answers on the internet and also here. need solution.
    This works really well for us, thank you!
    Thanks for the step by step tutorial. Works like a charm! The solution worked for me thanks to the community and the members for the solution.

Comments are closed.