December 15, 2020

The still-unfolding breach at network management software firm SolarWinds may have resulted in malicious code being pushed to nearly 18,000 customers, the company said in a legal filing on Monday. Meanwhile, Microsoft should soon have some idea which and how many SolarWinds customers were affected, as it recently took possession of a key domain name used by the intruders to control infected systems.

On Dec. 13, SolarWinds acknowledged that hackers had inserted malware into a service that provided software updates for its Orion platform, a suite of products broadly used across the U.S. federal government and Fortune 500 firms to monitor the health of their IT networks.

In a Dec. 14 filing with the U.S. Securities and Exchange Commission (SEC), SolarWinds said roughly 33,000 of its more than 300,000 customers were Orion customers, and that fewer than 18,000 customers may have had an installation of the Orion product that contained the malicious code. SolarWinds said the intrusion also compromised its Microsoft Office 365 accounts.

The initial breach disclosure from SolarWinds came five days after cybersecurity incident response firm FireEye announced it had suffered an intrusion that resulted in the theft of some 300 proprietary software tools the company provides to clients to help secure their IT operations.

On Dec. 13, FireEye published a detailed writeup on the malware infrastructure used in the SolarWinds compromise, presenting evidence that the Orion software was first compromised back in March 2020. FireEye didn’t explicitly say its own intrusion was the result of the SolarWinds hack, but the company confirmed as much to KrebsOnSecurity earlier today.

Also on Dec. 13, news broke that the SolarWinds hack resulted in attackers reading the email communications at the U.S. Treasury and Commerce departments.

On Dec. 14, Reuters reported the SolarWinds intrusion also had been used to infiltrate computer networks at the U.S. Department of Homeland Security (DHS). That disclosure came less than 24 hours after DHS’s Cybersecurity and Infrastructure Security Agency (CISA) took the unusual step of issuing an emergency directive ordering all federal agencies to immediately disconnect the affected Orion products from their networks.

ANALYSIS

Security experts have been speculating as to the extent of the damage from the SolarWinds hack, combing through details in the FireEye analysis and elsewhere for clues about how many other organizations may have been hit.

And it seems that Microsoft may now be in perhaps the best position to take stock of the carnage. That’s because sometime on Dec. 14, the software giant took control over a key domain name — avsvmcloud[.]com — that was used by the SolarWinds hackers to communicate with systems compromised by the backdoored Orion product updates.

Armed with that access, Microsoft should be able to tell which organizations have IT systems that are still trying to ping the malicious domain. However, because many Internet service providers and affected companies are already blocking systems from accessing that malicious control domain or have disconnected the vulnerable Orion services, Microsoft’s visibility may be somewhat limited.

Microsoft has a long history of working with federal investigators and the U.S. courts to seize control over domains involved in global malware menaces, particularly when those sites are being used primarily to attack Microsoft Windows customers.

Microsoft dodged direct questions about its visibility into the malware control domain, suggesting those queries would be better put to FireEye or GoDaddy (the current domain registrar for the malware control server). But in a response on Twitter, Microsoft spokesperson Jeff Jones seemed to confirm that control of the malicious domain had changed hands.

“We worked closely with FireEye, Microsoft and others to help keep the internet safe and secure,” GoDaddy said in a written statement. “Due to an ongoing investigation and our customer privacy policy, we can’t comment further at this time.”

FireEye declined to answer questions about exactly when it learned of its own intrusion via the Orion compromise, or approximately when attackers first started offloading sensitive tools from FireEye’s network. But the question is an interesting one because its answer may speak to the motivations and priorities of the hackers.

Based on the timeline known so far, the perpetrators of this elaborate hack would have had a fairly good idea back in March which of SolarWinds’ 18,000 Orion customers were worth targeting, and perhaps even in what order.

Alan Paller, director of research for the SANS Institute, a security education and training company based in Maryland, said the attackers likely chose to prioritize their targets based on some calculation of risk versus reward.

Paller said the bad guys probably sought to balance the perceived strategic value of compromising each target with the relative likelihood that exploiting them might result in the entire operation being found out and dismantled.

“The way this probably played out is the guy running the cybercrime team asked his people to build a spreadsheet where they ranked targets by the value of what they could get from each victim,” Paller said. “And then next to that they likely put a score for how good the malware hunters are at the targets, and said let’s first go after the highest priority ones that have a hunter score of less than a certain amount.”

The breach at SolarWinds could well turn into an existential event for the company, depending on how customers react and how SolarWinds is able to weather the lawsuits that will almost certainly ensue.

“The lawsuits are coming, and I hope they have a good general counsel,” said James Lewis, senior vice president at the Center for Strategic and International Studies. “Now that the government is telling people to turn off [the SolarWinds] software, the question is will anyone turn it back on?”

According to its SEC filing, total revenue from the Orion products across all customers — including those who may have had an installation of the Orion products that contained the malicious update — was approximately $343 million, or roughly 45 percent of the firm’s total revenue. SolarWinds’ stock price has fallen 25 percent since news of the breach first broke.

Some of the legal and regulatory fallout may hinge on what SolarWinds knew or should have known about the incident, when, and how it responded. For example, Vinoth Kumar, a cybersecurity “bug hunter” who has earned cash bounties and recognition from multiple companies for reporting security flaws in their products and services, posted on Twitter that he notified SolarWinds in November 2019 that the company’s software download website was protected by a simple password that was published in the clear on SolarWinds’ code repository at Github.

Andrew Morris, founder of the security firm GreyNoise Intelligence, on said that as of Tuesday evening SolarWinds still hadn’t removed the compromised Orion software updates from its distribution server.

Another open question is how or whether the incoming U.S. Congress and presidential administration will react to this apparently broad cybersecurity event. CSIS’s Lewis says he doubts lawmakers will be able to agree on any legislative response, but he said it’s likely the Biden administration will do something.

“It will be a good new focus for DHS, and the administration can issue an executive order that says federal agencies with regulatory authority need to manage these things better,” Lewis said. “But whoever did this couldn’t have picked a better time to cause a problem, because their timing almost guarantees a fumbled U.S. response.”


122 thoughts on “SolarWinds Hack Could Affect 18K Customers

  1. Ken McCarthy

    Anyone looking at their firewalls? This is plain incompetence with Krebs (CISA director) at the top of the heap of idiots.
    Yeah looking at firewall logs is eye-wateringly boring but ITS YOUR JOB!

    1. JamminJ

      CISA is a new agency, Trump created. DHS has other cyber protection teams who also failed. NSA didn’t catch this either. It was everyone’s job.

    1. JamminJ

      FAKE NEWS.
      Stop with the Qanon nonsense, it won’t work here.

      Dominion does NOT use Solarwinds Orion, never has. Nor does it use software by their competitor, Smartmatic.

      1. botazefa

        “Stop with the Qanon nonsense, it won’t work here.

        Dominion does NOT use Solarwinds Orion, never has. Nor does it use software by their competitor, Smartmatic.”

        Do you have a link to a statement from Dominion that they do not use SolarWinds Orion anywhere in their infrastructure? SolarWinds is pretty ubiquitous and it would be interesting if they don’t use it.

        1. George McDowell

          SolarWinds is ANYTHING but ubiquitous. As such, it is not interesting in the least. And we know two things for certain:
          1. Russia/Putin is behind this attack.
          2. Traitor Trump is Putin’s Puppet and has not said ONE WORD about this attack even though he is supposedly OUR president!

          So if the Russian were in control of Dominion, Republicans and Putin would have won every election where their systems are used.

          Go away, QAnon.

          1. JamminJ

            Conspiracy theories need to be shut down with facts immediately. And the very simple fact of whether or not Solarwinds Orion was used, is sufficient to shut up Qanon nutjobs posting here. A logo from the same company, only shows their use of an FTP server called Serv-U, which was NOT affected by the compromise. This conspiracy theory has been debunked many times already…. by several cybersecurity people who know what they are talking about. It keeps getting rehashed by conspiracy theorists quoting Qanan or far-right GatewayPundit.

      2. Jaystar

        Dominion didn’t need to hack anything. They had American Democrat Traitors changing the ballots locally. You’ve seen the videos of storgae cards being edited, suitcases of ballots appearing and being added to the count after hours and the woman hired to illegally change and fill in ballots.
        The only traitors in this country are the Communist Democrats.

        1. JamminJ

          All fabricated evidence that isn’t even convincing Federalist Society Conservative Judges.

          Russian trolls love manipulating video to stir up conspiracy theories in the weak minded sheep.

          1. Jaystar

            Well we see you’re a Communist Democrat backing Traitor Joe and Hunter and their illegal Chinese Dealings.
            You believe everything about Trump but Biden is an Angel with 47 years of doing nothing but graft and public racism. It’s all fake news when it comes to Bottom Feeder Biden.
            Right Comrade JamminJ ?

            1. JamminJ

              You try and try to create a false equivalency between the two, but nobody is worshiping Biden. He doesn’t have a cult following like Trump does.

              There was time about 100 years ago when a similar person rose to power, and commanded such dedication from millions of followers. He also drummed up fears of communists everywhere trying to take him and his country down. His name was Hitler. And every time you call someone a communist, you sound like a Nazi.

              1. Comrade Stalin

                Real Communism hasn’t been tried yet, right comrade?

                1. JamminJ

                  Neither has pure capitalism.
                  Both are capable of oppression. Communism is just a failed economic principle, and we know enough from history to not to try any more.

                  The problem is that the uneducated masses (now in the US and 100 years ago in Germany) don’t know what is or is not communism. Is universal healthcare communist?

                  It is right to fear communism, but it is important to realize that there is no credible threat from that failed ideology. Rather it’s a boogeyman ghost story used solely to frighten ignorant little children into compliance.

  2. mark

    You know, the obvious question that no one seems to have asked about all this: how recently has the CIA/NSA executed a major hack of this size in Russia?

    1. JamminJ

      Something like, “MoonFarts network monitoring suite was hacked by NSA”?
      I doubt it, purely because their infrastructure isn’t anything like ours. The west values capitalism and lets private entities run as they see fit. It’s generally a good thing for business, growth and our economy. But is vulnerable in certain ways. In Russia, they are willing to go back to paper and pencil for critical operations if they suspect we might be able to infiltrate. They are willing and able to shrink their attack surface beyond what is acceptable here.

    2. George McDowell

      No, the obvious question is that with the degenerate Traitor Trump family already having given Putin’s people all the intelligence they asked for, what the Russians could gain from this hack.

      1. Sector

        So you missed those Hunter Biden emails and pics. You want some proof of colluding with the enemy, there’s a ton of it for the Biden family. CCP throws money around and apparently many listen.

        1. JamminJ

          More Qanon bull crap.
          You guys have become a laughingstock.

          1. Jaystar

            The real laughing stock is the blatant voter fraud and we have an illegitimate President Elect.

            1. JamminJ

              Cry all you like. The election was conducted as well as expected, and just as safe as previous elections. Human error abounds, with recounts, audits and canvassing catching the errors and making corrections.

              Trump was always wildly unpopular, he was going to lose.
              Triggered Qanons are now flooding the internet with butthurt conspiracy theories because they are emotional train wrecks.

    3. Les

      They don’t have to. The NSA code is shipped with the networking and computer hardware. Malicious network monitors are built in.

  3. Andrew H

    Guys,
    Leave the political bickering out of it. What technical and administrative controls can be put in place in the future to prevent this. That is what this forum should be about. If the twitter reports are accurate, it looks to me like Solarwinds are criminally? negligent in allowing ‘anyone’ to upload trojanized code. Then someone was able to usurp one of Microsoft’s ‘trusted’? IP addresses. This would be comical if not so serious.

    1. JamminJ

      Political bickering happens here and on Twitter alike. Qanon trolls are out and about, and conspiracy theories need to get immediately challenged with facts, or else they spread. Sorry if that clutters the comment section.

      There a several controls to mitigate the threat.
      Network segmentation and robust firewall rules that do not allow backdoors to reach their C2 servers on the internet.

      The use of 3rd party software and services has been common for so long. The benefits of not having to develop and manage an entire security stack is huge, and allows companies large and small to save money and effort while remaining reasonably secure. This unfortunately results in technical debt and the risk of supply chain attacks.
      I doubt it is “criminal” as they can probably prove due diligence. Solarwinds is probably going to suffer economically, and with civil cases… but probably not criminally. Not for being a victim of a nation-state attack. Twitter sure did not after their debacle.

      Supply chain attacks are rare, but extremely dangerous and hard to detect. They involve many stages and compromising signing keys. It wasn’t “anyone” uploading trojanized code. It was a complex attack, and both government and private sector experts agree the level of sophistication indicate a nation-state level actor.
      A good supply chain attack could go unnoticed for years. It is a good thing that Fire-eye was one of the victims… because not everyone would have found this unless/until they knew the IOCs to look for.

  4. ACES ETM

    Thanks for the update and quick reply. I’ll be sure to keep an eye on this thread. Looking for the same issue. Bumped into your thread. Thanks for creating it. Looking forward for solution.

  5. ACES ETM

    Thanks for the update and quick reply. I’ll be sure to keep an eye on this thread. Looking for the same issue. Bumped into your thread. Thanks for creating it. Looking forward for solution.

  6. ACES ETM

    The preliminary breach disclosure from SolarWinds got here 5 days after cybersecurity incident response agency FireEye introduced it had suffered an intrusion that resulted within the theft of some 300 proprietary software program instruments the corporate supplies to shoppers to assist safe their IT operations.

  7. RitchieM

    Are there realy only their customers affected? What about possible security breaches in the customers software (MS, Cisco, Symantec… )?

Comments are closed.