Communications at the U.S. Treasury and Commerce Departments were reportedly compromised by a supply chain attack on SolarWinds, a security vendor that helps the federal government and a range of Fortune 500 companies monitor the health of their IT networks. Given the breadth of the company’s customer base, experts say the incident may be just the first of many such disclosures.
Some of SolarWinds’ customers. Source: solarwinds.com
According to a Reuters story, hackers believed to be working for Russia have been monitoring internal email traffic at the U.S. Treasury and Commerce departments. Reuters reports the attackers were able to surreptitiously tamper with updates released by SolarWinds for its Orion platform, a suite of network management tools.
In a security advisory, Austin, Texas based SolarWinds acknowledged its systems “experienced a highly sophisticated, manual supply chain attack on SolarWinds Orion Platform software builds for versions 2019.4 HF 5 through 2020.2.1, released between March 2020 and June 2020.”
In response to the intrusions at Treasury and Commerce, the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) took the unusual step of issuing an emergency directive ordering all federal agencies to immediately disconnect the affected Orion products from their networks.
“Treat all hosts monitored by the SolarWinds Orion monitoring software as compromised by threat actors and assume that further persistence mechanisms have been deployed,” CISA advised.
A blog post by Microsoft says the attackers were able to add malicious code to software updates provided by SolarWinds for Orion users. “This results in the attacker gaining a foothold in the network, which the attacker can use to gain elevated credentials,” Microsoft wrote.
From there, the attackers would be able to forge single sign-on tokens that impersonate any of the organization’s existing users and accounts, including highly privileged accounts on the network.
“Using highly privileged accounts acquired through the technique above or other means, attackers may add their own credentials to existing application service principals, enabling them to call APIs with the permission assigned to that application,” Microsoft explained.
Malicious code added to an Orion software update may have gone undetected by antivirus software and other security tools on host systems thanks in part to guidance from SolarWinds itself. In this support advisory, SolarWinds says its products may not work properly unless their file directories are exempted from antivirus scans and group policy object restrictions.
The Reuters story quotes several anonymous sources saying the intrusions at the Commerce and Treasury departments could be just the tip of the iceberg. That seems like a fair bet.
SolarWinds says it has over 300,000 customers including:
-more than 425 of the U.S. Fortune 500
-all ten of the top ten US telecommunications companies
-all five branches of the U.S. military
-all five of the top five U.S. accounting firms
-the Pentagon
-the State Department
-the National Security Agency
-the Department of Justice
-The White House.
It’s unclear how many of the customers listed on SolarWinds’ website are users of the affected Orion products. But Reuters reports the supply chain attack on SolarWinds is connected to a broad campaign that also involved the recently disclosed hack at FireEye, wherein hackers gained access to a slew of proprietary tools the company uses to help customers find security weaknesses in their computers and networks.
The compromises at the U.S. federal agencies are thought to date back to earlier this summer, and are being blamed on hackers working for the Russian government.
In its own advisory, FireEye said multiple updates poisoned with a malicious backdoor program were digitally signed with a SolarWinds certificate from March through May 2020, and posted to the SolarWindws update website.
FireEye posits the impact of the hack on SolarWinds is widespread, affecting public and private organizations around the world.
“The victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East,” the company’s analysts wrote. “We anticipate there are additional victims in other countries and verticals.”
Update, 8:30 p.m. ET: An earlier version of this story incorrectly stated that FireEye attributed the SolarWinds attack to APT29. That information has been removed from the story.
Not a good idea to allow Vendors to state which Companies use their products.
True, but security through obscurity isn’t really security. Further, almost all public entities have to disclose where their money is going to, if not exactly what is purchased.
THIS *points up*
It’s trivial to scan and discover such things. Or, in the case of Solarwinds, just assume they use it. You’d have a better than 50% chance of it being true.
I couldn’t agree more. The vendors shouldn’t be advertising who uses their products, as it is basically an open record of where to exploit after it is know that a vulnerability could exist. Further, why would the Federal government or any business entity want to allow a vendor to make that information publicly available on their website. You gain nothing from it as the user, but the vendor gets the marketing perk.
Read your contracts, folks! Many vendors try to slip in there that they can use your company name in their marketing materials. I NEVER say yes to that.
When Windows XP was preparing for release someone at Microsoft wanted to work from home and used a non approved app to do that. The digital signatures granting Microsoft access to all Windows XP computers was stolen. Now we see Microsoft is listed as a vendor who has used this software exposing all Windows users in the world is telling. The ripple affect is imminent.
Keep everything on US soil in closed systems. Require physical transfer for any data to go outside the closed system to create an air gap. Otherwise crap like this will keep happening and get worse and worse.
Airgap is 80s talk.
Limited networking is fine. But it needs to be very limited to exactly what is needed, and there should be numerous boundaries and choke points between systems. There is zero reason why most networks need to have direct Internet access.
Do you know when the U.S. signed (what year) the first contracts for SolarWinds cyber security to handle their accounts?
On Tuesday, the National Security Council announced the administration would be invoking Presidential Policy Directive-41, or PPD-41, which “facilitates continuous and comprehensive coordination for whole-of-government efforts to identify, mitigate, remediate and respond to this incident,” according to a statement from NSC spokesperson John Ullyot.
The directive, signed in the last years of then-President Barack Obama’s administration, creates a chain of command for responding to cyber incidents. The directive states the White House Cybersecurity Coordinator—or “an equivalent successor”—will serve as the chair for a Cyber Response Group to develop a strategy while the Cyber Unified Coordination Group will coordinate between federal agencies.
Within the CUCG, the directive designates the FBI and the National Cyber Investigative Joint Task Force, the Department of Homeland Security’s National Cybersecurity and Communications Integration Center, and the Office of the Director of National Intelligence’s Cyber Threat Intelligence Integration Center as the lead agencies for certain types of responses. Agencies, however, maintain operational control over their networks, “unless mutually agreed upon by agency heads or their designees.”
Since the policy was signed, a few organizational changes have occurred. In 2018, then-National Security Adviser John Bolton eliminated the White House Cybersecurity Coordinator position and the Cybersecurity and Infrastructure Security Agency—previously known as National Protection and Programs Directorate—became a standalone agency.
The Office of the Director of National Intelligence confirmed they would be joining the Cyber Unified Coordination Group, as outlined in PPD-41.
Might as well just turn off all computers.
Its time to go back to no computers until we fix this mess
Easier said than done.
Software, Infrastructure, Platform, etc. as a Service has made most networks exposed. The benefits are significant, and we tend to disregard the benefit when their is a breach… but it still drives the decisions the rest of the time.
There can be an argument to keep all government away from SaaS, and its a good argument. But they see this as inevitable progress.
Quick! To the Sneakernet!
BTW. The Iranian nuclear centrifuges that were attacked were also air-gapped.
That ship has already sailed, my friend
I never opted to have Solarwinds for monitoring as they have only Windows-based distribution, not Linux but wondering even more how they have convinced all top government agencies, almost all fortune 500 just with Windows distribution? It doesn’t make sense to me 🙂
Agree 100%. We looked at SolarWinds, but didn’t move forward because they need a Windows server back end to work.
TOTALLY agree with both of you. Doesn’t make sense to put a Life Support system on windows. A monitoring solution you should be able to lean on to ensure your environment is working and when something is wrong let you know right away. Nothing Windows is reliable except knowing you’ll have a patch about every day and that its very very old technology.
I don’t trust Windows systems as well. So many vulnerabilities. And not as stable as Linux systems to my experience.
All Gov’t agencies have plenty of windows boxes for AD and user workstations.
Because SW is simple enough that you can generally sit down any half competent desktop support level person in front of it and they will get reasonably good results quickly.
When I was just such a desktop guy I looked at linux based tools but didn’t have the basic linux server admin chops to make fast progress. By the time I had those skills I was costing employers twice as much per hour and so they preferred to delegate the monitoring tasks down to a cheaper employee.
Solarwinds niche has always been in that balance of cheap enough licensing, easy enough to use, and flexible enough to get whatever you need out of it.
This! (looks up)
Haresh, hate to admit it looks like $$ is key in choosing government contractors. 🙁
MS Windows is very secure!!! My neighbors dog told me so…
Your neighbor’s dog is Dogbert? You can’t trust him!
So you got your dogtalkinator working? Took you long enough, must have switched the OS from Linux to windows!
I see the update. I owe you an apology. Thank you.
HAHA while the folks at SolarWinds were busy popping champagne bottles, the folks at Russia’s Kremlin were gleefully rubbing their hands together in anticipation..
You knew about this back door in the update, if you didn’t what kind of analysis were you doing? It was YOUR job to know about this. When was the update done Mr Krebs?
Wrong Krebs, Redhead. Brian didn’t run cyber security.
*facepalm*
I wonder if China will give Chris Krebs a big Xmas bonus?
They are Russians not Chinese.
Why are you so sure?
White Rose…
although it could very well end up being the russians, with no strong evidence presented it is probably best to leave those allegations out. Especially as it could just as easily be the chinese.
Attribution is hard… but there are ways to do it with a fair bit of accuracy down to the APT groups.
It may be less than 80% certainty that its Russia… but that does not mean it could equally be China… it is ‘unlikely’ to be China, but there could be a 20% chance.
Always remember…. Russian disinformation campaigns working against government and playing out here (social media)… has incentive to smokescreen.
It is far more likely that it was Russia, and that some comments here are intentionally trying to shift suspicion away to someone else. It happens all the time.
I would be interested to see where the experts derive their understanding that the hack is 4 times more likely to be russian than the authoritarian chinese government. I don’t doubt the russians are capable, but the chinese are no slouches. Add in that the chinese are more on a war footing with america than russia, i do wonder why the chinese don’t get more of a look. Hope the blinders has nothing to do with the recent discovery of the 2 million CCP agents spread around the world. Which as a sidenote, is a rather smart move by the dictatorship in Beijing, leveraging their sizeable population effectively like that.
Methods of attribution are mostly about TTPs (Tactics, Techniques and Procedures). There is a lot of sharing of code once it’s out in the public space, which is why leaks like Vault7 are so bad for intelligence gathering since it becomes hard to attribute an attack.
But for the most part, a lot of these APT groups are still very much siloed and like to do things a specific way that, when combined, can profile them in a fairly accurate way.
Then their is the higher level targeting and objectives…
Rule of Thumb, if its attacking finance, business or intellectual property… China is indeed the first suspect. If it is military, infrastructure or energy… we looking at you, Russia.
Russia and China are both waging war. China’s is a trade war for the most part. Russia’s is something entirely different. They both want us weak, but in different ways.
The media frenzy around 2 million CPC members throughout the world is just another Red Scare.
CPC members, are just members of the communist party, not necessarily spies as the pro-american business news would have you believe. And CPC is not the same as MSS (Ministry of State Security), which was a legit advisory by CISA.
It is absolutely not surprising that a communist country, with a billion people, which exports a lot of talent, would have people everywhere who have not renounced party affiliation. Doesn’t mean they are spies. It’s McCarthism all over again.
Look closely about which part of the media is reporting this. Business News.
It is similar to what happened in 2018, which was another US propaganda campaign in Trump’s trade war.
China got blamed for the last big supply chain hack way back in 2018.
“The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies”
Of course it was reported by business news as fact, and not by cybersecurity or tech blogs as such.
It was a huge accusation, that all the victims (Super Micro, Apple and many others) denied. What happened? Nothing. Just the trade war with China, that made Bloomberg sound the alarm for a hypothetical attack that never happened. It was a political stunt that garnered continued support for a Trade War.
Secret chips were never found.
When you say “City of Nashville” do you mean all of it…or just some of it?
Last I heard the “US telecommunications” company was running giant cables under the sea to power the internet.
This is why the US doesn’t need a space program.
Wondering if the Google outage is related to this? Maybe they were trying to remediate?
Have you SEEN GCP? They are waaaaaay past using an outside vendor for network anything.
jlooo
Just another inept security company who can’t secure their own software.
Telling customers to NOT run Anti-Virus is exactly the kind of HOLE that a hacker would look to exploit. SolarWinds made their product easy to use, security was a distant second.
Krebs: “guidance from SolarWinds itself … products may *not* work properly unless their file directories are exempted from antivirus scans”
https://support.solarwinds.com/:
Title:”Files and directories to exclude from antivirus scanning…”
“consider excluding the following items from antivirus or security software”… “Exclude whole folders, including subdirectories”
This is unfortunately common. The deeper the hooks your program has… the more it becomes indistinguishable to malware.
DLP, AV, EDR, and other software, often needs to be excluded depending on how well tuned the AV is.
This is one of the pros of having Microsoft provide everything and not to use 3rd parties. Of course, there are cons to this as well.
Not surprising from a company that makes fake review sites and rates their products number one in every category.
netadmintools.com
ittsystems.com
dnsstuff.com
pcwdld.com
comparitech.com
There’s more.
Maybe a little less time making bogus software review sites and a little more time working on product security?
no way, that’s nuts! just checked the rubbish on dnsstuff.com as it fails miserably to purport neutrality by mentioning competitors, albeit at the end of the list with the top 3 being SW products. Good find tcp thanks for the heads up!
No mention that the voting machines run on windows backends with Orion.
Looks like their was a state sponsored attack on dominion voting systems.
Not seeing any reports of voting systems using SolarWinds Orion. Windows yes, but not Orion.
Dominion Voting sure enough did seem to be using SolarWinds – the internet is forever:
https://twitter.com/CodeMonkeyZ/status/1338850031445536769
That tweet isn’t publicly available.
If you’re referring to the logo of SolarWinds on a Serv-U FTP Server login page, I’ve seen it. There is a archive image floating around.
Serv-U is NOT Orion.
SolarWinds is a big company and they make many products. Serv-U is listed as not being affected by this compromised. And before you ask, Fire-eye and others who are working hard to reveal the scope and scale of this attack, has not even suggested that all of SolarWinds products are affected. So there is no reason to even think that there is a problem with Dominion.
Also, even if Dominion is running SolarWinds Orion on their corporate systems. The “voting systems”, tabulators and management systems and such, that are owned and operated by State and County governments… aren’t running the SW Orion network management software.
The voting machines I have booted, when working the polls, ran what looks to be Windows CE. They are not networked. The vote count comes from paper tapes printed at the end of the day. Why would a none networked device be monitored by Solar Winds? Makes no sense.
We used to use Solarwinds Orion. We disposed of in the fall of 2019 because it was so full of bloat and unknowns. Now I am so glad we did. If you can’t review the code, you shouldn’t have it on your network. US Goverment needs to take that stance.
Easier said than done.
Government often takes the lowest bidder, but often politics influence the juicy government contracts.
The ability to review code does not improve your security if you have not reviewed the code. Most open source projects are only supported by relatively few people and/or companies. Most organizations do not review the code of open source software.
The ability to do something and actually doing it are quite different.
@VinceR “The ability to do something and actually doing it are quite different.”
Many view the ability to simply review open source code as deterrent enough as a method of prevention. While in reality, there will exist a vulnerability hiding in plain sight.
Interesting that SolarWinds was able to issue a hot fix almost immediately. I wonder how long they know about this issue before it was reported? For the many organizations using their products, I wonder if SolarWinds will survive this mess?
Having a quick hotfix is good… and is not an indication that they knew beforehand.
What happens is that if there is a clear function that can be patched, without having to rewrite significant portions, it can be hotfixed with maybe a few lines of code.
Since only a few versions are affected, a rollback can be applied as a hotfix too.
The compromise looked to be just a single modified dll file, and the more recent versions were already not compromised so not exactly surprising that they were able to patch it quickly.
I’m fairly certain the patch released today was just more general CYA and security stuff that was previously lower on the priority list. Given that solarwinds didn’t know they had been hacked and the problem disappeared before they were made aware of it I’ve been suspecting that the hackers had already accomplished their goal by June and had repaired the file in question to remove their tracks.
I’m probably one of the few not surprised that another software being used on the very vulnerable WINDOZ Operating System is attacked. No one uses WINDOZ for critical systems.
I also agree with the several comments sorta surprised by announcing what companies to focus your attacks on.
Also surprised by the big companies that use this software which is very unreliable during testing. The SQL database server has to be monsterous and we have found several short comings with this monitoring software the more addons you install from Solarwinds. Its like adding more weights to your legs and tell you to run across the country.
You realise that similar techniques could be used to compromise most OSes?
What if someone compromised the back end of a Puppet Enterprise instance used to service multiple orgs?
So sick of this blinkered view re Windows when this particular kind of exploit could apply to *anything* that’s highly automated and using high privileges where you put all your eggs in one basket.
At the moment, the top 8 products in terms of CVE vulnerabilities are Linux, Android, Mac and various browsers. Then we get the first Windows product, at no 9, Server 2008 (guess how old that is).
The worst single vendor is Adobe and the worst product is Flash. How much physical infrastructure is still managed with products using embedded flash as a UI? (Hint: A lot.)
What happens if any backend management system is compromised, especially enterprisey ones managing multiple orgs? e.g. puppet, openstack/ansible, cloudstack, etc etc
The trouble with tunnel vision like yours is the fact that you think your non-windows kit would be immune to this kind of thing, when it really isn’t. It’s a very dangerous form of complacency.
Mr. Krebs you may want to add an update to your article, This AM a security researcher is indicating the solarwinds core installer is currently carrying the sunburst malware. I am sure you can confirm this independantly.
https://twitter.com/JohnBasham/status/1338707319123795973
As far as other news this is tied getting tied into.
Solarwinds products were used by Dominion based on people finding an internet facing portal with a solarwinds logo on it; There’s some speculation that Solarwinds was used to backdoor the voting machines.
State governments are now involving forensics experts in evaluating what occured on voting machines, e.g. Maricopa county just authorized a full forensic audit of all voting machines so I expect a preliminary report in a week or two. More local governments will follow as appropriate.
Suffice to say in the few forensic reports I’ve seen have shown a lack of basic security architecture and chain of custody in place. They’re showing things like audit logs being wiped or edited. I would not waste the energy speculating on the use of solarwinds by dominion until the forensics guys have taken the time to do their job.
Thanks Bob.
I would also point out the preliminary forensics report was done by ASOC which is run by a hyperpartisan former Republican Candidate, who has submitted previous signed affidavits full of mistakes such as Minnesota Counties he claimed were in Michigan.
So we have to wait for “independent” forensics investigators to examine the machines.
The speculation that Solarwinds is being used by Dominion voting machines seems premature, and there is a history of wild speculation based on bad analytical methodology. For instance, finding a logo on some random website that says dominion. This is similar to previous speculation from ASOC that Edison Research has a server in Iran, because of a similar subdomain/hostname in an open source search for DNS records.
Voting machines are not networked devices. This is nothing but conspiracy theory nonsense. Volunteer to work the polls and you will see how stupid this is.
Confirmed. Speculation was false.
The Dominion corporate system only had Serv-U.
Solarwinds Orion is not the same as the Serv-U FTP server.
Also, even if this system were compromised, a corporate file server is not the same as the voting systems that were sold to state and county governments years ago.
“…don’t virus scan our software…”
This is REALLY bad coding. And they’re the tops in their field? We are in serious trouble.
imgs.xkcd.com/comics/voting_software.png
It isn’t a fair analogy, although funny.
Accidental failure is very different than failure from malicious intention.
Ask the airplane designer about being bullet proof or resistant to anti-aircraft guns. It’s not that they can’t make them more resilient if they have to be used, but there are no guarantees.
Or the building engineers about if elevators are safe in a terrorist attack or even a fire… they would say the same as the software engineer. “Don’t do it, that’s terrifying”.
Why? Because a smart, persistent attacker can always overcome safety controls. Safety is not the same as security. The best way to secure it, would be to eliminate the attack surface as much as possible,… and that is the reason against computerized voting. There isn’t really much of a benefit that is worth the risks.
How in hades did anyone accept security software that fails if its working dir’s are scanned for malware? Apparently lots of security nerds haven’t been reading Krebs, or Schneier.
The real fix is punch input as transfer: helluva lot harder to insert a bug in sequentially numbered cards! Good for the trucking industry, too…
Be interesting to see if simply using VMS would’ve brought this to lot a lot earlier.
More context about a connection between this compromise and the 2020 Election. (TLDR: There is none)
They found a Dominion Voting Systems server using Solarwind’s Serv-U FTP. There is a archive image floating around.
Serv-U is NOT Orion. And Serv-U source code has NOT been compromised.
SolarWinds is a big company and they make many products. Serv-U is listed as NOT being affected by this compromise.
Fire-eye and others who are working hard to reveal the scope and scale of this attack, has not even suggested that ALL of SolarWinds products are affected. So there is no reason to even think that there is a problem with Dominion.
Also, even if Dominion is running SolarWinds Orion on their corporate systems (no indication thus far). The “voting systems”, tabulators and management systems and such, that are owned and operated by State and County governments… are NOT running the SW Orion network management software.
We as cybersecurity professionals have to understand scale and scope of these attacks. We have to be careful to acknowledge the seriousness of a breach without being alarmist.
What is happening is politically motivated hysteria.
The leaps of logic are as follows:
Compromise of Orion source code > owned by Solarwinds > they also make Serv-U FTP server > Dominion Voting Systems corporate network uses Serv-U > Dominion also makes voting systems for a few counties being contested
This amounts to a logic test…
Orion does not equal Serv-U, Dominion corporation does contain Serv-U, Dominion corporation sells systems to election officials, subset of voting systems and subset of systems with Serv-U do NOT overlap.
Some people will try to connect and lump everything in together, in order to confuse, conflate and ultimately de-legitimize.
This effort to uncover the scope and scale of this attack is underway, and there is no reason to even speculate about if this has anything to do with the election.
i used to compete with solarwinds for monitoring at nyc agencies and our solution was $100k plus while solarwinds was $5k … no wonder everyone had it.
With all these talks on china and Russia hackers let us not forget north korea also Iran hackers included others. “LEARN FROM DUMMIES.”
ROFL!! “Do not use anti-virus scans on our installer directory structure”. Why am I not surprised coming from those d-bags at SolarWinds.
It’s time for all those brilliant computer science people in the “free world” to stand up and get involved in politics and put an end to this nonsense. Did you all not think about this day when you created something so fundamentally powerful and set it loose in the world? Worse is coming. Now the clock is ticking. How will you secure our future against the attacks that are to come with the information they will sift? Will you continue to sit on the sidelines and watch while the Russian-orchestrated anti-science “let it wash over us” leadership presides over the rallying of extreme right and extreme left forces prepared to rip us apart? When is it enough? You cannot content yourselves to sit back and try to stay out of it. If you do not act now, we might lose it all. Will you have it said by history that it was on your watch that America fell? Patriots arise!
We are reaping what was sown a little over 25 years ago when the Internet’s original prohibitions against E-commerce were removed. The Net was designed to be secure for a heavily vetted small group of government, university and elite corporate research center (e.g., Bell Lzbs, IBM) users. The Net’s creators predicted the types of breaches we are seeing today, once Mosaic and then Netscape made the Net available for anyone with a phone line.
We are reaping what was sown a little over 25 years ago when the Internet’s original prohibitions against E-commerce were removed. The Net was designed to be secure for a heavily vetted small group of government, university and elite corporate research center (e.g., Bell Lzbs, IBM) users. The Net’s creators predicted the types of breaches we are seeing today, once Mosaic and then Netscape made the Net available for anyone with a phone line.