New research into the malware that set the stage for the megabreach at IT vendor SolarWinds shows the perpetrators spent months inside the company’s software development labs honing their attack before inserting malicious code into updates that SolarWinds then shipped to thousands of customers. More worrisome, the research suggests the insidious methods used by the intruders to subvert the company’s software development pipeline could be repurposed against many other major software providers.
In a blog post published Jan. 11, SolarWinds said the attackers first compromised its development environment on Sept. 4, 2019. Soon after, the attackers began testing code designed to surreptitiously inject backdoors into Orion, a suite of tools used by many Fortune 500 firms and a broad swath of the federal government to manage their internal networks.
Image: SolarWinds.
According to SolarWinds and a technical analysis from CrowdStrike, the intruders were trying to work out whether their “Sunspot” malware — designed specifically for use in undermining SolarWinds’ software development process — could successfully insert their malicious “Sunburst” backdoor into Orion products without tripping any alarms or alerting Orion developers.
In October 2019, SolarWinds pushed an update to their Orion customers that contained the modified test code. By February 2020, the intruders had used Sunspot to inject the Sunburst backdoor into the Orion source code, which was then digitally signed by the company and propagated to customers via SolarWinds’ software update process.
Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer system, and to lie in wait until specific Orion source code files were accessed by developers. This allowed the intruders to “replace source code files during the build process, before compilation,” Crowdstrike wrote.
The attackers also included safeguards to prevent the backdoor code lines from appearing in Orion software build logs, and checks to ensure that such tampering wouldn’t cause build errors.
“The design of SUNSPOT suggests [the malware] developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” CrowdStrike wrote.
A third malware strain — dubbed “Teardrop” by FireEye, the company that first disclosed the SolarWinds attack in December — was installed via the backdoored Orion updates on networks that the SolarWinds attackers wanted to plunder more deeply.
So far, the Teardrop malware has been found on several government networks, including the Commerce, Energy and Treasury departments, the Department of Justice and the Administrative Office of the U.S. Courts.
SolarWinds emphasized that while the Sunspot code was specifically designed to compromise the integrity of its software development process, that same process is likely common across the software industry.
“Our concern is that right now similar processes may exist in software development environments at other companies throughout the world,” said SolarWinds CEO Sudhakar Ramakrishna. “The severity and complexity of this attack has taught us that more effectively combatting similar attacks in the future will require an industry-wide approach as well as public-private partnerships that leverage the skills, insight, knowledge, and resources of all constituents.”
Brand reputation compromise have to be dealt within hours.
Being transparent was excellent.
New supply chain attacks shows the total asymmetric nature of cybersecurity.
Humility, ability to laugh at ourselves and attackers are the beginning of human emancipation.
The final statement kind of makes them sound like they’re saying, “well, sure we got hacked, our software spread the hack, but there’s more software out there that was also compromised.”
The article is pretty generic, but what exactly in the development environment was “compromised”?
How were checks bypassed?
If this indeed a development-level hack, does it mean that there’s something in a language that needs to be patched?
After reading the technical analysis the question remains how did their own log monitoring tools miss the new processes created, new files, elevated privileges, scheduled tasks, etc.
Or were they not running monitoring on their dev server?
If the setup was done outside normal work hours (such as in the middle of the night), it is quite possible that the attackers scrubbed any logs and alerts before a human operator would have seen them.
I would expect that the attackers were monitoring the systems from the inside using one or more compromised accounts for at least a couple of months, silently gathering information on how to best attack before going for it.
This wasn’t a script kiddie.
Great lengths were taken by nation-state actors to ensure their code would not be detected on the build server.
“Analysis of a SolarWinds software build server provided insights into how the process was hijacked by StellarParticle in order to insert SUNSPOT into the update packages. The design of SUNSPOT suggests StellarParticle developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers.”
Thing is that operation of SolarWinds is massive so they don’t have one build server maintained by one admin or something that scale. Now you have tens of administrators with tens if not hundreds of servers. Above some level you cannot hire only “best of the best”, even the best admins get tired, have bad days, months. There is always more stuff that can be done than handled by IT so I can understand that some monitoring somewhere was not setup 100% perfect.
Today is Patch Tuesday !
Something tells me that Sudhakar Ramakrishna is trying to cover his rear and white wash what happened.
Didn’t they have one of the dev accounts under admin1234 password?
I mean it’s easy for developers to remember, right 🙂 plus wasn’t it uploaded to public source code repository as well? Not much sophistication for the Russians to find out hah?
So where to from here ? You can bet that the next step will be compromising the compilers and swdev systems that are used to create these products.
Imagine if VisStudio (among others) was inserting these malware code blocks for us ???
It indeed reminded me of “Trusting trust”:
https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf
Agree with some of the other commenters – there is a lot of information missing here. There is mixing of the terms development environment and build server.
Are developers committing compiled binary to source code? If code is compiled on build servers and not development environments, how was the build environment compromised? Is no one doing code reviews on pull requests? Was the build system itself hacked (Jenkins/Bamboo/whatever)?
Without an in depth description of how the *initial* hack was accomplished, this isn’t going to be something other companies can prevent. All of the damage done began when someone got in somewhere somehow.
…if you read between the lines in the crowdstrike blog linked by solarwinds it’s clear that to monitor when the compiler is running someone had pretty deep access to solarwinds build machines, and thus got in via a good, nearly undetectable hack…
…in fact had fireeye not found them…
Hey everyone commenting about the lack of detail. Click on the ‘technical analysis’ link above to read the CrowdStrike post. It will answer most of your questions.
Don’t know if this is exactly how they did it but from the description above, they may of replaced their source code control tool in the build VM (git or something else), wrap the real tool, forward commands to the real tool, and then do a diff patch on selective pulled source files.
Otherwise something similar could be done with whatever they are using to orchestrate their actual build – wrap the real tool and insert a source code diff before compiling, and revert the local source afterwards.
”… they may of replaced their source code…”
What does that mean?
Or did you meant to write “they may have replaced their source code”.
“Or did you meant to write ‘they may have replaced their source code’.”
What does that mean. Or did you mean to write “or did you mean to write…”.
🙂
Ah technical analysis explains all. Clever attack.
Thanks to this article I can learn more. Expand my knowledge and abilities. Actually the article is very real.
These kinds of megahacks are only possible with over-centralization and lack of due diligence on a massive scale.
Sounds like a civilization problem in the sense that wealth makes lazy and less vigilant, such lazynesses being a classical trap wealthy societies fall into during history.
Back to human sized structures where love can be fully expressed (aka lived by) might help.
Just read about solarleaks.net selling stolen data from the various breaches. Media reports this strengthens the case that it was perpetrated by Russian state-sponsored actors. Website is claiming to sell data stolen from Microsoft for $600,000, and Cisco for $500,000. Also, the website has listed red team tools and source code stolen from FireEye for $50,000, and SolarWinds source code and a dump of their customer portal for $250,000 or they’ll sell everything for $1M. The website domain is registered through a registrar known to be used by other Russian hacking groups. The website reportedly mentions that this is the first batch of stolen data and more data will be put up for sale later.
I don’t see how this supports the theory that the breaches were Russian state-sponsored. If it were, it seems to me that Russia wouldn’t sell the information. One, they don’t need the money and two, why give the intel to others. If you’re spying for state gain, you’re going to keep the information to yourselves. Sounds more like a criminal group to me.
Could also be a false flag to push the blame to certain people and create an image which the media sells to the public.
False flag would need a massive conspiracy involving both government and private security researchers who are all saying this is highly likely to be from Russia.
The only false flag in recent memory didn’t get very far because they couldn’t get the alleged victim companies to go along with it.
Remember SuperMicro?
A certain US business publication (not cyber security) accused China of a massive supply chain breach involving tiny chips hidden on motherboards sold to US companies.
All the companies investigated and outright denied this happened. Physical evidence, unlike software, would have been found by now, but nothing was ever found.
It turns out, this was at the height of the Trade War with China, and certain US business interests felt it was worth it to stir up suspicion of China to gain leverage in this war.
THAT was a false flag.
This attack was sneaky, but was confirmed by some of the best experts. Attribution isn’t 100% and it’s still early… but so far, nothing other than pure politics, points to China. Russia is the likely suspect.
Russian groups often sell access. The primary goals can be espionage, but when that fails (when they are discovered)… secondary objectives can still be fulfilled.
Destabilization and overall chaos is a known objective for Russia. General weakness in the west, bolsters the geopolitical influence that Russia has.
Also, cyber crime groups are VERY active in Russia. They are given a free pass by the Russian government to attack the west, and they all know the will never be extradited.
So yeah… this does “strengthen the case that it was perpetrated by Russian state-sponsored actors”.
Am I wrong in believing that static code analysis and a proper secure development pipeline would have found this? Every piece of code injected into software should have both automatic and manual review.
My guess is that SolarWinds at most had some basic automated checks and did not have very strong change control.
While the nation-state actor could have subverted this type of security, it is definitely a cautionary tale for software companies. Will be following closely to learn the lessons.
If you read the detailed analysis Crowdstrike link above, the weakness was not lack of change control or lack of code review: the bad code was injected outside of source code control during the build process, as the binary was generated.
To summarize, the only way they had a chance to catch this was to catch a rogue process running on the build server; nearly impossible due to the stealth nature of the malware.
Damn clever hack. It detects when the build process is started on the build server (MsBuild.exe) and modifies the source code just before it is read by the compiler.
Aside from the extreme difficulty of caching the hack in the act of modifying the source code, any detection would have been very hard given the many counter measures taken to keep the hack hidden.
This should reinforce the importance of keeping hackers out of critical systems, instead of thinking that intrusion detection will work.
Why is this labelled “Jan 21”? Does someone have a time machine?
If so, I have a few questions about lottery numbers…
12 JAN 21 –> January 12, 2021
SolarWinds stated that the malicious code was not present in their source control repository. This means that a key problem (pun intended) they had (have?) is that the code signing system had no coupling with the source control repository so it signed anything it received from an authenticated user.
Even if the signing key was in an HSM, if the build server has access to the HSM it can sign whatever it wants under this model.
What they need to do is have the code signing system validate that what it is signing matches what is in the source control repository. This would then require the attacker to commit the malicious payload to the repository where it is easier to detect (assuming code reviews are part of their SDLC).
So… what to do about it?
I run a website for a small non-profit. Our host company uses SolarWinds Spam Protection.
Is our site compromised? How would we know? What should we do about it?
https://www.solarwinds.com/securityadvisory
The Solarwinds Spam product, “SpamExperts” is listed as “NOT AFFECTED”.
Pretty much only the components of their “Orion” suite was compromised.
If you are still concerned, you can hire a cybersecurity firm to perform an assessment. Threat hunting teams know what to look for.
From what we are seeing, the targets have been government agencies, and large IT companies that would do business with government. So small non-profit companies normally would not be a target.
And even if you ran Solarwinds Orion…. when the malware communicated with command and control servers, and determined that your system wasn’t a target, the malware would shutdown.
https://www.wired.com/story/solarwinds-russia-hackers-turla-malware/
Attribution to Russian Intelligence is becoming clearer.
Many thanks JamminJ. I will sleep better for knowing this.
What I wonder is nobody using firewalls like a Fortigate? All kinds of intrusion attempts are detected by lets say, a 201E , and are blocked. Wouldn’t something like that have detected the communications with the CnC server?
Lenovo updates? Wondering if anyone has heard that Lenovo’s device update procedure using their Vantage software is secure?
We (and others), had a Russian language Amazon Music popup appear today from Lenovo. Disappeared after a few seconds, but we’ve never gotten these notices before and having them be in Cyrillic shook up my users.
Thanks for any information you can provide.