A key malicious domain name used to control potentially thousands of computer systems compromised via the months-long breach at network monitoring software vendor SolarWinds was commandeered by security experts and used as a “killswitch” designed to turn the sprawling cybercrime operation against itself, KrebsOnSecurity has learned.
Austin, Texas-based SolarWinds disclosed this week that a compromise of its software update servers earlier this year may have resulted in malicious code being pushed to nearly 18,000 customers of its Orion platform. Many U.S. federal agencies and Fortune 500 firms use(d) Orion to monitor the health of their IT networks.
On Dec. 13, cyber incident response firm FireEye published a detailed writeup on the malware infrastructure used in the SolarWinds compromise, presenting evidence that the Orion software was first compromised back in March 2020. FireEye said hacked networks were seen communicating with a malicious domain name — avsvmcloud[.]com — one of several domains the attackers had set up to control affected systems.
As first reported here on Tuesday, there were signs over the past few days that control over the domain had been transferred to Microsoft. Asked about the changeover, Microsoft referred questions to FireEye and to GoDaddy, the current domain name registrar for the malicious site.
Today, FireEye responded that the domain seizure was part of a collaborative effort to prevent networks that may have been affected by the compromised SolarWinds software update from communicating with the attackers. What’s more, the company said the domain was reconfigured to act as a “killswitch” that would prevent the malware from continuing to operate in some circumstances.
“SUNBURST is the malware that was distributed through SolarWinds software,” FireEye said in a statement shared with KrebsOnSecurity. “As part of FireEye’s analysis of SUNBURST, we identified a killswitch that would prevent SUNBURST from continuing to operate.”
The statement continues:
“Depending on the IP address returned when the malware resolves avsvmcloud[.]com, under certain conditions, the malware would terminate itself and prevent further execution. FireEye collaborated with GoDaddy and Microsoft to deactivate SUNBURST infections.”
“This killswitch will affect new and previous SUNBURST infections by disabling SUNBURST deployments that are still beaconing to avsvmcloud[.]com. However, in the intrusions FireEye has seen, this actor moved quickly to establish additional persistent mechanisms to access to victim networks beyond the SUNBURST backdoor.
This killswitch will not remove the actor from victim networks where they have established other backdoors. However, it will make it more difficult to for the actor to leverage the previously distributed versions of SUNBURST.”
It is likely that given their visibility into and control over the malicious domain, Microsoft, FireEye, GoDaddy and others now have a decent idea which companies may still be struggling with SUNBURST infections.
The killswitch revelations came as security researchers said they’d made progress in decoding SUNBURST’s obfuscated communications methods. Chinese cybersecurity firm RedDrip Team published their findings on Github, saying its decoder tool had identified nearly a hundred suspected victims of the SolarWinds/Orion breach, including universities, governments and high tech companies.
Meanwhile, the potential legal fallout for SolarWinds in the wake of this breach continues to worsen. The Washington Post reported Tuesday that top investors in SolarWinds sold millions of dollars in stock in the days before the intrusion was revealed. SolarWinds’s stock price has fallen more than 20 percent in the past few days. The Post cited former enforcement officials at the U.S. Securities and Exchange Commission (SEC) saying the sales were likely to prompt an insider trading investigation.
Excellent updates!
Why can’t we all just get along? 🙂
Good updates. Interesting story so far, with a lot more to come I’m sure.
Because the US regime operates as a terrorist proxy of the Zionist entity- simple as
Another racist
How’s the weather in Tel Aviv, Moishe?
Anti-Semitic racists think the world is out to get them. And they think everyone is jewish apparently.
How much do you all want to bet that this guy Ken has something to do with this, the at&t bombing was the C&C center cover up and that it wasn’t the Russians but the CCP, as all of the sudden they offer help with this but can’t say how many dead are from covid.
assumption, after reading the US China Trump Xi deal word by word it’s like a parent (US)tells their 15 year old teen “Pleas be better and play fair and buy random stuff worth $200B -I would be insulted.
Given the reality on the ground the CCP is planing to make the conflict between the US and RU stronger and stronger. as they have realized that drawing lines on maps and playing with sand at the sea is pointless what they really need is Siberia which is already theirs by just the amount of chines people whom live there and the % of the local RU federations budget that is filled via bribes and taxes on clear cutting everything.
To those that live in lala land russia is starting to boil and if there is a civil war there the CCP will move on them.
Kenneth Hao is chairman of Silver Lake, a private equity firm specializing in making technology investments.
Silver Lake is based in Silicon Valley and oversees $43 billion.
Hao joined Silver Lake in 2000. He previously worked as a managing director at Hambrecht & Quist.
Hao led Silver Lake’s expansion into Asia, opening offices in China and Japan and leading Silver Lake’s profitable investment in Alibaba Group.
Hao is a director of ServiceMax, SMART Global Holdings, SolarWinds and Symantec, and previously served on the board of Broadcom.
Ken if you are not compromised and not a agent of the ccp, I am sorry your resume just looks perfect for this kind of multi decade operation and your new boards..
Isn’t this called Sinkholing?
Typical sinkholes are either just a dead end or provide tracking and statistics to whomever owns the domain. This sinkhole has the added ability to respond back and attempt to neutralize a portion of the infection. I could be wrong in how the actual neutralization/kill switch is implemented. If could be a simple a response from the sinkhole or it could be interacting with the infected hosts. This man not have been the intended use of the domain by the threat actors.
The domain used as a kill switch for WannaCry was built into the package by the threat actors, which is now sinkholed.
Thanks for the details, pointing out the difference and adding the Wanacry reference. A sinkhole with a kill switch. Nice.
I haven’t dug very deep into the code yet, but there was a part that looked like pretty standard anti-forensics: if the C2 resolves to a private network address, exit(). Since we tend to intercept all DNS queries and have them resolve to our own machines when analyzing malware (sandboxes too), it’s quite common for malware to try and detect us that way, and refuse to run.
So if I understood that part correctly, a wildcard entry returning 10.0.0.1 for all names should be enough to put the backdoor into hibernation, basically.
Correct concept, incorrect conclusion. Since the infected systems were all on internal corp networks the private IP was expected but needed to be translated on the outer routers via NAT.
Look at the code again and you may find the real IP(s) that is causing the self destruct
The Twitter thread has a decoded list. They aren’t all proper domain names.
https://pastebin.com/raw/6NukuxBN
The py code is fully commented…in Chinese characters!
很好！
Link to twitter thread please
A story/compromise that will keep on giving for many, many weeks! The level of complexity plus the intrigue of greed! Thank you Brian!
Why do you suppose they did this?
https://insiderpaper.com/dominion-voting-deleted-solarwinds-link-website/
was Serv-U an impacted product? Last I checked, it was not.
Dominion hasn’t issued a statement, there’s no sign Krebs has asked for a statement. So I guess we’ll never know. Oh well, good thing Dominion isn’t important enough to ask any tough questions!
Why is a statement needed for stuff we already know? Or do you want a denial before anyone even accuses them?
Dominion, it turns out, DID issue a statement claiming it never used Orion. As to why they should issue such a statement, isn’t it obvious? Wasn’t there just an election during the time the hack was ongoing? Doesn’t everyone know there was an election in the US?
I dont think that Serv-U is an impacted SW product?
Serv-U FTP was not impacted. Nothing to do with Orion.
Looks like the Dominion IT guys didn’t want people screenshotting and making unfounded accusations. Didn’t know they were web crawled.
Sure makes them look guilty of something.
Maybe. Maybe not.
The type of people looking at Dominion right now are after blood and are looking for anything to validate their prejudice.
It would have looked guilty to them either way.
Oh, has Dominion issued a statement regarding SolarWinds? I was not aware (I can’t find it anywhere).
Statement was made by Solarwinds. This product was not affected.
https://www.solarwinds.com/securityadvisory
Common IT Challenges
Network Trust and Malware
IT needs to ensure that users and devices can safely connect to the Internet, regardless of where they are connecting from, without the complexity associated with legacy approaches. Additionally, IT needs to proactively identify, block, and mitigate targeted threats such as malware, ransomware, phishing, DNS data exfiltration, and advanced zero-day attacks for users. Zero Trust security can improve your security posture while reducing risk of malware.
Secure Application Access
Traditional access technologies, like VPN, rely on antiquated trust principles, which has resulted in compromised user credentials which have led to breaches. IT needs to rethink its access model and technologies to ensure the business is secure, while still enabling fast and simple access for all users (including 3rd party users). Zero Trust security can reduce risk and complexity, while delivering a consistent user experience.
Complexity and IT Resources
Enterprise access and security is complex and constantly changing. Traditional enterprise technologies are complex and making changes often takes days (and often across many hardware and software components) using valuable resources. A Zero Trust security model can reduce FTE hours and architectural complexity.
Did Dominion use Orion?
No.
The scope and scale of this attack is huge. But we should still avoid such assumptions that implicate such a wide net of people/organizations, just because they are a political target.
Why aren’t you asking about Deibold, ES&S and Hart voting systems? Vulnerabilities were also found in those systems over the years.
A question isn’t an assumption. Diebold was probably more secure. Anyway, Dominion issued a statement making clear they didn’t use Orion at all, ever. Or at least so they say, it’s not like anybody is allowed to check, or like anyone cares, so long as they like the outcome, the means by which they got it.
Don’t your arms ever get tired carrying all that water for Dominion?
It was a simple question, with an easy answer.
No, as much as you try to fabricate a connection… This massive security breach has nothing to do with Trump losing the election.
I would delete a link to Solarwinds if I had one on a website. Solarwinds is a victim here though. I would restore a link at an appropriate time, but I fear Solarwinds may not survive.
I no longer retell Bill Cosby jokes either, for the same reasons. I do not consider Bill Cosby to be a victim. I cannot envision ever retelling his jokes. I haven’t seen his shows in television rerun listings, why do you suppose they don’t rerun his shows?
Will this create renewed focus to fortify blockchain in the software development change management and delivery processes?
Not likely.
I’d be interested in Brian’s take on: 1. Is there a set of “best practices” that could dramatically cut down on these kinds of attacks?
2. Any possibility of “best practices” being mandated for say publicly traded companies or something similar?
Thanks in advance,
Roger
The tl;dr version: Its all over the place, but hard (and expensive).
For starters, “good” or “better” practices, maybe not quite “best,” are available at no cost and relatively easy to find from NIST: (https://www.nist.gov/) or DOD (https://public.cyber.mil/). The DOD site has an option to login with a DOD Common Access Card, that would be available to DOD employees and contractors and allow access to additional material classified “for official use only.”
The problem with this material is that although it is comprehensive and free to obtain, it is a lot of work, at serious hourly rates, to implement. (Especially for those who haven’t yet paid the price for a serious compromise). Moreover, the processes and configuration standards they describe require constant attention, again at high rates, maintain compliance, and they require things that sometimes (often) are inconvenient or that employees do not like for other reasons; BYOD comes to mind as something many – other than the Information Assurance staff – are likely to want but have a lot of potential security fallout.
And then, in any case, there is the malicious administrator – Ed Snowden, for example – who can reduce to rubble even the best security program, or a distracted one who overlooks a small, but critical task or makes a mistake and leaves an exploitable vulnerability.
In the end, perfect security is unattainable, and even the best “best practices” can do no more than reduce the fault probability and limit the effects of those which will occur.
Nicely posed. I suppose the whole mindset that thinks a wall is a wall does not think about holes. And the drainage thereof. But shouldn’t we always consider the fact that any guard is doomed to failure, sooner or later? Rather, count on someone getting in some day and have secondary controls. Or more.
The concept of “Defense in Depth” covers this quite nicely.
I honestly believe this notion of “best practices” is a major contributor to the poor state of security in many corporations and their difficulties detecting a breach with anything less than the intruder blowing the entire infrastructure to pieces.
The closest thing to a “best practice” I can recommend is; add a post for security to the monthly operational budget for the IT department. Security needs to be seen as an operational cost, not a “sometimes” investment or a “project” with an end date.
Have a look at OSSTMM (at isecom.org). Or well, read it, and then consider how long it would take the current technicians to perform a full assessment if they had to do that on top of their current workload. During that time, vulnerabilities will continuously be introduced in the environment, and some of them may remain unknown until the next audit, so there’s quite a large window for a threat actor to use it 🙂 So yeah, that is what it would take. Increased costs in order to have resources dedicated to security.
I bet SolarWinds would happily pay the money they would have spent if they had hired a few security people and paid to fix any issues they found, like ten years ago, rather than what this incident will cost them 🙂
Something tells me best practices would not have caught this one. They definitely didn’t leave a port open, or use admin/admin.
You have not defined “these kinds of attacks” well enough in the context of the article. For supply chain attacks in general, there is a lot already being done to validate suppliaers, and any improvements would be a significant and long discussion out of the scope of a comment reply.
For being more resistant to attacks on your supply chain, there is more that can be done. I’d hesitate to call them best practices, because they are hard, and most organizations do not have the resources to execute on them.
For instance, if your network monitoring tools needs any access to the Internet, it should be restricted by domain name or IP address to only that which is needed. That sounds easy, but it is resource intensive to manage because IT constantly changes, and good security people are in short supply.
You could load vendor software in a test environment first and monitor it for a while. This particular code had a delay built in. How long do you wait?
As far as the US Government goes, some of this is going to be most efficient if done centrally. So which agency is going to validate COTS so that the rest of the federal government can use it and determine what those firewall rules should look like so that auditors can check against a baseline?
A more stringent or robust configuration change management process at Solarwinds could have caught this at the source, by detecting an unauthorized change to the software update prior to it being released. Beyond that, their process for access to their code signing certificates would also likely need hardening.
For the customers downstream, it would have been extremely hard to detect as the updates were obtained directly from the vendor and were digitally signed. You were essentially victimized by doing the right thing, patching.
Supply chain security has been a focal point from the federal/federal contractor space for at least the last couple of years. It’s difficult, it can be messy and cumbersome but this disaster scenario is exactly what it’s meant to avoid.
Whether the trojan was inserted by an unauthorized or authorized user and other surrounding circumstances are not yet public information, as far as I am aware. If done by an authorized user, it also is not clear whether it was done in connection with one or more authorized changes.
In addition, I have seen no mention of the possibility that it entered the SolarWinds by way their supply chain.
It seems a bit early to be either assigning blame or prescribing corrections.
I did say *could* have been caught, you’re right in that the details aren’t public knowledge yet. One would assume the initial breach into SolarWinds was by some other vector, but they (being the supply chain in this case) were leveraged as a means to infect their customers downstream.
No… SolarWind just uploaded the password of their update server to a public GitHub repository as mentioned in an earlier post on this hack on this site.
Don’t create homogeneous computing networks that have little to no diversity (all windows AD domain joined PCs and servers). Don’t install ‘management compliance’ systems that have complete access and control of all nodes on a network. Seek diversity (Mac, Linux, Windows, FreeBSD) and segment everything,
Best practice is to keep your security tools updated
Ooops, that didn’t work in this case
There’s some IT managers saying, glad I never got around to that SolarWinds update this year
Always count this site for updates that contain factual information. Thanks BK!
The second thing at Christmas that’s such a pain to me
“Rigging up the internet ”
And reading a yet another cybercrime security article
Does anyone know if we are even trying to stop these attacks at thier source (Russia)?? Is our government even communicating with the Russians about this?!
Even if there was any irrefutable evidence that their intelligence service did this, what do you think Putin would say? He’d be singing “Wasn’t me” to the tune of “Can’t touch this”, and the only thing the US could do about it would be to have a stern discussion with the russian ambassador. And possibly retaliate by having their “cyber warriors” destroy some oil refinery or whatever.
I mean, don’t think for a minute that there aren’t hundreds of implants on russian networks phoning home to american-owned infrastructure
as much as I like that these companies collaborate to try and fix this mess,
it is quite scary to see that these companies can take over domains without a proper process by authorities or any court orders.
Domains aren’t like other physical parcels of property. Domains are often owned by 3rd parties, then leased out, and they are the only authorities. GoDaddy doesn’t need court orders to rescind what is ultimately a lease when violations of agreements are pretty obvious.
Also, domain names are just like entries in a phonebook. “Taking over a domain” sounds like it’s an invasion or some night raid on a compound… but it’s literally just telling DNS servers to return a different IP address. It’s more like a change of address than a raid.
Well put
Actually, I believe that GoDaddy is acting as an agent for ICAAN. See https://www.icann.org/en/system/files/files/guidance-domain-seizures-07mar12-en.pdf GoDaddy may be the registrar but they are not the owner of the domain name. In a case like this, a court order (or seizure order) could probably be obtained in a few minutes. Requiring a court order sounds reasonable to me.
In the spirit of considering all angles. There is a faction of the US government that would like to break up the entrenched bureaucracy. Maybe a group wanted to see who was playing cards under the table. Just saying…we have the motive to do this to ourselves.
What I am wondering is what other digitally signed malware this hacking group could have created? Whilst they had the access to a huge range of government and private organisations over the last 9 months from the initial attack.
Any software company using the compromised version of Orion could potentially have released further compromised software updates.
Only way to know is to check the code the release code line by line for every single company affected…
bingo
…where is there reliable info on how they got into solar winds?…
…al I’ve seen is speculation, rumor, “I don’t like broccoli”, etc…
Then you didn’t read the links in the article. Try that.
…it only says what malware was distributed, not how they got into sw…
Referenced from a prior Krebs article:
https://twitter.com/briankrebs/status/1338549067924770817
They did it to themselves.J
We don’t know yet if this was the point of intrusion.
Sometimes companies have multiple failures in security. Actually, quite often.
There are no coincidences. Between the servers confiscated by Mil spec ops in at the US consulate/farm in Frankfurt last month and this SolarWinds breach in critical US agencies, it is obvious we are in the middle of unprecedented cyberwarfare.
…doh…
…we have been for over 30 years now…
…did you ever read Cuckoos Egg?…
Yes, we’ve been in a cyber war for a while now.
But Frankfurt was fake news.
https://apnews.com/article/fact-checking-9754011363
contact me for logs. My whole house got hacked. Was able to obtain information and it still active. I have mac and several ubuntu and win10 full of scripting and logs I was able to grab. To note my phone has bootstrap. I was able to stop services and several. It is just waiting to reboot to start networking again and my ford wifi was hacked. Long list but I have several ip’s from router logs and firmware added to routers and win10 bios etc.
contact me for logs. My whole house got hacked. Was able to obtain information and it still active. I have mac and several ubuntu and win10 full of scripting and logs I was able to grab. To note my phone has bootstrap. I was able to stop services and several. It is just waiting to reboot to start networking again and my ford wifi was hacked. Long list but I have several ip’s from router logs and firmware added to routers and win10 bios etc.
Have you contacted Putin to demand he cease forthwith to avoid the most dire consequences?
I can see “vms.ad.varian” in the list. Is it Varian Medical Systems ? They have full access to every radiotherapy medical devices they sold around the world.
You can try again and will see that for sure!
Future of USA not Looking Good.
USA own debt to Germany one day the Germany will call in the debt, You Can’t pay youll Work for it.
The debt has to be payed one day You Can’t borrow and spend it endlessly the Germany will make USA pay soon
Accountant St Neots? Hey check this out helpful blog,Accountant Cambridgeshire
Actual, not rhetorical, question here: does the success of this attack tarnish the reputation/legacy of Chris Krebs, fired by POTUS DT? Was defense against this sort of attack his bailiwick?
CISA is a new agency, Trump created. The Trump White House also dismantled a lot that Obama put in place to prevent this. DHS has other cyber protection teams who also failed. NSA didn’t catch this either. It was everyone’s job.
But yes, NSA and CISA should have been one of many eyes that should have seen something. It is a combination our failure, and their prowess. A nation-state attack like this, with Supply Chain infiltration… one of the hardest to catch without the benefit of hindsight.
Thanks. Good info.
On Tuesday, the National Security Council announced the administration would be invoking Presidential Policy Directive-41, or PPD-41, which “facilitates continuous and comprehensive coordination for whole-of-government efforts to identify, mitigate, remediate and respond to this incident,” according to a statement from NSC spokesperson John Ullyot.
The directive, signed in the last years of then-President Barack Obama’s administration, creates a chain of command for responding to cyber incidents. The directive states the White House Cybersecurity Coordinator—or “an equivalent successor”—will serve as the chair for a Cyber Response Group to develop a strategy while the Cyber Unified Coordination Group will coordinate between federal agencies.
Within the CUCG, the directive designates the FBI and the National Cyber Investigative Joint Task Force, the Department of Homeland Security’s National Cybersecurity and Communications Integration Center, and the Office of the Director of National Intelligence’s Cyber Threat Intelligence Integration Center as the lead agencies for certain types of responses. Agencies, however, maintain operational control over their networks, “unless mutually agreed upon by agency heads or their designees.”
Since the policy was signed, a few organizational changes have occurred. In 2018, then-National Security Adviser John Bolton eliminated the White House Cybersecurity Coordinator position and the Cybersecurity and Infrastructure Security Agency—previously known as National Protection and Programs Directorate—became a standalone agency.
The Office of the Director of National Intelligence confirmed they would be joining the Cyber Unified Coordination Group, as outlined in PPD-41.
…completely useless, and incompetent at that…
…the failure to detect the attack is the issue and this group can’t do that…
…Einstein, the last attempt to do detection, was a complete failure…
I’ve been dealing with CIS for months now thanks to the 2020 vote. As much as I respect the organization they keep hounding me for specifics about my network in the name of understanding attack vectors… If MDBR gets hacked, which seems perfectly reasonable now, what do I do to project my data?
Does ANYONE still think online or electronic voting is a good idea?
No.
And we’ve actually been moving away from electronic voting.
https://verifiedvoting.org/verifier/#mode/navigate/map/ppEquip/mapType/normal/year/2020
Thanks for the inforamtion
It’s a BASIC things to envolve.
So the Orion application somehow had admin privileges and could stop/start any Windows service? Seems like a ring 0 type issue or do I not understand?
Thanks,
Thanks for the information
A bit ironic that Trump admin unwilling to name Russia for anything,uncluding his election. After capital storm I wouldn’t be surprised if Trump held open the “Front door” for SolarWinds. If it benefited Trump financially, there’s nothing he wouldnt do.