A faulty software update from cybersecurity vendor Crowdstrike crippled countless Microsoft Windows computers across the globe today, disrupting everything from airline travel and financial institutions to hospitals and businesses online. Crowdstrike said a fix has been deployed, but experts say the recovery from this outage could take some time, as Crowdstrike’s solution needs to be applied manually on a per-machine basis.
A photo taken at San Jose International Airport today shows the dreaded Microsoft “Blue Screen of Death” across the board. Credit: Twitter.com/adamdubya1990
Earlier today, an errant update shipped by Crowdstrike began causing Windows machines running the software to display the dreaded “Blue Screen of Death,” rendering those systems temporarily unusable. Like most security software, Crowdstrike requires deep hooks into the Windows operating system to fend off digital intruders, and in that environment a tiny coding error can quickly lead to catastrophic outcomes.
In a post on Twitter/X, Crowdstrike CEO George Kurtz said an update to correct the coding mistake has been shipped, and that Mac and Linux systems are not affected.
“This is not a security incident or cyberattack,” Kurtz said on Twitter, echoing a written statement by Crowdstrike. “The issue has been identified, isolated and a fix has been deployed.”
Posting to Twitter/X, the director of Crowdstrike’s threat hunting operations said the fix involves booting Windows into Safe Mode or the Windows Recovery Environment (Windows RE), deleting the file “C-00000291*.sys” and then restarting the machine.
The software snafu may have been compounded by a recent series of outages involving Microsoft’s Azure cloud services, The New York Times reports, although it remains unclear whether those Azure problems are at all related to the bad Crowdstrike update. Update, 4:03 p.m. ET: Microsoft reports the Azure problems today were unrelated to the bad Crowdstrike update.
A reader shared this photo taken earlier today at Denver International Airport. Credit: Twitter.com/jterryy07
Matt Burgess at Wired writes that within health care and emergency services, various medical providers around the world have reported issues with their Windows-linked systems, sharing news on social media or their own websites.
“The US Emergency Alert System, which issues hurricane warnings, said that there had been various 911 outages in a number of states,” Burgess wrote. “Germany’s University Hospital Schleswig-Holstein said it was canceling some nonurgent surgeries at two locations. In Israel, more than a dozen hospitals have been impacted, as well as pharmacies, with reports saying ambulances have been rerouted to nonimpacted medical organizations.”
In the United Kingdom, NHS England has confirmed that appointment and patient record systems have been impacted by the outages.
“One hospital has declared a ‘critical’ incident after a third-party IT system it used was impacted,” Wired reports. “Also in the country, train operators have said there are delays across the network, with multiple companies being impacted.”
Reactions to today’s outage were swift and brutal on social media, which was flooded with images of people at airports surrounded by computer screens displaying the Microsoft blue screen error. Many Twitter/X users chided the Crowdstrike CEO for failing to apologize for the massively disruptive event, while others noted that doing so could expose the company to lawsuits.
Meanwhile, the international Windows outage quickly became the most talked-about subject on Twitter/X, whose artificial intelligence bots collated a series of parody posts from cybersecurity professionals pretending to be on their first week of work at Crowdstrike. Incredibly,Twitter/X’s AI summarized these sarcastic posts into a sunny, can-do story about Crowdstrike that was promoted as the top discussion on Twitter this morning.
“Several individuals have recently started working at the cybersecurity firm Crowdstrike and have expressed their excitement and pride in their new roles,” the AI summary read. “They have shared their experiences of pushing code to production on their first day and are looking forward to positive outcomes in their work.”
The top story today on Twitter/X, as brilliantly summarized by X’s AI bots.
This is an evolving story. Stay tuned for updates.
thanks for updating us!
Hi Brian — Just wanted to let you know you have a typo in the title of this article. It should be crowdstrike, not crowstrike.
Actually it’s clownstrike today.
They’ve had better days for sure
Muphry’s Law!
i dunno, it might be “Crowstrike” after today.
The fix is easy if the computer is not encrypted with BitLocker and SecureBoot.
Otherwise, IT needs to visit each machine to enter the recovery key. Try that on a global scale with remote IT shops and WFH users.
And I’m sure Crowdstrike is going to put up a grand total of zero dollars from their billions in annual revenue towards labor, travel, etc to their clients…
even 100 million would be enough to make everybody whole
Not easy at all. If the box is a part of corporate network and users don’t have admin privileges (and they usually don’t) – then bad luck, the users can’t boot in safe mode. Admins have to unlock every box manually.
Can someone explain how a CrowdStrike agent update could potentially bring down Microsoft’s systems? I understand if you’re using the agent and have deployed it in your environment as your EDR, but how does that impact Microsoft itself? Does Microsoft use CrowdStrike for EDR? Please explain.
The update that CrowdStrike (CS) pushed caused the CS driver to crash which resulted in the BSOD.
Hooked into the kernel. Kernel drive update can easily cause havoc.
Can someone explain how a CrowdStrike agent update could potentially bring down Microsoft’s systems? I understand if you’re using the agent and have deployed it in your environment as your EDR, but how does that impact Microsoft itself? Does Microsoft use CrowdStrike for EDR? Please explain….
Maybe a subcontractor?
Strike that. Microsoft have now clarified (https://azure.status.microsoft/en-us/status) that the Azure outage (Tracking Id: 1K80-N_8) was not related to the CrowdStrike incident.
(However, some Azure customers are running CrowdStrike on their Azure VMs and those are of course affected.)
You give almost enough information for a fix, but you didn’t mention the faulty file lives in
C:\Windows\System32\drivers\CrowdStrike
Good write-up otherwise!
The square pegs goes into the square, and the cirle goes into the circle.
As an IT professional, I feel the pain of every corporate IT worker dealing with this today. We dodged this one where I work today, but it could have just as easily been some other information security tool that pushed a bad update, and we could have been hit.
This is yet another example of how supply chain attacks can be so effective, even though this one was not due to malicious intent.
Uh..always test on in a lab environment first, then a tiny subset of production before sending an update to all systems. I’m blown away at this widespread screw up. Glad I retired 8 years ago and use Mac at home. Windows is a disaster
Similar but went Chromebook. I don’t have the time or patience to maintain a windows machine.
And do it on a weekend.
Imagine if it was. All of our eggs are in one weak, porous, easily damaged basket being carried by someone who stumbles often…
Yeah, no kidding, especially the people at Crowdstrike. — Even “good” updates can create surprises. So many times after pushing an update I would get the phone call,
User: “Yeah, I can’t do [something] anymore after the update.”
Me: “Why are you doing [something]? The app isn’t designed to do that.”
User: “We know, but we have to do [something] and we figured out how to do it with the app.”
Me: “We didn’t know you had to do that.”
User: “So and so told us we need to do it and we figured out how to do it and now we can’t”
SIGH……. [EYEROLL]
Grok doesn’t know how to use a semicolon.
This is yet another example of how supply chain attacks can be so effective, even though this one was not due to malicious intent.
Tthat we know of. I mean the comapny name isss “Crowdstrike”, right?
here’s a grain of salt tip for all current IT “pros” currently working in the corporate world…
Find a way to secure your powershell, DFS, gpolicy, and lock down all remote access software progs any and all auto update os software or third party software. If you don’t need it, remove it.
Good luck, anyone on the front lines of IT. If the c-suites won’t listen to your warnings about attack vectors of infection, potential data leaks and you have presented it to them in writting and in person and they still don’t listen.. you hands are clean and the fault lies with them.
Yo yosemite, with auto updates turned off, how do you solve for the ever constant need to keep modern browsers fully patched. And browsers are just one of several hundred different apps most enterprises are running today. I’m not in the camp of “turn off all auto updates”…there are scenarios when auto updates is the best approach. Clearly, all security supporting tools (Clownstrike clearly falls into this category) need auto updates on just to be effective and providing the security they are intended to provide. Also there are no clean hands of any front line at workers today because they are all busy running from machine to machine, I don’t appreciate the clean hands metaphor, it’s disrespectful to level 1s as they are ones cleaning up the mess…not senior leadership. Yes to auto updates & give kudos to your IT field tech when they show up!
read the entire post carefully. Zero trust, zero access is your friend.
Just by having the computer offline it’s still doing background ppi capture and side channel survies.
Please do more research and testing before replying, your showing lack of attention to detail.
You can’t block autoupdates of winoows, even with corporate edition and policy. Autoupdater process never stops so they can push anything if they want so. And you can’t also block all MS IPs on firewall as Windows tests connectivity using an URL which hosted on one of these IPs. Nlocked IPs – no internet for you as teh interface will be marked as “no connectivity”.
Um why wasnt the update first tested an vetted prior to being released everywhere?
We’ll make sure to test the update in a test environment next time. Sorry about the issue today, glad you’re back up and running 🙂 /s
That would require hiring a QA person. This is a dev shop, we don’t need no stink’n QA.
Why have nation-state hackers when we can just do it ourselves? /s
The email said Crowstrike.
It even shut down the McDonalds in Tokyo.
The Crowdstrike CEO might have claimed too quickly, “This is not a security incident or cyberattack.” Crowdstrike clearly has a vulnerability in their release process (not doing sufficient testing), and an attacker inserted malicious code into one of their released files. How does the CEO know this was not deliberate?
Exactly. I cannot imagine CS’s QA team did not test the update on some Windows PCs before pushing to production.
This was not a code change. A lot of dev shops only QA codes changes. This was a “content update”, thus no QA.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching “C-00000291*.sys”
4. Boot normally
Except if you have an encrypted drive, then seek help. And have a stiff drink.
How do they propose to do this remotely at scale? Are we talking cloud servers, individual appliances/client devices or both? What a way to rock the world of IT on a Friday, LOL!
GL if you remote support 5k computers and they are all bitlockered. they will be dealign with reverting this for weeks
I just put in my resignation I was looking at another job anyway and about to give my 2 weeks next week.
I know that when you resign at my work they walk you out immediately no matter what xD
best wishes to the other 8 members of my team.
I’m sure like everyone else I’m wondering how this could’ve been pushed so far before it was realized; we have early release sensors so would’ve expected those lower risk endpoints to be impacted, not the others. Am I misunderstanding how a content update works?
If you don’t understand my definition …
“Software Updates: trading known problems for unknown problems.”
… then you’re doing it wrong and may be “in for some hurt:”
For all of you IT folks out there trying to remediate this Clownstrike screw-up — just keep telling yourself this is job security for you!!!
How is it even possible that this bug was not found during testing. I dont believe the innocence in a small human error. The know how deep tgeir software ties into windows kernel, it should have never happened in the open. To be continued?
DIDN’T ANYONE CHECK TO SEE IF THE UPDATE WORKED ?
ARE THE GOD LIKE EGOS THAT BIG ?
I’ve been trying to get help for months via MS and then their arbitration dept and many others. Now I’ve finally got a local DA investigator to come out since nobody is really equipped. In my case I forced this MS System screwup from the hard drive then unplugged it before it could jump back in. Long story. All hell broke loose. Still is.
Crowdstrike is known for their incident response on high profile computer intrusions. I would think they would be very embarrassed for causing more damage due to their mistake than the bad actors they try to uncover. There credibility going forward should be questioned.
If:
a. You’re running Windows for critical systems
and
b. You’ve given a third party full access to update kernel drivers dynamically
then
you deserve everything thats happening to you.
I’m waiting for the sequel: The Crowd Strikes Back.
Please explain the diversity of experirnces. Obviously all servers have not been impacted. Why so?
I feel for Crowdstrike, so here’s my rant to the critics. Of the various reactions, this business of apology-extortion infuriates me (rooted in overzealousness/fear I see most of the time), because even giving an apology is usually not enough. Most of the overzealous commenters, claiming to be engineers and jumping to conclusions about the competencies at Crowdstrike (or even free software like log4j for example when I recall), are just glorified tool users and technicians and wanna-be engineers or wanna-be scientists. The reality is that a bunch of driven individuals drive the STEM space with discoveries and inventions, and the rest use their efforts, and behave like entitled customers. We are all peers – you had the option to develop every tool you used, but you can’t/won’t. No one forced you to use suppliers, except your own incapability/infeasibility to create said tooling. Most critics of any tool maker are incapable of even thinking about the idea, let alone building it. One can go on about the responsibilities of Crowdstrike, but no one forced the users to increase their dependency on 3rd parties! Write your own OS, language, any core software. Build your own hardware like Alan Kay/Jobs said. But obviously you can’t do it, so your functioning will always be owned by a 3rd party, and it’s not their fault. “Engineering” (used so rhetorically) includes limits, and there are limits to how many variables a system or a process can be designed for. The more global the supplier, imagine the bloated system with variables involving the whims and fancies of every overzealous security professional – it will collapse as it should. Do your own “engineering” from the scratch and don’t outsource any function, if apologies are what one wants to extort out of suppliers, as if they woke up that day wanting to cause it. I am using rhetoric to beat rhetoric – if we all rely on each others’ creation – stop these games of finger-pointing, ego massaging via apology-extortion, finding the head for the chopping block and other BS games. Either way, now that he said sorry, that word isn’t going to go and delete that .sys file, right?! Or are the apology-extorters going to write an apology-instrumentation tool, that magically uses the apology to delete the .sys file?
Yeah, and we should all grow our own food too. Your “logic” is impeccably ridiculous.
The logic is to grow your own food, if one is arrogant enough to extort apologies, and not recognise the interconnected nature of human society.
If one is arrogant enough to extort apologies, then yes. Else, have some shame, and let them work on fixing this.
To add to my comment, it’s a go at overzealousness around the internet on this incident, and not specifically just the commenters on this site.
Somehow Trump is behind this whole thing