A faulty software update from cybersecurity vendor Crowdstrike crippled countless Microsoft Windows computers across the globe today, disrupting everything from airline travel and financial institutions to hospitals and businesses online. Crowdstrike said a fix has been deployed, but experts say the recovery from this outage could take some time, as Crowdstrike’s solution needs to be applied manually on a per-machine basis.
A photo taken at San Jose International Airport today shows the dreaded Microsoft “Blue Screen of Death” across the board. Credit: Twitter.com/adamdubya1990
Earlier today, an errant update shipped by Crowdstrike began causing Windows machines running the software to display the dreaded “Blue Screen of Death,” rendering those systems temporarily unusable. Like most security software, Crowdstrike requires deep hooks into the Windows operating system to fend off digital intruders, and in that environment a tiny coding error can quickly lead to catastrophic outcomes.
In a post on Twitter/X, Crowdstrike CEO George Kurtz said an update to correct the coding mistake has been shipped, and that Mac and Linux systems are not affected.
“This is not a security incident or cyberattack,” Kurtz said on Twitter, echoing a written statement by Crowdstrike. “The issue has been identified, isolated and a fix has been deployed.”
Posting to Twitter/X, the director of Crowdstrike’s threat hunting operations said the fix involves booting Windows into Safe Mode or the Windows Recovery Environment (Windows RE), deleting the file “C-00000291*.sys” and then restarting the machine.
The software snafu may have been compounded by a recent series of outages involving Microsoft’s Azure cloud services, The New York Times reports, although it remains unclear whether those Azure problems are at all related to the bad Crowdstrike update.
A reader shared this photo taken earlier today at Denver International Airport. Credit: Twitter.com/jterryy07
Reactions to today’s outage were swift and brutal on social media, which was flooded with images of people at airports surrounded by computer screens displaying the Microsoft blue screen error. Many Twitter/X users chided the Crowdstrike CEO for failing to apologize for the massively disruptive event, while others noted that doing so could expose the company to lawsuits.
Meanwhile, the international Windows outage quickly became the most talked-about subject on Twitter/X, whose artificial intelligence bots collated a series of parody posts from cybersecurity professionals pretending to be on their first week of work at Crowdstrike. Incredibly,Twitter/X’s AI summarized these sarcastic posts into a sunny, can-do story about Crowdstrike that was promoted as the top discussion on Twitter this morning.
“Several individuals have recently started working at the cybersecurity firm Crowdstrike and have expressed their excitement and pride in their new roles,” the AI summary read. “They have shared their experiences of pushing code to production on their first day and are looking forward to positive outcomes in their work.”
The top story today on Twitter/X, as brilliantly summarized by X’s AI bots.
Matt Burgess at Wired writes that within health care and emergency services, various medical providers around the world have reported issues with their Windows-linked systems, sharing news on social media or their own websites.
“The US Emergency Alert System, which issues hurricane warnings, said that there had been various 911 outages in a number of states,” Burgess wrote. “Germany’s University Hospital Schleswig-Holstein said it was canceling some nonurgent surgeries at two locations. In Israel, more than a dozen hospitals have been impacted, as well as pharmacies, with reports saying ambulances have been rerouted to nonimpacted medical organizations.”
In the United Kingdom, NHS England has confirmed that appointment and patient record systems have been impacted by the outages.
“One hospital has declared a ‘critical’ incident after a third-party IT system it used was impacted,” Wired reports. “Also in the country, train operators have said there are delays across the network, with multiple companies being impacted.”
This is an evolving story. Stay tuned for updates.
thanks for updating us!
Hi Brian — Just wanted to let you know you have a typo in the title of this article. It should be crowdstrike, not crowstrike.
Actually it’s clownstrike today.
Muphry’s Law!
The fix is easy if the computer is not encrypted with BitLocker and SecureBoot.
Otherwise, IT needs to visit each machine to enter the recovery key. Try that on a global scale with remote IT shops and WFH users.
Can someone explain how a CrowdStrike agent update could potentially bring down Microsoft’s systems? I understand if you’re using the agent and have deployed it in your environment as your EDR, but how does that impact Microsoft itself? Does Microsoft use CrowdStrike for EDR? Please explain.
The update that CrowdStrike (CS) pushed caused the CS driver to crash which resulted in the BSOD.
Hooked into the kernel. Kernel drive update can easily cause havoc.
Can someone explain how a CrowdStrike agent update could potentially bring down Microsoft’s systems? I understand if you’re using the agent and have deployed it in your environment as your EDR, but how does that impact Microsoft itself? Does Microsoft use CrowdStrike for EDR? Please explain….
You give almost enough information for a fix, but you didn’t mention the faulty file lives in
C:\Windows\System32\drivers\CrowdStrike
Good write-up otherwise!
The square pegs goes into the square, and the cirle goes into the circle.
As an IT professional, I feel the pain of every corporate IT worker dealing with this today. We dodged this one where I work today, but it could have just as easily been some other information security tool that pushed a bad update, and we could have been hit.
This is yet another example of how supply chain attacks can be so effective, even though this one was not due to malicious intent.
Uh..always test on in a lab environment first, then a tiny subset of production before sending an update to all systems. I’m blown away at this widespread screw up. Glad I retired 8 years ago and use Mac at home. Windows is a disaster
Grok doesn’t know how to use a semicolon.
This is yet another example of how supply chain attacks can be so effective, even though this one was not due to malicious intent.
here’s a grain of salt tip for all current IT “pros” currently working in the corporate world…
Find a way to secure your powershell, DFS, gpolicy, and lock down all remote access software progs any and all auto update os software or third party software. If you don’t need it, remove it.
Good luck, anyone on the front lines of IT. If the c-suites won’t listen to your warnings about attack vectors of infection, potential data leaks and you have presented it to them in writting and in person and they still don’t listen.. you hands are clean and the fault lies with them.
Um why wasnt the update first tested an vetted prior to being released everywhere?
We’ll make sure to test the update in a test environment next time. Sorry about the issue today, glad you’re back up and running 🙂 /s
Why have nation-state hackers when we can just do it ourselves? /s
The email said Crowstrike.
It even shut down the McDonalds in Tokyo.
The Crowdstrike CEO might have claimed too quickly, “This is not a security incident or cyberattack.” Crowdstrike clearly has a vulnerability in their release process (not doing sufficient testing), and an attacker inserted malicious code into one of their released files. How does the CEO know this was not deliberate?
Exactly. I cannot imagine CS’s QA team did not test the update on some Windows PCs before pushing to production.