If you use Windows XP and haven’t yet updated your system with the applicable security updates that Microsoft issued Tuesday, you might want to hold off for a bit. Turns out, a non-trivial number of XP users are reporting that their systems suffer from the dreaded Blue Screen of Death (BSoD) and fall into an interminable reboot loop after installing the latest batch of patches from Redmond.
The problem seems to be affecting only some XP systems. This thread on a Microsoft.com answers forum seems to include a fix that works. However, the fix requires users to have their XP install CD handy (in a practice that should be outlawed, many computer makers get away with shipping systems without an install/reinstall disc)
According to the support forum threads I’ve seen on this, affected users noticed the problem on the reboot following the installation of Tuesday’s patch batch. The folks who complained of the bootup problem said the BSOD error page is accompanied by the message “PAGE_FAULT_IN_NONPAGED_AREA”.
If you’re experiencing the above-described problems after installing Tuesday’s bundle of updates, follow these steps, which a number of affected users have said seem to fix the problem:
1. Boot from your Windows XP CD or DVD and start the recovery console (see this link on how to use recovery console)
Once you are in the Repair Screen..
2. Type this command: CHDIR $NtUninstallKB977165$\spuninst
3. Type this command: BATCH spuninst.txt
4. Type this command: systemroot
5. When complete, type this command: exit
Unfortunately, there is an entire subset of users who might be in for a whole mess more work to fix this kind of problem: Netbook users. One of the things that makes netbooks so light and small is that they do not have optical (CD/DVD-ROM) drives. If you’re a netbook user who has this problem AND a copy of a Windows XP install CD handy and a computer with a CD drive, you may still be able to rescue your system by building a custom XP install/bootup disc on a USB drive.
If all of that sounds like too much work, home users are eligible for no-charge support by calling 1-866-PCSAFETY (and/or 1-866-234-6020 and/or 1-800-936-5700) in the United States and in Canada. Microsoft says there is no-charge for support calls that are associated with security updates.
Update, 8:34 a.m. ET: Based on a review of various help forums discussing this problem, it appears that the problematic update is KB977165 (MS010–15:Vulnerabilities in Windows kernel could allow elevation of privilege”). Note that systems experiencing a BSoD may do so or hang in Safe Mode when loading the system driver “mups.sys”.
The help instructions above have been modified to specify the removal of just this one patch. A previous version of this blog post included instructions for removing all of the patches Microsoft shipped for XP systems on Tuesday.
Update, Feb. 12, 10:09 a.m. ET: Microsoft has a blog post up acknowledging this problem, saying that it stopped shipping the problematic update via Windows Update as soon as it recognized the issue. Redmond says it is still investigating the cause of the conflict. Microsoft notes that in lieu of applying the patch, XP users can use Microsoft’s click+install “Fix it” tool, which disables the vulnerable Windows component. That workaround is available here.
Based on the thread it appears the KB977165 has been narrowed down as the culprit. So the rest of the patches are probably safe to install.
I have downloaded my patches for the month but have not yet installed them. What advice do you have for me?
From BK’s post above: “you might want to hold off for a bit. “
Well yeah, I got that I shouldn’t install the updates, but what do I do about this bum update that has been downloaded to my machine? Can I delete it? Will I receive the good update on top of this one when MS fixes it, or will Windows Update think I have it already? I think it was a fair question and that you were kind of snotty.
Bunny,
I was trying to be helpful, no snot intended.
Windows 7 Ultimate 32bit – stuck on Loadind screen after first restart. Repairing didnt help so thank God for the restore point before the patched application.
Hope we get some info soon
The recovery console and installation repair options both require a Win XP CD which, unfortunately, is no longer possible to get from OEMs as BK already mentioned. Between cost-cutting by the manufacturers and MS treating everyone like pirates, it’s unethical at best and criminal at worst to leave users with no real recourse to quickly recover from a botched update other than to wipe everything out and start over…all the worse if one hasn’t backed up recently or have dual-boot systems since Windows doesn’t like to share.
(And yes, you’d be surprised at how many people don’t regularly back up.)
Are MicroSD cards the possible replacement of lugging around DVDs & CDs with the reader.
It’s now possible to boot them as USB devices and they have a write protect to protect against being violated (assuming the reader supports it and its not a software thing that could be over written by a rootkit.)
Any feedback on MicroSD for large 50+ deployments/fixes?
It’s possible to buy OEM CDs on ebaY. I bought mine for like $16 and used it along with my prior computer’s reg key to load XP on my new system.
Not to blindly defend Micro$oft, but I’d bet most of the crashing machines have malware on them.
Hmmm…interesting theory, you may be on to something here. I’ve updated all (10) of my office clients as well as all all (3) of my home machines without a single problem.
The only machine I have not updated is my SBS 2003 server, that has Automatic Updates switched OFF just for reasons like this, and even though it is probably malware free, I will follow BK’s advice (from the last post) about waiting to update this one until the culprit is found.
All-in-all though, the thing to take away from this would be to WAIT a few days before installing major patches to see if there are issues like these
Apparently there is anecdotal support for the infected computer theory, posted recently on the MS Update support thread by Patrick W. Barnes:
“I had an Eee PC with XP Home brought to me with this same problem. I rolled back KB977165, rebooted and the system worked fine. I reapplied KB977165 and the rest of the updates available at Microsoft Update, and the problem returned. I replaced %System32%\drivers\atapi.sys with a clean version from a XP SP3 distribution folder and rebooted… voila! Problem solved.
For reference, the SHA1SUMs of the atapi.sys files:
Non-working:
bb3e36ad0c8ed6daab38653ea4a942d74b9f4ff6
Working:
a719156e8ad67456556a02c34e762944234e7a44
If anyone wants to look at the non-working atapi.sys:
https://patrickwbarnes.com/pub/atapi.sys
I will be looking at this more in-depth. If I find anything more, it will be posted in a follow-up comment at the ISC:
http://isc.sans.org/diary.html?storyid=8209
UPDATE :
I uploaded the non-working atapi.sys file to VirusTotal, and this is the result:
http://www.virustotal.com/analisis/85aa49f587f69f30560f02151af2900f3dc71d39d1357727ab41b11ef828a7ff-1265925529
Apparently, this update problem is the result of an infection.”
Good work Patrick and good work Brian for scooping everyone on this story.
I hate it when I’m right… ;^)
I think you are on to something, although folks won’t want to hear that it may be partly their own fault …
Yeah right, its always someone’s fault, just not microsoft’s.
About two weeks ago I purchased a laptop from hp. Upon arrival, it had so much crapware on it I normally re-install the system. But in today’s world, hp doesn’t ship computers with actual OS discs. I called and asked if they would send a Windows 7 disc to me, but they said they were all out. OK. In addition, if I wanted one I would have to purchase one for over $100.
Maybe these problems won’t happen in the future with Windows 7, but I’ll never know since I returned the hp and got yet another Mac!
Partly their own fault????
A friend of mine has been hit by this. And it does look like an infection in the atapi.sys is involved.
Not sure you could say it is her fault. She has full internet security suite installed from Bitdefender. Runs MalwareBytes weekly and SuperAntiSpyware weekly.
She does not visit extra-curricular websites. Has no risky behavior I could identify. Obviously there is something at work that let in the malware. But at this point it seems to me the industry is letting her down more than she is messing up.
To maintain a clean computer at this point is not in the ability of the average and above average computer user.
I’ve been running her through the gauntlet required to even ask a question on majorgeeks.com and it is beyond daunting for the average user.
Something has got to change. People can function like this.
The main problem with most users is they do NOT use a defense in depth strategy to properly secure their systems. Their primary and usually only defense is AV/Security software. The days of relying on just AV/Security software are long gone. In fact, I never thought it was ever a good strategy.
While I use AV software, my biggest defenses, besides a hardware firewall and then a software firewall on each system, are a blocking hosts file from MVPS (see link below) and running as a non-admin (limited user). The blocking hosts file is usually updated monthly and blocks a slew of known malicious websites. Besides the firewalls, I strongly believe the blocking hosts file alone has kept malicious stuff from even getting to my system, even as I’ve been to risky parts of the Internet (using IE none the less).
Of course, running as a non-admin will stop most malware in its tracks should it actually get to your system.
One other thing, NEVER use P2P file sharing software (ex. Limewire) as it will almost guarantee malware on your system.
Blocking hosts file:
http://www.mvps.org/winhelp2002/hosts.htm
While the computing industry has its part to play, end users also have theirs. Computers are powerful and complex tools. They are not appliances like a toaster, albeit, it’s still possible to get electrocuted by a toaster or to cause a fire by using it improperly. 🙂
The problem is that it’s not just any one AV package. And scan-behind doesn’t help. Java and Adobe product now must also be updated. Perhaps run Secunia also.
What a smug attitude you have! People who read and post in computer forums, including this one, are interested in security and are likely to be the most careful to keep their computers secure. Have you taken every possible step to secure *your* computers, including hanging a garlic clove in front of the screen?
No not smug – just impatient.
My bet would be that there is something the patch fixes that is interfering; ie. AV Protection (ironic, I know).
and I’ll bet they drink cokes too….
The Microsoft Security Response Center is reporting that the BSOD is known to occur on computers that already have malware installed on them, but they are not ruling out other possible causes (yet):
http://blogs.technet.com/msrc/archive/2010/02/12/update-restart-issues-after-installing-ms10-015.aspx
Apparently this has been found to be true.
Also, all may have a program called SecureIT.
you sir/maam are correct. alureon.ct is the name of the malware that causes the kb file to spazz out. they say that the
mslive onecare scanner will detect and fix it but i cant get it to work
I have tried this, but after windows setups runs from the cd I get the blue screen of death again so I can’t do the instructions. So frustrating!!
not sure if it will help im just a trouble shooter. have you tried disabling the hdd in boot sequence?? thats how i got mine to boot from cd without the bsod.
couldnt help but notice a small difference in the ops command and the one i got from tech support.
CD $NtUninstallKB977165$\spuninst
>BATCH spuninst.txt
>exit
vs.
CHDIR $NtUninstallKB977165$\spuninst
3. Type this command: BATCH spuninst.txt
4. Type this command: systemroot
5. When complete, type this command: exit
when i tried it using the CHDIR command it didnt work but when i just put CD it did.
Thanks a lot for the solution given here. I was also encountering the Blue Screen of Death after installing these patches. I followed the steps and everything is working fine for me now. As of now i have deferred the installation of these patches. Am i not supposed to install these patches ever?
And what could be the reason for these patches not working for some users only?
Brian, thanks as always for your news. I hope things are going well for yoiu and your family since you left the Post.
As for the notebook problem, for less than $50 you can buy an external USB DVD burner. It’s a handing thing to have for loading software, burning rescue disks, or copying files.
Trying to uninstall that one update results in a scary warning that removing that update might cause like 8 others to stop working properly. So much for being modular. So I had to choose — would I rather be damned if I do or damned if I don’t?
Well from my extensive experience very little is modular when Microsoft is concerned (eg. anti-trust for IE+OS and WMP+OS in Europe).
So are you damned doing or damned donting?
It’s possible that uninstalling this update would potentially leave other updates not doing what they were meant to but rest assured it will get fixed when you install your next service pack. right?
Wouldn’t the needed updates just how up in March?
And of course there’s another subset of XP users who can’t get online to even read this after a blue-screen.
Thx MS for keeping my part-time computer repair business mostly busy!
😉
Ha, ha, ha
When I first read about the bug, I immediately downloaded the code that let me get to a cmd prompt with super-user privileges. I also decided that, come what may, I would never patch this feature. Right now I’m a very happy bunny.
Automatic Updates installed this (and other updates) last night on 150 XP Pro machines last night – no problems. I wonder if it’s specific to XP Home?
No, I have XP Home/SP2. I installed the 10 updates yesterday and so far, no BSoD.
Some blogs are reporting issues with KB977165 and Windows 7; however my W 7 machine updated OK.
Many WinXP users experienced BSODs after installing the previous Kernel update KB971486 (MS09-058) in October 2009. While removing either update may resolve the BSOD, doing so leaves the computer subject to the security vulnerabilities addressed by the update (which Automatic Updates may reinstall anyway). Best course of action (i.e., resolving the BSOD and getting the WinXP box patched) is to open a free support incident by phone or via email (https://support.microsoft.com/oas/default.aspx?gprid=6527). More at https://consumersecuritysupport.microsoft.com/
We deployed these patches to our pilot group which consists of about 50 of ~1900 computers. At this point we have not seen a single issue. We run Window XP with SP3.
You can try to boot into safe mode. Once in safe mode, you should be able to remove the update that is causing the problem. If you can’t get into safe mode, then try the recovery console, or use WinPE or BartPE. BartPE (http://www.nu2.nu/pebuilder/) is a reverse engineered version of WinPE. Both are liveCD versions of Windows. The only problem I have with BartPE is that sometimes it doesn’t have the proper IDE controller driver and therefore the liveCD does not see the hard drive after booting up, but if it is built on the system you are trying to recover, then you should have no problems. Unfortunately, Winternals ERCD is now a Microsoft Enterprise Product (http://www.microsoft.com/windows/enterprise/products/mdop/dart.aspx). So, the relabeled ERCD is of no help to the average user. Perhaps Brian could have a post on BartPE sometime as a follow on to the Ubuntu LiveCD post he had on SecurityFix.
It should be noted that BartPE requires a full XP installation CD (>= SP1) in order to build the image. This could be a problem for those who obtained their XP boxes with hidden partitions or “recovery” discs.
For those without XP CD — you an build a repair console CD with this ISO:
http://www.thecomputerparamedic.com/rc.iso
If no is burning software or you don’t know how to use yours to burn ISO images — this ISO burning software is free & easy to use.
http://www.snapfiles.com/get/burncdcc.html
Download the burncdcc.zip> unzip it> run it> point it to the RC.ISO you downloaded> follow prompts. I have it finalize the CD.
The ISO is simply the recovery console — nothing more but will get you there to follow OP’s post.
thanks for the fix, oopsie.
anyone without an xp disc try this!
Mup.sys is a Microsoft system network driver. The driver is performing an illegal action which is causing the kernel to fail to load causing the BSoD. You may be able to boot up in Safe Mode w/o Networking, but previous mup.sys errors have even stopped Safe Mode from working. Until Microsoft gets enough calls and knows which systems are failing and what software and hardware they have on them, then uninstalling the mup.sys update is the only workaround for now. Once Microsoft knows the underlying cause of the problem, then the patch will be fixed. The cause could be a third party device driver causing mup.sys to fail or it might be a bug in the mup.sys update that only causes a failure due to a third party driver being present. It is futile to speculate about the cause of the failure at this point. If people want to know why this happened to them, they need to post details of what computer this happened on (Manufacturer, Make, Model, extra software added) on support forums. However, Microsoft will likely have a support page available within 1-3 days about this particular error along with a fix. If you think you have problems, I guarantee you that many Microsoft Support techs are trying to help some big business customer with many severs and hundreds or thousands of clients that are disabled with this error as I write this.
My four year old desktop with XP Pro SP3 ran the updates early Wednesday morning and has rebooted successfully twice since.
Before doing so I came here and the ISC and found no warnings. I guess we should wait a day or two before the MS update?
You have to assume MS has some kind of benchmark process prior to releasing updates, but this one seems to have been skimpy. I’d be interested to know how many platforms are affected.
Uh, no. BartPE requires the i386 folder from the OEM Install CD. If no OEM Installation CD has been provided by the manufacturer, then there pretty much has to be an i386 folder on the C: drive of the computer. If the OEM did not include an i386 folder on the C: drive, then there is an OS image on a hidden partition somewhere on the drive. If this latter issue is the case, then you should call your OEM and demand an install CD because your system was crippled when you bought it because it had no repair or feature installation capability to begin with. The OEM is supposed to sell you a complete fully functional licensed copy of Windows with your computer. Without the i386 folder, you do not have a complete fully functional operating system bacause you are missing components you need.
I ran into the problem with no Windows distribution CD when I bought my first complete retail computer (I had always built my own PCs prior to that). It was an HP Pavilion from one of the main retail chains and I didn’t realize it didn’t include software media until I opened the box at home. I called HP customer support requesting the media which I felt I rightly owned by virtue of the Microsoft license sticker on the side of the unit. Their response was that their OEM agreement with Microsoft specifically prevented them from providing said media to customers (so they deftly passed the buck to Microsoft). :-\
I guess it’s back to building my own systems from here on out, because after having to recover my wife’s XP PC from this recent update BSOD event using the XP CD I had, I never want to be in the position of being stuck again (keeping my fingers crossed that the retail HP unit running Vista behaves itself).
I have Vista Home Premium 32 bit and noticed after the update that some of my news sites no longer work correctly. I thought it was the site itself, but now I am wondering if it had to to with the update since I never had this problem on these sites before. I no longer can join in discussions on my news sites because after I post my comments they never show up-they just disappeared. And I no longer can update my account on one of the news sites. Has anyone else noticed any web sites not working correctly since updating with Windows Update?
Darn, now I hear about it. I’ve already rebuilt my entire system. I turned off my machine one night (with auto updates enabled) and the next day it wouldn’t boot. Blue screen would appear but not long enough to read, tried to pause it but gave up. After I rebuilt I put the durressed C: drive into my machine as a secondary and it all read perfectly fine and still is. At that point I knew something was fowl.
Shouldn’t this link, “building a custom XP install/bootup disc on a USB drive” forward to instructions on just how to accomplish that? There is no mention of bootable USB drives on the page I was forwarded to.
I’ve several low-capacity USB flash drives that are no longer being used; a backup boot drive would be nice.
Chris,
Apologies. I will fix that link momentarily. Thanks.
A poster on the story at ISC ( http://isc.sans.org/diary.html?storyid=8209#comment ) stated that in his case it was due to a rootkit which, among other things, modified the system file atapi.sys . I have confirmed on one system I have where this update caused blue screens was similarly infected.
Hi Folks
Thanks for the great page Brian
For those of you with this issue and without a CD/DVD player to boot from, consider if you can, PXE booting
I use PXE booting at work/home to access tools and utilities. It’s great
Here’s a post to assist in creating a PXE environment for a few decent utilities
http://wiki.contribs.org/PXE_booting_to_BARTPE
Here’s a shameless plug for my own messy page on PXE booting if anyone wants to learn FreeBSD
http://www.isgsp.net/freebsd/pxe.html
I hope this helps
Take care
Steve
More interesting info on this from comments at http://isc.sans.org/diary.html?storyid=8209
“Because antivirus software is likely not to be able to detect malware on a running rootkit-infected system (because the rootkit will ‘cloak’ its existence), this may help people (who’ve not patched yet) to determine if their PC is infected with the malware identified by Patrick W. Barnes. However, I need some help to make sure.
The length of the original XP SP3 atapi.sys file (which lives in c:\windows\system32\drivers\) is 96,512 bytes. The malware version on Patrich W. Barnes’ website has the same length, so this doesnt help. Furthermore, most people don’t understand “sha1sums” and do not have sha1sum.exe on their PC.
The binaries are mostly identical; the malware version has 4 bytes changed at the beginning of the file, while, interestingly, it’s version information block has been overwritten with the apparent malware code, probably leaving all original functionality intact.
Therefore, a modified atapi.sys by this particular malware can *probably* easily be identified on a running system by right-clicking c:\windows\system32\drivers\atapi.sys (Explorer must be configured to show system files): a *completely missing* Version tab in the file properties dialog box definitely means you’ve got a problem.
However, a present Version tab doesn’t necessarily mean your system is okay. The malware *may* have saved the version info data to a separate file (or the registry) before overwriting the section in atapi.sys.
Therefore, I’m very interested to know if anyone observes missing version info in atapi.sys on an (unpatched, otherwise it would BSOD) XP PC.
posted by Bitwiper, Fri Feb 12 2010, 00:55″
Just checked my ‘old’ atapi.sys – rootkit infected apparently (although AVG found nothing when run under the infected system). So it looks like there is a lot of merit in this theory, as my PC BSOD’ed…
2 weeks ago I had problems completely unsolved by AVG 9.x
Malwarebytes cleaned up the machine in one swipe except for one item which was cleaned up by emailing their SUPERB tech support.
AVG= shaken faith I am afraid.
Each item has it’s strengths. I have to use AVG, MalwareBytes, SuperAntiSpyware, Spybot Search and Destroy to keep my machines clean. It’s a huge battle. I would love to find one single product that does everything. I would sign up as a reseller in a heartbeat.
Just curious. What do you do on your PC that requires all those products to keep it clean? 🙂
Got a Vista Home Premium system here that I’m working on for one of my users (side biz… clean/repair systems) and it’s blue-screening after updating Tuesday night / Wednesday. The error is 0x00000007E and the failing internal that is mentioned is *always* SCFLTR.SYS.
Is this the same problem as the Windows XP machines are having? FWIW, I had one machine I had to rebuild at work today after auto-updates. Would not even start in safe mode, due to Deep Freeze having been installed. Needless to say I did not reinstall DF when I rebuilt the system.
Anyway, if anyone knows how to fix a Vista system that has started blue screening immediately after installing Windows updates, I’d love to hear!
I’m not certain this is a related problem, but since it’s a BSoD issue that appears to have started after installing the update, I’ll report it.
Windows XP boots up OK, but an attempt to shut down or restart results in a BSoD most of the time. Also, my computer no longer connects to my cable modem, either by plugging in the Ethernet cable or through wireless. (A USB Verizon Wireless Broadband internet connection still works.)
If, following some advice on the Lenovo notebooks forum, I go into Device Manager and uninstall the Network Adapter (an Intel 82566MM Gigabit Network Connection), I can then access the Internet via my Ethernet cable connection the next time I boot up (the driver automatically reloads), but the connection is gone after the next shutdown and bootup.
I would recommend everyone install the Recovery Console on the hard drive.
\winnt32.exe /cmdcons
(XP machines)
Very handy and frees you from needing the CD in most cases.
MS removed this tool in Win 7 (sigh)