February 17, 2010

The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date.

Hap Cluff, director of the information technology department for the City of Norfolk, said the incident began on Feb. 9, and that the city has been working ever since to rebuild 784 PCs and laptops that were hit (the city manages roughly 4,500 systems total).

“We don’t believe it came in from the Internet. We don’t know how it got into our system,” Cluff said. “We speculate it could have been a ‘time bomb’ waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.”

Cluff said the malicious software appears to have been designed to trash vital operating files in the Windows\System32 folder on the infected machines. Cluff said a healthy, functioning System32 directory weighs in at around 1.5GB, but the computers infected with this as-yet-unidentified malware had their System32 folders chopped down to around a third of that size, rendering them unbootable. Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.

IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.

“Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.

Cluff said the city is treating the incident as a crime, and that it has notified the FBI. “We will be quarantining several PCs from various locations and tracking their chain of custody to assist in any forensics analysis,” he said.

Only those PCs that happen to have been “shut down” between 4:30 p.m. and 5:30 p.m. Tuesday, Feb. 9 were impacted by the attack, Cluff added. That’s in part because of the data destruction, but also because the malware also modified the “boot.ini” file, an essential file that tells the computer the location of the Windows operating system.

“This was the amount of time it took our network and security engineers from discovery to containment,” he said. “So all those employees who were being ‘green’….we now know who they are.’”


123 thoughts on “‘Time Bomb’ May Have Destroyed 800 Norfolk City PCs

  1. Joel Helgeson

    LET THIS BE A LESSON TO ALL YOU SYSTEM ADMINISTRATORS whom I have heard saying (repeatedly) – “it is not a critical server, it is only a print server… we can wait to patch it later.”

    From just the article, I have a pretty good guess as to what or how it happened… or how I could replicate such an event with two commands, and little or no evidence left behind.

    A disgruntled citizen comes in to use a public access terminal placed there for citizens to look up public records, and PRINT THEM OUT. This public terminal is locked down – sure, it is also on its own private VLAN, lest anyone plug into the network with their own laptop… heck, lets go one further and say they even bound the mac address to the switch port to make sure that any other network device plugged in wouldn’t work (unless they spoofed the mac address).

    So, our Disgruntled Citizen Hacker (DCH) takes a bootable USB thumb drive/boot CD and inserts it into the computer and reboots it to Backtrack4 or some other utility – or they simply plug into the network using their own laptop…

    Once booted from his device, DCH launches an ancient exploit against the print server that “doesn’t contain any sensitive data” according to the SYSADMIN “and can be rebuilt within hours if it ever got infected.” – except that DCH isn’t all about stealing data, he’s all about getting revenge against the cop that gave him that speeding ticket – and HE’S GONNA SHOW YOU!

    Once his script kiddie exploit has him sitting at the c:\ prompt, he does a “NET VIEW” and sees that the print server is on the domain, and can see the entire network from its secondary interface that connects it to the internal network. This system administrator has even copied the SYSINTERNALS suite of tools to the hard drive (he even added them to the PATH! -OR- he copies the SYSINTERNALS suite from his boot device) and with one command, DCH gets to work. “PSEXEC \\* DEL c:\boot.ini” and hits enter, the command starts cycling through all the computers on the network -but he screwed up… it is taking much too long to connect to each computer – only to screw up the boot.ini file? Naw, thats too easy to recover from.

    CTRL+C

    -DCH’s Adrenaline is now pumping-

    PSEXEC -d \\* DEL *.* /F /Q /S

    This time, it runs in disconnected mode.

    “Ah yes, much faster.” DCH says to himself – except he screwed up again, he forgot to put the “C:\” in front of the *.*, so it is (Q)uietly, yet (F)orcefully deleting all the files listed under the %SystemRoot%\System32 folder and (S)ub-folders (including those files marked as read only), instead of the entire C: drive. Major adrenaline sets in – he’s not gonna cancel it this time. He’s already committed, it’s too late now. That and he’s lost his nerve and is visibly shaking as he’s feeling the rush.

    He retrieves his boot device, reboots the computer, and quietly walks away, trying oh-so-hard to not raise any suspicions as he quietly walks back to his car. “Take THAT..Your Honor.” he mumbles to himself as he jams the key into his Honda Civic, it fires up with a roar as the ported exhaust reverberates throughout the parking garage. He revs the engine and squeals the tires as he leaves the ramp – radio blaring.

    One hour and 800 computers later the print server is taken offline -and promptly rebuilt- exactly according to the disaster recovery plan. Doesn’t matter – even if they did forensically analyze it, the only evidence they’ll find is a single error (among thousands of errors) in the event log that was caused by the exploit, of itself signifying nothing conclusive. The admins never did set up event log correlation, so once the server was rebuilt, all bet were off. So, our DCH walks away, scot free.

    But wait! Did he really?

    Check the courthouse cameras. On Tuesday, Feb. 9, sitting down at 4:07pm you’ll see the DCH take his seat at the public terminal. He looks around and cannot believe that the stupid IT department didn’t lock away the entire computer case… they left it completely open!

    OR – he did it in the morning. He deleted the boot.ini files – then stopped it – “too obvious” he thought. So he then entered the same command to delete the files, but he put it in quotes and preceded it with “SOON 10000” to schedule it as a job that will run 3 hours after he’s left the building.

    Yes, your Disgruntled Citizen Hacker is going to be a system administrator himself… and he wanted to teach you a lesson.

    p0wn3d!!!1

    1. Well then,

      Joel, if you know so much why don’t you offer them your consulting time? I am sure they will be glad to have such an expert in their grounds.

      How long did it take you to google up all that info you dumped here? It sounds to me you have this Joel Helgeson name but you were probably that disgruntled employee. Do you go to that city’s library to scan their network.

      Probably they use outdated anti virus software or use weak passwords or do not have the money to afford the best security. remember this is tax payer money you are talking about here.

      Based on the last two paraghraphs, which I actually read, it said it took 1 hour for this attack to destroy all these machines hadn’t the IT people find the culprit and eliminated. To me that sounds like a good thing they reacted quick enough to contain that attack in such short time, otherwise they could have ended up with nothing more than scrap metal all over the city.

  2. Scribner

    Here we go again with the OS bashing. It doesn’t matter what os you use, there is always someone to exploit a weakness. I’ve seen everything from os to actual hardware exploits. When you have automated tools such as metasploit and fuzzing applications, it makes it easier for crackers, script kiddies and blackhats to poke holes. as posted in the article, they don’t know what happened as the first thing they did was “rebuild the server” Every malware I see has %systemroot%system32% as the default location. Sure you can lock it down, but as other posters have written, it can break things. When something trips a sensor, first thing I do is isolate it if I can and then use Encase, a write blocker and create a forensic copy, seal the original. I understand that Encase is expensive, there are a lot of good, free alternatives out there. I used to write viruses, rootkits and the like :/ Every OS has security flaws. From what I’ve read here, there were very fundamental mistakes made here. Create sound backup startegies and test them regularly, create security policies and enforce them. Make no exceptions. In my company, which is a global company, there are no exceptions, even for executives. We audit systems on a regular basis to make sure they comply with policy. As stated, they are not sure what happened there.

  3. Abe

    “Those who cannot remember the past are condemned to repeat it”

    “Those who don’t know history are destined to repeat it.”

    “Those who ignore history are bound (or doomed) to repeat it”

    No matter how you say it, they all mean the same. If you don’t learn from past mistakes, eventually will happen to you.

    We hear or read stories like this every day, so why does it keep happening? Here is my assessment:

    1. Wishy Washy IT staff who know no more than clicking options without giving any serious thoughts to their actions are the worst kind. It is the responsibility of the IT manager for not having qualified professional staff and s/he should be fired.

    2. IT manager should know better. Windows is hard to secure and manage; it cost much more and especially when your data is at risk. Electronic data is the biggest asset of any organization. Isn’t time to look for alternatives, which are better, safer, and much less costly? S/He should be fired.

    3. Many IT manager consider their positions to be a job to collect a salary and have power. IT manager ought to be professionals in IT. This manager ought to be fired for lack of qualification and his boss is at fault too.

    4. Foresight, planing, preparing, and organizing is 80% of any project. Organizations without leaders who possess such attributes are doomed to fail. It is time for the city of Norfolk to re-evaluate its IT department. They need new staff who know what they are doing and how disastrous it could be if they don’t act on the current wishy washy IT organization.

    1. Rhavin

      ” Isn’t time to look for alternatives, which are better, safer, and much less costly? S/He should be fired.”

      Please provide a list of applications targeted at local government ERP, Tax Collection, Utility Billing, Zoning and Planning, Permitting, and Agenda Management that have Linux/Apple versions and are much less costly (Open Source?) and are certified safe. I’d love to see what you come up with.

      If you can’t perhaps you should be fired?

      1. N3UJJ

        Good Point,
        Most of the time you DON’T have a choice.
        You pick the application because it does the job, and work around the fact that the developer has no concept of security.
        Wouldn’t it be great have this info public?

  4. packetscan

    So as they rush to reload all those machines, the evidence goes with it.. Every machine should have gotten a new HDD..

    They will never find the culprit, but they will pin it on someone.

    1. Dalmatian90

      >Every machine should have gotten a new HDD..

      $800 x $50 = $40,000 in drives.

      We need to consider “hard” and “soft” costs in making decisions like this.

      New hard drives is a hard cost. Wasn’t budgeted, you need to come up with cash to pay for it.

      The time for the techs you already employ to re-image existing drives is a soft cost. You already budgeted their salaries, it’s just they’re doing an unexpected activity.

      Organizations need Incident Response & Forensic policies to help make decisions in these cases, but simply preserving every single workstation isn’t likely the least impact solution.

      The $40,000 is just the beginning of costs. What do you do with the drives afterwards? Are you going to pay forensic qualified specialists to exam each and every drive? Even using tools like Encase you’re looking at months worth of work and I’m guessing $250/machine in time to exam and document.

      What is likely today to make the key difference is whether you confidently believe this was just a malicious attack aimed at whacking system32 — in which case you chose the fastest, cheapest option to repair the damage. Or do you suspect this was masking a more serious attack — someone who breached security to gain confidential information or alter records, in which case a thorough forensic review is necessary.

      We, as a society, constantly make these trade offs.

      Plenty of lawsuits are settled out of court simply because it’s the cheapest, most expedient and certain way to make weak cases go away.

      Police do not close highways and call out reconstruction specialists for every auto accident. A fender bender isn’t even worthy of a report in some areas now, but as the seriousness of injuries to people increase so does the thoroughness of the investigation.

  5. Abe

    Have you heard of Google? Obviously not and I am not going to do the research for you, but here are a link per each application I just picket at random for you to investigate:

    1. ERP:
    Google search:
    http://www.google.com/search?q=ERP+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a

    Article:
    http://ezinearticles.com/?ERP-Products-For-The-Linux-Operating-System&id=937132

    2. Tax Collection:
    Google Search:
    http://www.google.com/search?q=Tax+Collection+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a

    Article:
    http://www.tectonic.co.za/?p=4435

    3. Utility Billing:
    Google Search:
    http://www.google.com/search?q=Utility+Billing+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a

    Article:
    http://www.capterra.com/utility-billing-software

    Well, you do the rest. If you don’t find what is acceptable, develop your own by hiring FOSS developer. That is what I meant by IT Managers who ought to be Professionals with foresight to innovate and create.

    1. Rhavin

      Let’s look at that list:

      1. Majority targeted at manufacturing, not Local Gov’t. The needs are different.

      2. What US Local Gov’t would, in their right mind, use a product with support based in South Africa? Heard of Time Zones much?

      3. Has to integrate with the ERP.

      As for hiring a developer, we’re talking about a local gov’t here who’s budget is based off of utilities and taxes. They don’t sell widgets to drive revenue and in the current economy, LG budgets are shrinking dramatically, not growing. Sure, they could hire a whole team of FOSS developers to build everything from the ground up, but are you willing to take a tax/utility increase to make that happen? COTS applications are the reality for LG IT because there’s simply not the funding to have a development team in house. COTS developers will target Windows because, let’s face it, they have the majority of the market share.

      It’s a cold hard reality those IT Directors deal with on a daily basis, not the pie in the sky world you indicate where everything is all Linuxy, open sourcey, and smells like roses.

      IT Budgets at the LG level are set by elected officials who generally don’t know a thin client from a hole in the ground. If it’s not something that stops crime, puts out a fire, fills a pot hole, or gets them re-elected, good luck getting in your budget.

      Try this google search and come back when you finally understand it:

      http://www.google.com/search?hl=en&q=define:reality&aq=f&aqi=&oq=

      1. Abe

        Ok, let me point to the City of Munich, they moved the majority of the IT computer (14,000) to Linux.
        http://news.cnet.com/Munich-fires-up-Linux-at-last/2100-7344_3-6119153.html

        There are many other municipalities in Europe and all over the word who already using Linux in local governments reaping its benefits in cost, reliability and security. There are many more who already see what other have accomplished and moving ahead in phasing Linux into there IT operations. There must be something wrong with the US local governments, don’t you think?

        Does that tell you something about Linux viability in government or what. There are many I can list for you, but again, I leave it to you. If you are too lazy to Google for them, then I am sorry, I am not going to waste my time on you.

        The list was just random selections I picked to get you started, obviously you have one mind set and not willing to inform yourself.

        Manufacturing and governments use computers in similar ways, but their data, applications and processes are deferent. Both use ERP, process transactions and generate outputs. Google and you shall find what you need for government.
        I am not going to waste my time on you, but read this 2 parts article which was written on Jun 10, 2003 and Jun 19, 2003 By Tom Adelstein.

        Part I
        http://www.linuxjournal.com/article/6927

        And make sure you read part II.
        http://www.linuxjournal.com/article/6952

        Open Standards in Massachusetts
        http://www.desktoplinux.com/news/NS3926478427.html

        If these links don’t enlighten you, nothing will.

      2. Rick

        The reality is these systems are getting clobbered mercilessly. The reality is banks are admitting they lose hundreds of thousands every day (and they’re telling people to just get over it). The reality is nothing has and nothing can improve with such a security foundation.

        The reality is also that no one is alone on the Internet. The reality is the situation today – read more Bk if you need a clue – is totally out of control, way beyond the pale, was years ago, and cybercrime is now worth billions.

        Take that machinery company. It was their PC that screwed up. Oh gee, what expense to get a secure computer. Oh gee, what happened to Excel? What happened to my GAMES? Use the same box but with Linux – the same hardware takes you farther. So you save money. Opt for a turnkey Apple system and you’re out an extra thousand. One thousand bananas.

        Now how much was it that company lost and will never recover?

        Click your own link.

  6. Dalmatian90

    [quote]Open Standards in Massachusetts[/quote]

    To quote the google:
    [quote]Your search – Commonwealth of Massachusetts filetype:ODF – did not match any documents. [/quote]

    Plenty of hits on .doc and .xls.

    It would seem the Massachusetts Open Source initiative went nowhere. I work in that state and have never heard anyone mention that, and most of the hiring for state positions I see revolve around a Windows facing world with your normal variety of different backend systems.

  7. Robert Kimmel

    Here’s how you get your company understanding why backups are important.

    If your company is too big to do all the computers, take the most effective one, like the guy in charge of calling the shots.

    Take his harddrives out, replace it with another, install just the basic OS that your office is using.

    Wait for him to call the IT department, freaking out.
    Tell him it’s all cool, providing he’s been making the backups.

    Let him suffer for a few.

    Then tell him if he’s really lucky, you might beable to save his data.

    This is best to do before a big meeting, presentation, etc. And never, never tell him the truth, you’ll get fired. Trust me, you’ll get fired, ’cause if he was smart enough to understand why you did it, he would of been making backups to begin with. Or her, guess just men aren’t stupid.

    Most people won’t do extra crap unless they understand why they are doing that. And whats the best way to see how much you value something? yep, to lose it.

    be seeing you…

  8. Germic Trem

    I suppose it’s time to move to Linux and open source software also in Norfolk and stop burning money paid by taxes etc.

  9. an0n

    At least you “sort of” mentioned it only affects MICROSOFT systems.

  10. Tony

    Going on what the City is reported to have said, it seems to me that IT didn’ t have a disaster plan, and like badly trained detectives charged around the place tramping all over and destroying the evidence instead of preserving it – why not have just unplugged the blamed ‘print server’?
    – now it’s isolated for examination.
    The head of IT services should roll. Absolute amateur. Did far more damange than the reported trashing of system files. Unless … well here we go, conspiracy theories will abound.

  11. Peacock

    Many of the comments recommend punishing the people who made an error by firing them.

    Perhaps these IT guys are undereducated in security and instead of punishment, they should be sent to training.

    As an IT guy, it is difficult to keep up on security, it is a lower priority. These are my sysadmin priorities:

    1) immediate problems that keep people from working
    2) creating/deleting accounts
    3) backup
    4) patching
    5) upgrades
    6) security

  12. Baldemar Huerta

    I can’t even get my staff of 5 people to even save their word files in the officially sanctioned network folder so they can be backed up. When yet ANOTHER seagate died, I shrugged when the sales weasel started crying about lost files.

  13. Tony Smit

    quote:
    IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.
    endquote

    As I understand it, print servers have one or more hard drives. All the technicians needed to do was remove the hard drives from the print server and isolate them for forensic review. To make the print servers usable, install new-from-the-print-server-maker hard drives and begin the setup. That’s what spares are for.

  14. Malcom Reynolds

    That totally sucks. i do not understand people who do this. What do they have to gain from causing other people grief and trouble? I hate these kind of people.

Comments are closed.