Subscribe to RSS
Follow me on Twitter
Join me on Facebook
The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date.
Hap Cluff, director of the information technology department for the City of Norfolk, said the incident began on Feb. 9, and that the city has been working ever since to rebuild 784 PCs and laptops that were hit (the city manages roughly 4,500 systems total).
“We don’t believe it came in from the Internet. We don’t know how it got into our system,” Cluff said. “We speculate it could have been a ‘time bomb’ waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.”
Cluff said the malicious software appears to have been designed to trash vital operating files in the Windows\System32 folder on the infected machines. Cluff said a healthy, functioning System32 directory weighs in at around 1.5GB, but the computers infected with this as-yet-unidentified malware had their System32 folders chopped down to around a third of that size, rendering them unbootable. Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.
“Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Cluff said the city is treating the incident as a crime, and that it has notified the FBI. “We will be quarantining several PCs from various locations and tracking their chain of custody to assist in any forensics analysis,” he said.
Only those PCs that happen to have been “shut down” between 4:30 p.m. and 5:30 p.m. Tuesday, Feb. 9 were impacted by the attack, Cluff added. That’s in part because of the data destruction, but also because the malware also modified the “boot.ini” file, an essential file that tells the computer the location of the Windows operating system.
“This was the amount of time it took our network and security engineers from discovery to containment,” he said. “So all those employees who were being ‘green’….we now know who they are.’”
Tags: city of norfolk, hap cluff, logic bomb, time bomb
I see variously that the PCs were destroyed, and that the data were destroyed. I assume it was just the data (including the operating system) that was destroyed, and that the operating system, at least, can be restored.
As far as user data: “Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.”
As a policy, “urging” sounds a little wishy-washy to me. For those employees who do not, despite the urging, what is the policy for their backing up data?
Perhaps they should try cajoling.
Well-loved. Like or Dislike:
57 
14
The big news is the vector of attack was an “ancillary system” that probably no one really cared about unless it wasn’t working. It would be interesting to see what level of print server was violated. If it was a PC with sharing or if it was a dedicated piece of vendor equipment like a Print Server/NAS. Would be highly interesting if it was a violation of the embedded engine suach as like in one of those big printers by Xerox, Konica Minolta or a Ricoh which only businesses can afford.
Now the kicker:
Given the recent Tdss rootkits I suspect, once rooted, many diligently made full backups on WinTel’s are not functional yet “appear” to be so until its too late.
Let the disaster recovery risk assessments begin!
As for the loss of data:
PC users are typically unable to really discern if their data is being backed up and poor application design often doesn’t help. Apple’s Time Machine is the best GUI I’ve ever seen for aiding users by giving a sense of what is being backed up
Unless the accounts were setup to use the Domain properly there are many ways to lose data. Given the write up I suspect the Home Directory of the accounts wasn’t in the Domain.
Critical business files often get stashed onto the desktop, or in local folders like my downloads or my photos etc, ever thought about backing up that Sticky notes app? Bookmarks & Cookies, from browsers also are a pain to re-create. Users are lazy and forget all their web-logins let alone the site to log in to, obscure fonts are added over time sometimes with all the little productivity apps. User Settings, forms and templates for business productivity apps have to also be considered as being littered all over the system.
Unless the PC is configured properly these files are not going to get backed up or sync’d 100% of coverage 100% of the time.
Hot debate. What do you think?
18 
15
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
3 
24
What are the chance that this actually IS a piece of
malware here, and that it’s not just a bad update that Norfolk pushed out to their own machines?
With the “print server” cleared, there’s no evidence that it was ever actually implicated, and I have seen more than one case of people blaming machines that were unrelated to a problem just because they saw some funny network traffic from it.
At this point the cleanup has been so horribly bungled that there isn’t even any evidence of the original problem; if I were a paranoid sort I would suspect this to be an update gone wrong and a bunch of IT people attempting to cover their rear ends by blaming it on malware.
Well-loved. Like or Dislike:
22 
6
The chances are OK. Zeus can self-destruct boxen on command.
Like or Dislike:
3 
2
Time to dump Active Directory and go to Network File System (NFS) or Andrew FIle System (AFS) for true data redundancy. Of course that means dumping that other OS for some flavor of ‘NIX, Linux, NetBSD, etc…I know at my worksite I trust my data on the CentOS machine at my desk (NFS mounted filesystem) over the WinXP machine any day (an AD royal mess…) and both are supported by competent IT folks.
Hot debate. What do you think?
12 
11
“Time to dump Active Directory and go to Network File System (NFS) or Andrew FIle System (AFS) for true data redundancy.”
Active Directory is an LDAP implementation, not a network file system… SMB… DFS…..
Like or Dislike:
2 
2
I keyed in on the data comments as well. First they say the systems won’t boot because the System32 directory is trashed but any IT person worth a damn knows you can boot off CD/USB Boot disk and recover local data. Secondly I agree what was the policy because saying you urge users to do something when those systems contain sensitive and private data is not an IT policy.
Whole story sounds fishy and either they have their facts wrong or they need some new IT management.
Well-loved. Like or Dislike:
18 
2
You are absolutely correct!!! I automatically boot from CD\USB key in these types of incidents to try to save any worthwhile data while at the same time look for anything suspicious…
I mainly work on people’s home PCs but I used to work for City government; this smells of a lot of CYA’ing…
If they had the slightest inkling that a SERVER (of all things) may have been infected, a case should have been started, image created, etc…
Just saying
Well-loved. Like or Dislike:
7 
1
It is absolutely true that you can boot to a recovery CD (my favorites are Ubuntu or Knoppix) to recover non-encrypted data. Sometimes this is a little difficult to do if the Big Wig or Highly Regarded Politico has “accidentally” gained access to the PC repair lab, and is rocking back and forth from one foot to the other, all the while moaning about “when am I getting MY laptop back?”
In that case it is more expedient to load a fresh image, and let them get back to work …
Like or Dislike:
1 
1
Destroying data does not destroy machines as effectively as headline hyperbole destroys credibility. No wonder it is so hard to get people intersted.
Well-loved. Like or Dislike:
65 
25
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
15 
60
yeah, because i can’t just mount that drive up under another machine and pull anything off of it that i want! no way nuh uh!
Well-loved. Like or Dislike:
26 
6
Or, stick a rescue disk into the machine and copy the data to an external drive.
Well-loved. Like or Dislike:
26 
2
Interesting that it came on Patch Tuesday.
Well-loved. Like or Dislike:
24 
4
Maybe because whoever placed the Malware scheduled it for that day knowing a large number of computers would be shut down and restarted due to Microsoft Updates?
Well-loved. Like or Dislike:
32 
2
“Interesting that it came on Patch Tuesday”. Agree. There was news about anwindows update that came out that same Tuesday that effected users running XP. Microsoft quickly recalled it. This sounds like the same issue. Why are they saying this is a virus though?
Well-loved. Like or Dislike:
17 
10
Krebs blogged that too;
http://www.krebsonsecurity.com/2010/02/rootkit-may-be-culprit-in-recent-windows-crashes/
Well-loved. Like or Dislike:
7 
3
The data is not gone. There are free tools such as sleuthkit, scalpel, and foremost which can recover deleted files from disks or disk images. We aren’t told whether users’ data was intentionally deleted, or just system files were deleted. If the latter, then user’s data is easily recoverable by just copying over an identical system32 folder from an intact identical system. Even their deleted data would be recoverable provided the drives were imaged before they were restored from backup or rebuilt. But their document folders should have been on network shares any way. It seems that their system admins were not paranoid enough or diligent enough to preserve data or evidence so that they can minimize the damage and find out how the attack occurred. Chances are that whoever did this will escape and that this attack or a variation of it will happen again.
http://wiki.sleuthkit.org/index.php?title=FS_Analysis#Manual_Deleted_File_Recovery
http://www.forensicswiki.org/wiki/Scalpel
http://www.forensicswiki.org/wiki/Foremost
Like or Dislike:
3 
2
I’m no computer expert, but if the only thing damaged on the machines was the Windows\System32 folder, why not just start them up with a linux live CD and copy the users’ data files to an external drive?
Well-loved. Like or Dislike:
67 
5
Because it’s 20x faster just to restore the same, known good image to a bunch of PCs. The issue is, when a computer system gets compromised, particularly with such obvious, destructive intent, it’s always a good idea to flatten the system and reinstall.
Well-loved. Like or Dislike:
45 
18
Totally agree with you about the flattening. What I’m saying is, for those users who can’t access their data because the OS is hosed, here’s an easy way to grab it before flattening/reinstallation.
Well-loved. Like or Dislike:
41 
4
Ah, I get it. Well, I don’t think the City of Norfolk has much patience with that method at the moment. They’re basically trying to get back up to full operational state as quickly as possible. What you’re describing would probably take way too long on a per-machine basis than they are willing to expend at this point.
Well-loved. Like or Dislike:
22 
9
Easy solution: Grab hard drives from machines not in use/spare hard drives, image those, put them in the workstations.
If stuff is compromised, first reaction should always be containment and wipe with fresh storage media, and archive/hold the old media until it can be safely reviewed. Don’t just nuke-in-place on the compromised system, nuke from a different ‘known good’ platform.
Well-loved. Like or Dislike:
30 
7
The only problem with retrieving the data off of the infected computers is that they still have not identified the code that trashed the computers to begin with. Any data that is removed from the trashed computers would have to be treated as infected and I would not recommend letting the users access that data until the malicious code is identified. Returning to a newly installed image will ensure that the malicious code does not return to those machines will provide peace of mind and serve as a training aid for those who need explanations as to why they should utilize server resources.
Well-loved. Like or Dislike:
12 
3
How long did it take the city employees to create the files in their My Documents folders? Weeks? Months? Years? How long does it take to copy My Documents? About 5 minutes per machine? Boot a Linux liveCD, plug in a USB hard disk, copy the files.
Im just wondering if they had a backup system in place to backup people’s documents? Or was it all voluntary? Oh well.
live and learn I guess.
Well-loved. Like or Dislike:
18 
3
Depends on how much the data is worth and how much time it takes to recreate the data.
On the other hand it is a good opportunity to tell lazy users it serves them right and let that be a good lesson for them.
Hot debate. What do you think?
8 
8
Well thats what PXE is for, you could set up DSL (D*mn Small Linux) on a NFS or SMB server (only 40M) and set it up to backup what was left on the drive into “quarantine” space on a server, reformat the drive, reboot into PXE, chain boot into a network install of Windows to re-install / re-ghost the OS and apps. All the user has to do it press the necessary key at boot time and select network boot, go grab lunch or what ever. In the mean time the quarantined files are virus checked, executables and drivers et al removed, and then placed on a network share for the user to cherry pick back onto their new clean system. This is the sort of things sysadmins should be paid for. This is also the type of setup I have used before (but for Linux desktops not Windows desktops, but it should work just fine).
Well-loved. Like or Dislike:
5 
1
It’s Government, how “functional” was it to begin with.
Like or Dislike:
2 
5
epidemic – I think this is the proper analog to the Norfolk case.
It sounds like despite vigilant prevention a pandemic plan of action was missing.
When all internal boxes are suspected as tainted, how to compartmentalize and quarantine newly restored devices from being re-rooted or infected? Especially when the vector of attack has no residual evidence left, since the flattening and rebuilding was destructive, to analyze and synthesize a defense.
I’ve had scenarios that until you are absolutely sure all the active infections, worms, etc are taken care of, the first machines rebuilt are still at risk to be taken out once again by an infected laptop that just rolled into the site because it was off site during the epidemic.
Would internal compartmentalization and internal hierarchical trust relationships prevented this widespread epidemic?
Should Norfolk’s citizens, suppliers, vendors, contractors, etc now have to fail-closed, ala self-quarantine to ensure they are not at risk.
Anyhow, doesn’t Lenovo/IBM/HP/Sony offer a hidden checksumed partitioned based rescue and restore even on PCs? Im curious if most support teams wipe this vendor solution out, reclaim the space, and re-deploy their own kit-bashed solution.
Like or Dislike:
3 
1
Most wipe it out on PC’s in the enterprise. Because instead of restoring to a vanilla MS system, it restores to a “factory” MS system which is not setup for the environment and often has unneed/wanted extra’s. And over the last 3 years (about half a corporate lifecycle) MS has release 3 OS’s and a 4 on the way before the end of 2012, so often that partition is also for the wrong OS.
Well-loved. Like or Dislike:
5 
1
It looks like this is dissipated energy on OEM’s part if corporate shops just nuke the recovery partition.
On paper it seems like a great idea but as you cited the execution of that idea is already stale out of the box. It would be better to torrent an image regularly and occassionally blasting an image when upgrading into the recovery partition .
This shoots disk requirements through the roof. 1 chunk for active, 1 chunk for recovery, 1 chunk for virtualized install/upgrade. Reorder the boot sequence to get to a working/upgraded OS. However this would mean built in pre-OS support to redirect everything. I’ve done this with AIX and Linux but now recovery becomes an extended form of revision control. Perhaps disk will be cheap enough one day or data will be so expensive one day to warrant the investment for consumers.
Its amazing how many shops have to be so inventive for a certain product line’s deficiencies.
Like or Dislike:
3 
1
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
0 
13
What I can’t understand is an IT division that would allow their users a choice of where to store their documents and files. In a corporate infrastructure, it seems to me that the expense of replacing data, however it’s done, far outweighs the choices given to a user. It ought to be corporate policy that the data is always stored in a central location, or barring that, backed up every day, prior to shutdown.
Well-loved. Like or Dislike:
22 
11
@Henry S. Winokur
I am an administrator for a medium sized network (about 160 computers, 200 plus users).
I have redirected “My Documents” to the users “home share” on the network, and have office templates that default where to save Word & Excel files. So every user has to “GO OUT OF THEIR WAY” to save files where they won’t get backed up.
This works for 80% of the users, but you would not believe how many files we have lost because they were saved to the desktop or the local drive.
No matter how hard the IT department tries, if a user wants to save files locally, they will.
Well-loved. Like or Dislike:
43 
3
@n3ujj
If you made it so the only areas of the system that are writable by the user are on network mounts, then they would be forced to write their data to the file server/share. This is easily accomplished in environments where the user has no administrative privileges and entire home directories are network mounted (and not just a My Documents folder).
If you worry about removable media, don’t allow non-administrators to mount. Although I doubt any users would make an effort to permanently store files on a USB stick instead of their home directory.
Well-loved. Like or Dislike:
16 
12
*face palm*
I don’t know where you get your ideas… Imagine the strain on your servers if you had 200 people pulling profiles off the network and storing all information to a mapped drive.
Even with smaller group of users, the chances of profile corruption are far too high.
Now let us factor in older (and some newer) software that requires admin rights to the local machine.
The point of this is that no matter how you look at it, there is no foolproof system to prevent a situation like this.
Well-loved. Like or Dislike:
20 
15
@Ryan:
Roaming Profiles were made for just this reason. The load is only applied during login (copying from server to workstation) and logout (copying from workstation to server).
If the file system has the appropriate ACLs in place, Wesley Miaw’s technique of My Documents/Desktop folder redirection is perfectly valid and correct – except for notebook users who work off-site.
Well-loved. Like or Dislike:
13 
1
@ryan – I’ve administered networks with 30-40k people and had single servers managing home directories for 10-15k users at any given time w/o a sweat – it’s not hard, but it’s not windows either.
Notice what was compromised, people. A benign print server. Security of every system can never be overlooked. The damage is done. The time is now to review the security policy and put one in place in an attempt to prevent this in the future.
With M$ solutions in play (read CLOSED, proprietary) you never know what you will get so you have to keep the kimono as closed as possible.
Well-loved. Like or Dislike:
13 
4
@Rya
You hit the nail on the head “there is no foolproof system to prevent a situation like this”
Hot debate. What do you think?
5 
3
@Wesley Miaw
In our organization, we run a lot of “Legacy” applications, which forces use to allow writing to both the windows folder as well as other folders on the local drive. Until get can get the “legacy” applications to catch up we are at the mercy of the applications.
Like or Dislike:
2 
2
There are several valid reasons for users to store files locally rather than on a network server.
1. Speed. With some larger documents on a slower fileserver, it can be frustrating in the extreme to save frequently. Of course, you SHOULD copy the file up when you are done with it, but people get lazy.
2. Server space. Most companies I’ve worked for have instituted some limit to the amount of data that can be stored in a /home or \My Documents directory on a server. Sometimes the only place you have room is on your local drive.
3. Personal files. I’ve had requests to restore \My Documents folders at previous companies during computer changes, only to find out that the only things in them were pictures and music.
But my cynical side says… Slacktime. Saying that you had an absolutely critical TPS report on your hard drive will either make your boss give you the time to reconstruct it, or will make them delay giving you back your computer while they retrieve the oh-so-critical file. Meanwhile, you get paid. And you don’t have to work.
Well-loved. Like or Dislike:
12 
5
I can understand the speed issue but not the storage issue. Storage is cheap and I mean very cheap today. Explain you need more space for x file + room for growth. Explain respectfully that you are forced to work with storage on your client. If you get refused then ask what happens when the hard drive fails. You still get refused then you e-mail your boss and advise that this big file with lots of work hours put into it is vulnerable to loss if your client fails.
Like or Dislike:
2 
1
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
5 
13
Why not have the Desktop in a user share too?
And enforce policy when users loose their files for not complying with your requests?
I’d say that free extra work hours until they replaced *all* the lost work would be enough for them to take care.
Hot debate. What do you think?
5 
6
@anon
I actually tried that for a while, but shortcuts to applications were not consistent, created more problems than it cured.
Also tried roaming policies, that didn’t work well either.
It’s always a trade-off.
Well-loved. Like or Dislike:
7 
2
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
3 
12
Simple: Remove the damn ‘shortcut’ files that enable that annoying ‘New’ menu. Too many applications like to clutter the damn menu anyway.
Hot debate. What do you think?
6 
4
Good point… unless they use roaming profiles, or force the profile to the server. In this case, the desktop folder is also on the server, along with the rest of their profile. The IT department is paying the price for failing to standardize their infrastructure.
The real world always hurts…
Like or Dislike:
0 
3
It’s pretty simple to lock down systems so that only the most determined users can save stuff to anywhere besides the central servers
1) Take away admin rights except to those that truely need it (outside of IT that might 2 to 3 users in an org of thousands)
2) Redirect the Desktop, My Documents so forth to home shares on the network (some places just redirect the whole user folder to a network share, but unless you have good network don’t recommend it)
3) Lock down file/folder creation on the hard drives in everything except Program Files/Windows Dir (locking down these two totally, can and will cause many app’s to break)
4) Optional, If you are properly distributing app’s you can even lock most of the rights for file creation in Program Files/Windows
4) Block mounting of drives (USB so forth)
Then only way for users to create files on the hard drive is to go a few levels deep, vast majority cannot be bothered to go though that amount of effort and the few that will…well virtually nothing would stop them anyway
Plus with a standardized build and centralised application distribution it makes not on support a lot easyer but also makes your next upgrade (either app and/or OS) a doodle because you not longer have to worry about backing up their personal files which is generally the most time consuming and labour intensive tasks when performing a roll out (because users can never seem to do their own back up’s)
Hot debate. What do you think?
6 
4
Why not just admit that you’d rather go back to dumb terminals attached to a mainframe . . .
Hot debate. What do you think?
5 
3
ok, many of these are based on corporate general office systems that can (and should) be highly locked down.
The problems with the ‘solutions’ in so many of these comments is that they are presented as a panacea. Ok… not EVERYONE is promoting that but…
In any event, there are corporate systems that just do not fit into this mold. Consider a division that is responsible for CGI or digital image/video. The data for any given user can be in the TeraBytes per user. Also, what about ‘corporate’ systems that are used to operate specialized systems (such as lab devices) and or develop code?
Bottom line: to echo a previous sentiment, there is no Cure-All. No one system that meets the operational needs of ALL of the users across every org. Segmentation of environments to allow for containment seems the best overall, with each environment ‘locked-down’ to the maximum extent practicable (which may be very little in some cases)
Like or Dislike:
1 
1
why dont you just remotely copy their files using the $C share? thats what I used to do.
Hot debate. What do you think?
3 
8
Thinking a policy will lead users to do the right thing is a bit euphoric in the average enterprise.
Short of using AD to prevent users from storing files anywhere other than server shares, “urging” is about all you can do.
Basically you make the policy and then urge people to follow it. The average organization is not going to formally reprimand someone for storing and losing files on their Win Desktop.
I’m sure at some point they will have to get together some ballpark number (very ballpark) on what this incident has cost them. The mostly tangible numbers will come from re-imaging hours, not from lost work and lost opportunity expenses.
One of the best lessons for people to glean from this is around preserving forensic evidence. In a case like this, I would have cold-imaged any servers involved and a handful of likely desktops. I just made up the term “cold-imaged”.
Basically making a bit-level copy of the drives without booting in to the OS. The best way to do this is to pull the plug on the machine, not go through a formal shut-down. I would seal up the original HD, and then use the copy to attempt to find the evidence (obviously on a non-networked machine).
Assuming you can identify the malware and link it’s placement back to a user, you can hand the actual drive over for prosecution.
Cheers.
Well-loved. Like or Dislike:
10 
1
You haven’t worked in a municpal government, have you? Once the City Manager’s porn browsing gets blocked, and his nephew the Facilities Sub-Director’s naked photos of his wife get saved to a shared folder, and his secretary’s 500mb Google Earth cache causes her to need 20 minutes to log in after lunch all of your nice secure policies get tossed out the window and you’re back to “recommending” secure computing policies.
Security is nothing but an inconvenience to political appointees, since if there’s a problem it’s not their heads that are going to roll.
Well-loved. Like or Dislike:
28 
3
As a systems admininsrator, you can tell people where to put or save data, but what they actually do is another matter, and you have no control over what they do.
That is one major reason to put stuff on a server. You can force stuff like automatic backups, making archives read only or provide write access only to maintainers etc.
Well-loved. Like or Dislike:
6 
1
Brian, you wrote in a comment-to-a-comment “Ah, I get it. Well, I don’t think the City of Norfolk has much patience with that method at the moment. They’re basically trying to get back up to full operational state as quickly as possible. What you’re describing would probably take way too long on a per-machine basis than they are willing to expend at this point.”
I think they’re missing part of the point. If a user has unique data files stored on his local hard drive, with no copy on the server, the cost of recreating those can be MUCH higher than the cost of salvaging the files.
If this hit one of my clients, and getting operational were time-critical on some machines, I would buy new hard drives for each computer (probably under $50 each in the quantities needed in this event) and recover the unique data from the removed drives as time permitted. The removed drives could then become replacement drives for other machines as their hard drives fail in the future.
Well-loved. Like or Dislike:
38 
2
“…recover the unique data from the removed drives as time permitted.”
In the end, you would end up recovering data from maybe 10 users’ drives in that way. Time really wouldn’t be available for a looong time. It will already take a long time to rebuild the machines. Then they have to catch up on projects set aside to do repairs.
Just pulling and reinstalling the physical drives would take over 120 hours (10 minutes per machine). When you explained how many billable hours were going to be involved to your client, I suspect that they would have identified the 10 users they really cared about post haste.
Hot debate. What do you think?
9 
6
It can take 20-30 minutes to reinstall the operating system by itself. Using a drive imaging utility like GHOST will save more time because you reinstall the OS and the applications. The image can be reinstalled using a CD/DVD or over a network. Obviously, restoration via network can be slow if you are reinstalling 800 images all at once. If this is the case, then why not spend 20-30 minutes trying to recover the user’s files before you reimage the drive and destroy their data permanently? I am not even pointing out that the system administrators themselves are likely negligent in this instance. First of all, they destroyed the file server that was the hub of malicious activity. That’s destruction of evidence. Second, they should have global policies in place through Active Directory that MS Office always saves to a specified network share and that said network shares are always backed up.
About data recovery, programs such as sleuthkit, scalpel, and foremost can recover deleted files. It is best to make an image of the hard drive, but replacing the hard drive and recovering the data from the old drive will do in a pinch as an above comment suggests, especially if the user data is important. These programs are free and are usually used for digital forensics, but they can be used for data recovery as well. The problem most users and admins face is that they are ignorant of their options. Data recovery is not taught or emphasized as a skill.
Well-loved. Like or Dislike:
15 
4
I think you underestimate the scale. With one or two failed systems, you can take the time to do a careful data recovery. By the time you scale the problem to 100+ systems, it becomes a question of making the best use of available manpower.
Their problem is with 700+ machines. At this scale, unless they have a dozen techs hanging on the payroll due to ‘stimulus (or whatever), they can really only do a comprehensive data extraction for maybe a dozen or so machines.
To recover then reinstall data in a comprehensive way takes _at least_ 3 extra hours per machine – more if you test and verify. There is pain either way, but down time and recreation both have costs.
Well-loved. Like or Dislike:
7 
1
At best, they probably have ten techs on staff, working 8 hour days. My company has a few thousand scattered among a number of buildings and we have fewer than ten techs.
Spending just an extra 15 minutes on each computer attempting to recover data just added at least a day, if not more, to recovery time. Even swapping out the hard drives (assuming you can find a supply of 800 new hard drives that can be delivered to your facility in a matter of a couple of hours) can add a couple of days.
More importantly, during the recovery time, no one whose machine has yet to be recovered is getting any work done. And their bosses are calling you. Repeatedly.
My current company keeps our number of techs low by enforcing a very simple policy.
1. All company data is to be stored on a network server. All supported applications automatically save to the network. Users experienced enough to override this should know what they are doing and why, and back up important stuff to their personal network folder.
2. If your computer goes bad, the company will initiate a reimage remotely. If that doesn’t work, you’ll get one from the spares bin, delivered same day (usually within a couple of hours). Actual swapouts take about 10 minutes plus travel time.
If you violate policy 1, sorry. The company isn’t paying for enough techs for one to spend hours with you digging through your old hard drive finding and copying your data. They still have jobs to do setting up new machines for people, swapping out dead machines, etc.
Data recovery is still taught. When things get really bad, it comes in handy. But good planning means you have backups. Resorting to recovery techniques means your backup planning was inadequate, or something really catastrophic happened that took out all of your copies, and you’re in deep doo-doo.
You should never ever have fewer than three copies of anything valuable, each in separate locations. And it’s IT’s job to make sure policies are in place to ensure those copies exist.
Well-loved. Like or Dislike:
13 
3
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
1 
13
All it takes is a disgruntled employee setting up a simple VBscript running under a generic domain admin account to accomplish this. This is why you change admin passwords after someone quits or gets let go.
Hot debate. What do you think?
7 
4
Could have been even simpler than that, like a simple batch file. Sounds like someone planted it as shutdown script in the domain’s group policy.
If anyone should be in trouble, it should be the irresponsible IT staff that wiped and rebuilt the “print server” (probably some old crappy computer with a parallel port). Isolate and contain, not isolate and obliterate all evidence!!
Hot debate. What do you think?
8 
5
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
1 
8
What an unfortunate situation. If people used Linux, this could be avoided…
Hot debate. What do you think?
18 
21
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
5 
14
While you’re all discussing how to keep users from storing files on their desktop you’re totally missing something much more important. (BTW, what about the users’ favorites, and archive.pst files that also are stored by default to the local hard-drive?).
“IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.”
What a MAJOR screwup on the part of the IT staff this was. Let’s see, should we get the print server back up quickly so users that don’t have running PC’s can print (duh) or should we cool our jets for a few minutes more to image the print server so an analysis can be done to find out how system security was breached so we can keep it from happening again?
Well-loved. Like or Dislike:
11 
2
My company’s .pst files for outlook are stored on our network folder, not on our local drives. It’s pretty easy to set it up that way.
Like or Dislike:
3 
4
We use exchange (so there are no pst’s).
Favorites are stored on users “Home” Drive
I use group policy & scripts to hadle all redirection
Hot debate. What do you think?
3 
5
Users other than admins should not have the permissions to run something that could cause this type of damage. If the OS files were trashed, the data is still on the drive, it’s just not bootable. Still, recovering data from 800 systems is not a fun job for anyone. Sounds like an access control list on the print server could have saved a lot of time and money!
Like or Dislike:
2 
3
Yet another group may learn who difficult it is to secure Windows and just how expensive it is. I’ve heard people complaining how much more expensive a Linux or UNIX admin is compared to Windows admins. Maybe the people running Norfolk should rethink what they are doing on the desktop. After all, you might expect user data corruption but OS corruption and OS corruption across a network of computers?
And any admin with his weight in salt will know even enough Linux to use it to fix screwups like this. Most of these workers could probably get back to work if these admins built a Linux liveCD with the tools most of the workers might need. Like SAMBA for remote data access, WINE to run some apps like MS Office off the servers and maybe even KVM or VMWare so they could run images off the network for those critical positions.
Real admins know more then one OS but you get what you pay for when you hire most Windows admins. Experts at clicking icons and not much more.
Well-loved. Like or Dislike:
26 
17
if every system was using linux, you could still write some program to hose it. if you have physical access to a machine, you have root. if you have root, you have the machine. slapping ‘linux’ on something doesnt make it safe.
Hot debate. What do you think?
18 
16
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
8 
17
Oh some of us will of course agree. But most will not because you hit them where it hurts – clicking icons (and discussing security here) is about all they can do. So they’ll just keep modding you down. It’s symbolic for sticking their heads deeper in the sand. Now watch this post go too. What a bunch of deniers.
Hot debate. What do you think?
13 
15
A couple (or perhaps the same) people have mentioned data recovery on compromised hard drives. The solution is simple and should have been in place before.
I’m a network analyst at a school district, and we have pretty much eliminated these types of problems. Here’s how:
1) Setup WDS (RIS) to image computers
2) Use AD policies to direct post WDS (RIS) setup of workstations. This includes redirecting “My Documents” to a network share.
3) Scripted installs, MSI installer of all NOMINAL software used by most (all) computers.
A complete reset of a workstation takes approximately 10 minutes of tech time, including installing all the software. Total workstation downtime is approximately 2 hours. I would suspect it would be a tad higher with that many machines being imaged at once.
However, when the machine is done, it is FULLY patched, Antivirus is upto date, as is all the evil but necessary bits (flash, shockwave, Java, etc). I’ve even setup custom desktops for various departments because it takes less time to copy Default User Profile than it does to remember how stuff was setup the first time.
The point being, if data is important, it will be on the server. And Data is expensive, more expensive than most people estimate. If it isn’t worth saving the HD to get data off, then the data wasn’t worth that much, and buying a $50 hard drive + installation time (as one poster suggested) isn’t worth it, then the data wasn’t worthy of being backed up in the first place.
Just my $.02
Well-loved. Like or Dislike:
19 
5
Any botnet can be issued commands to “kill” its members that would have this effect. Hopefully Norfolk is looking out for evidence of fraudulent online banking, wire transfers, etc…
Well-loved. Like or Dislike:
8 
3
I’m an admin for a service company consisting of 3000+ workstations and 600+ servers. Here it goes something like this:
1. I tell my immediate supervisor “Our users HAVE to store their data on the file servers”.
2. My immediate supervisor tells a business liason “Our users really need to store their data on the file servers”.
3. The business liason tells a IT liason in the business unit “You know, your users really should store their data on the file servers”.
4. The IT liason tells a manager in the business unit “IT says we might want to think about storing our files on the file servers”.
5. The business manager says “Ok, I’ll think about it”.
Well-loved. Like or Dislike:
38 
4
Local Gov’t IT is chronically underfunded and understaffed. It’s easy to play armchair quarterback and sling around a bunch of “why didn’t they?” questions. I’d be surprised if their IT staffing ratios per desktop/server supported and total budget even came close to the average private sector firm. You simply can’t implement some best practices when you have neither the manpower nor the budget to do so.
As to the attack itself, if it did come down to the TDSS rootkit being the issue, I’d be having a long talk with my AV security vendors.
Well-loved. Like or Dislike:
14 
5
Agreed. AV should stop rootkits. It’s no longer reasonable to expect the OS to stop them. That dream went out with the noughties.
Hot debate. What do you think?
5 
8
Can you say digital forensics….
apparently neither could they. A malware outbreak, they trace down the origin…..and rebuild the system rather than performing digital forensics 101?
Only later do they consider it a crime and call in the FBI who will now ask why they destroyed the source…which will have more digital fingerprints on it than an infected machine including {possibly} the name and account that was used to put the malware on the system.
Well-loved. Like or Dislike:
17 
2
My point exactly.
Like or Dislike:
4 
3
Well there’s a pattern to their behavior. We all see that.
Like or Dislike:
2 
5
Situations like this one make me wonder why people still use this crappy OS called Windowze…
Hey you guys, there many other secure OSes out there!
Hot debate. What do you think?
22 
25
Yeah, but this was an inside job. Even if you’re using the most secure OS in the world, is it really going to protect you from a malicious admin?
Well-loved. Like or Dislike:
20 
9
How can you be sure it was an inside job? Had the IT staff not trampled all over the evidence a forensic analysis could have showed for sure. Maybe it was just designed to look like an inside job.
Well-loved. Like or Dislike:
17 
3
I’ll bet an admin ran a poorly written script.
I’ve seen it several times: a company writes a script that, as its last step, does a del *.*.
In testing the script works fine.
Then they deploy through some automation method. Well, if the script doesn’t explicitly set its current working directory, the default is system32.
You can imagine what happens when the script does a del *.* with system32 as the working directory.
Hot debate. What do you think?
9 
6
I’m behind you all the way. Most scripts end with *.*. We’ve all seen that. To blame insiders or worse, to blame Windows, is simply uncalled for.
Well-loved. Like or Dislike:
15 
6
Data recovery is a case by case basis. A high level supervisor, bureaucrat, or executive is not going to like having their workstation wiped and reimaged if they haven’t backed it up and there are important documents on it. Obviously, you would not try to recover a clerk’s system, and the majority of the systems would just be rebuilt, but there will always be certain systems that will be treated differently because of their function or their users. One would likely image any accounting systems that were sabotaged due to their function and as possible evidence.
Hot debate. What do you think?
3 
5
thinking that clerk’s systems don’t have important documents on them, well, thats about as dumb a thing as I think i’ve heard in a while.
Well-loved. Like or Dislike:
7 
3
How do you know it’s an inside job?
I don’t care how tight your Windows security is, you will get hit eventually. It is not possible to lock down the Windows operating system. i.e. yeah you can set NTFS permissions on the C:\Windows\System32 where no user nor application should be allowed to delete files. But this breaks many old improperly written applications. But even then, the hackers can blow right past your fancy NTFS permissions with a simple privilege escalation hack to become the SYSTEM account (root on Windows).
There are so many holes in Windows because it was never properly engineered to be a multi-user OS.
I’ve found malware on users systems with no local admin rights and heavily locked down. All they did was surf the web and an evil advertisement included exploit code that blew through an unpatched IE browser using a zero day exploit. i.e. no fix from Microsoft till months later! The malware was a lot more then just spyware, it managed to get SYSTEM access, install a rootkit, keylogger, avoid detection by the AV software by using encryption, etc. Make secure encrypted connections to the outside world through browser ports, etc. It tunneled right through the firewall. It infected other nearby computers. It stole admin account passwords and infected files on the servers, etc.
Time to abandon ship and get the heck off Windows! Run, don’t walk to Linux or a Mac. It’s your only hope!
These sort of security vulnerabilities just don’t happen on other operating systems. It’s not because they are not targeted, it’s because Windows is so full of holes and therefore is a sweet target!
Well-loved. Like or Dislike:
25 
10
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
6 
15
You’re completely correct. Linux is totally bullet proof and always has been.
http://lwn.net/Articles/341773/
http://www.geek.com/articles/news/linux-exploits-top-windows-for-first-time-ever-2003065/
http://news.cnet.com/8301-1009_3-10291022-83.html
Hot debate. What do you think?
5 
10
The CNET article talks about bypassing ‘security protections in the operating system’. Right there you know this can’t be about Windows – Windows has no security protections whatsoever. And it was a compiler bug. As the readme said, there is no exploit if you’re just looking at the source code. And all you do is use the GCC flag fno-delete-null-pointer-checks and your worries are over.
OK, one bug/potential hack down, 299 thousand to go. Oh wait – that’s Windows! lolz
Like or Dislike:
1 
2
There is no proof this was an inside job, or that it was even malware. It’s all speculation at this point. Unfortunately they may never really know what happened because they did not follow basic security incident response practices.
Hopefully what the City of Norfolk takes away from this is that they recognize how important it is to have a well funded and properly trained IT staff. The only way users are going to do anything with their computers such as backups, proper and secure PC usage, etc. is through IT resources (user training, proper infrastructure, etc.), and that starts at the top.
Well-loved. Like or Dislike:
7 
2
Hidden due to low comment rating. Click here to see.
Poorly-rated. Like or Dislike:
4 
10
No coincidence these were windows machines.
Ditch it and go with something better.
Well-loved. Like or Dislike:
14 
9
Mr. Krebs, some more precise language would be a good thing. Apparently the PCs survived and were not destroyed. They did not meltdown. Non-tech savvy users are confused enough, you don’t need to be making it worse.
To all the commenters slamming “lazy” users, a pox on you. Most shops are understaffed these days, and blaming users for not wanting to waste time on some poorly-implemented data storage scheme is dumb.
And why in the HECK is anyone still using Windows in any networked environment. That is INSANE. Look at all the suggestions in these comments for locking systems down to prevent mishaps…it’s all about removing functionality. And it still doesn’t work. Any shop that really wants a high degree of central control should set up terminal servers, and reserve PCs for staff that really need them, like developers, multimedia producers, and IT.
This is sad.
Well-loved. Like or Dislike:
15 
4
Everyone on this thread needs to start here on incident response –> http://csrc.nist.gov/publications/nistpubs/800-61/sp800-61.pdf this is a link to the nist800-61 which will give you a basic idea on what you should be doing on incididents like this because you all suck at it and after all of you read that, you should join Cluff in reading 800-83 and then pop over to the 800-86 and give that a whirl AND THEN everyone sign up for this class right here http://www.sans.org/security-training/curriculums/incident-handler and learn some stuff cause ya’ll really suck at this…but hey, what would I know? I’m just a public high school English teacher…….
Well-loved. Like or Dislike:
10 
1
You want us all to read 369 pages of tech stuff and then take a couple of 1-week out-of-town courses? You think time grows on trees?
Like or Dislike:
1 
0
2 Comments.
First, regarding everyone storing all files on the server through techniques like roaming profiles:
A lot of city computers are likely to be used in remote locations with only relatively narrow pipes back to the data center.
Second, to re-emphasize what a few folks above posted…recovering files from individual machines simply doesn’t scale well when you need to recover 800 machines. Re-image and move on.
You’d be looking at adding weeks to the recovery effort otherwise, or hiring a scad of techs at $75+/hour — neither one of which are good options.
The city already budgeted for someone’s salary, they can sit there and recreate what they truly need and it doesn’t cost the city a dime extra. Sooner they get a working computer, sooner they start recovering. The city didn’t budget for outside help to recover the files.
That’s a difference between a soft and hard cost that is very real in most municipal budgets.
Well-loved. Like or Dislike:
6 
1
LET THIS BE A LESSON TO ALL YOU SYSTEM ADMINISTRATORS whom I have heard saying (repeatedly) – “it is not a critical server, it is only a print server… we can wait to patch it later.”
From just the article, I have a pretty good guess as to what or how it happened… or how I could replicate such an event with two commands, and little or no evidence left behind.
A disgruntled citizen comes in to use a public access terminal placed there for citizens to look up public records, and PRINT THEM OUT. This public terminal is locked down – sure, it is also on its own private VLAN, lest anyone plug into the network with their own laptop… heck, lets go one further and say they even bound the mac address to the switch port to make sure that any other network device plugged in wouldn’t work (unless they spoofed the mac address).
So, our Disgruntled Citizen Hacker (DCH) takes a bootable USB thumb drive/boot CD and inserts it into the computer and reboots it to Backtrack4 or some other utility – or they simply plug into the network using their own laptop…
Once booted from his device, DCH launches an ancient exploit against the print server that “doesn’t contain any sensitive data” according to the SYSADMIN “and can be rebuilt within hours if it ever got infected.” – except that DCH isn’t all about stealing data, he’s all about getting revenge against the cop that gave him that speeding ticket – and HE’S GONNA SHOW YOU!
Once his script kiddie exploit has him sitting at the c:\ prompt, he does a “NET VIEW” and sees that the print server is on the domain, and can see the entire network from its secondary interface that connects it to the internal network. This system administrator has even copied the SYSINTERNALS suite of tools to the hard drive (he even added them to the PATH! -OR- he copies the SYSINTERNALS suite from his boot device) and with one command, DCH gets to work. “PSEXEC \\* DEL c:\boot.ini” and hits enter, the command starts cycling through all the computers on the network -but he screwed up… it is taking much too long to connect to each computer – only to screw up the boot.ini file? Naw, thats too easy to recover from.
CTRL+C
-DCH’s Adrenaline is now pumping-
PSEXEC -d \\* DEL *.* /F /Q /S
This time, it runs in disconnected mode.
“Ah yes, much faster.” DCH says to himself – except he screwed up again, he forgot to put the “C:\” in front of the *.*, so it is (Q)uietly, yet (F)orcefully deleting all the files listed under the %SystemRoot%\System32 folder and (S)ub-folders (including those files marked as read only), instead of the entire C: drive. Major adrenaline sets in – he’s not gonna cancel it this time. He’s already committed, it’s too late now. That and he’s lost his nerve and is visibly shaking as he’s feeling the rush.
He retrieves his boot device, reboots the computer, and quietly walks away, trying oh-so-hard to not raise any suspicions as he quietly walks back to his car. “Take THAT..Your Honor.” he mumbles to himself as he jams the key into his Honda Civic, it fires up with a roar as the ported exhaust reverberates throughout the parking garage. He revs the engine and squeals the tires as he leaves the ramp – radio blaring.
One hour and 800 computers later the print server is taken offline -and promptly rebuilt- exactly according to the disaster recovery plan. Doesn’t matter – even if they did forensically analyze it, the only evidence they’ll find is a single error (among thousands of errors) in the event log that was caused by the exploit, of itself signifying nothing conclusive. The admins never did set up event log correlation, so once the server was rebuilt, all bet were off. So, our DCH walks away, scot free.
But wait! Did he really?
Check the courthouse cameras. On Tuesday, Feb. 9, sitting down at 4:07pm you’ll see the DCH take his seat at the public terminal. He looks around and cannot believe that the stupid IT department didn’t lock away the entire computer case… they left it completely open!
OR – he did it in the morning. He deleted the boot.ini files – then stopped it – “too obvious” he thought. So he then entered the same command to delete the files, but he put it in quotes and preceded it with “SOON 10000″ to schedule it as a job that will run 3 hours after he’s left the building.
Yes, your Disgruntled Citizen Hacker is going to be a system administrator himself… and he wanted to teach you a lesson.
p0wn3d!!!1
Well-loved. Like or Dislike:
19 
0
Joel, if you know so much why don’t you offer them your consulting time? I am sure they will be glad to have such an expert in their grounds.
How long did it take you to google up all that info you dumped here? It sounds to me you have this Joel Helgeson name but you were probably that disgruntled employee. Do you go to that city’s library to scan their network.
Probably they use outdated anti virus software or use weak passwords or do not have the money to afford the best security. remember this is tax payer money you are talking about here.
Based on the last two paraghraphs, which I actually read, it said it took 1 hour for this attack to destroy all these machines hadn’t the IT people find the culprit and eliminated. To me that sounds like a good thing they reacted quick enough to contain that attack in such short time, otherwise they could have ended up with nothing more than scrap metal all over the city.
Like or Dislike:
2 
0
Here we go again with the OS bashing. It doesn’t matter what os you use, there is always someone to exploit a weakness. I’ve seen everything from os to actual hardware exploits. When you have automated tools such as metasploit and fuzzing applications, it makes it easier for crackers, script kiddies and blackhats to poke holes. as posted in the article, they don’t know what happened as the first thing they did was “rebuild the server” Every malware I see has %systemroot%system32% as the default location. Sure you can lock it down, but as other posters have written, it can break things. When something trips a sensor, first thing I do is isolate it if I can and then use Encase, a write blocker and create a forensic copy, seal the original. I understand that Encase is expensive, there are a lot of good, free alternatives out there. I used to write viruses, rootkits and the like :/ Every OS has security flaws. From what I’ve read here, there were very fundamental mistakes made here. Create sound backup startegies and test them regularly, create security policies and enforce them. Make no exceptions. In my company, which is a global company, there are no exceptions, even for executives. We audit systems on a regular basis to make sure they comply with policy. As stated, they are not sure what happened there.
Hot debate. What do you think?
6 
9
“Those who cannot remember the past are condemned to repeat it”
“Those who don’t know history are destined to repeat it.”
“Those who ignore history are bound (or doomed) to repeat it”
No matter how you say it, they all mean the same. If you don’t learn from past mistakes, eventually will happen to you.
We hear or read stories like this every day, so why does it keep happening? Here is my assessment:
1. Wishy Washy IT staff who know no more than clicking options without giving any serious thoughts to their actions are the worst kind. It is the responsibility of the IT manager for not having qualified professional staff and s/he should be fired.
2. IT manager should know better. Windows is hard to secure and manage; it cost much more and especially when your data is at risk. Electronic data is the biggest asset of any organization. Isn’t time to look for alternatives, which are better, safer, and much less costly? S/He should be fired.
3. Many IT manager consider their positions to be a job to collect a salary and have power. IT manager ought to be professionals in IT. This manager ought to be fired for lack of qualification and his boss is at fault too.
4. Foresight, planing, preparing, and organizing is 80% of any project. Organizations without leaders who possess such attributes are doomed to fail. It is time for the city of Norfolk to re-evaluate its IT department. They need new staff who know what they are doing and how disastrous it could be if they don’t act on the current wishy washy IT organization.
Hot debate. What do you think?
2 
7
” Isn’t time to look for alternatives, which are better, safer, and much less costly? S/He should be fired.”
Please provide a list of applications targeted at local government ERP, Tax Collection, Utility Billing, Zoning and Planning, Permitting, and Agenda Management that have Linux/Apple versions and are much less costly (Open Source?) and are certified safe. I’d love to see what you come up with.
If you can’t perhaps you should be fired?
Well-loved. Like or Dislike:
13 
5
Good Point,
Most of the time you DON’T have a choice.
You pick the application because it does the job, and work around the fact that the developer has no concept of security.
Wouldn’t it be great have this info public?
Well-loved. Like or Dislike:
10 
2
So as they rush to reload all those machines, the evidence goes with it.. Every machine should have gotten a new HDD..
They will never find the culprit, but they will pin it on someone.
Like or Dislike:
2 
1
>Every machine should have gotten a new HDD..
$800 x $50 = $40,000 in drives.
We need to consider “hard” and “soft” costs in making decisions like this.
New hard drives is a hard cost. Wasn’t budgeted, you need to come up with cash to pay for it.
The time for the techs you already employ to re-image existing drives is a soft cost. You already budgeted their salaries, it’s just they’re doing an unexpected activity.
Organizations need Incident Response & Forensic policies to help make decisions in these cases, but simply preserving every single workstation isn’t likely the least impact solution.
The $40,000 is just the beginning of costs. What do you do with the drives afterwards? Are you going to pay forensic qualified specialists to exam each and every drive? Even using tools like Encase you’re looking at months worth of work and I’m guessing $250/machine in time to exam and document.
What is likely today to make the key difference is whether you confidently believe this was just a malicious attack aimed at whacking system32 — in which case you chose the fastest, cheapest option to repair the damage. Or do you suspect this was masking a more serious attack — someone who breached security to gain confidential information or alter records, in which case a thorough forensic review is necessary.
We, as a society, constantly make these trade offs.
Plenty of lawsuits are settled out of court simply because it’s the cheapest, most expedient and certain way to make weak cases go away.
Police do not close highways and call out reconstruction specialists for every auto accident. A fender bender isn’t even worthy of a report in some areas now, but as the seriousness of injuries to people increase so does the thoroughness of the investigation.
Hot debate. What do you think?
5 
3
Have you heard of Google? Obviously not and I am not going to do the research for you, but here are a link per each application I just picket at random for you to investigate:
1. ERP:
Google search:
http://www.google.com/search?q=ERP+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a
Article:
http://ezinearticles.com/?ERP-Products-For-The-Linux-Operating-System&id=937132
2. Tax Collection:
Google Search:
http://www.google.com/search?q=Tax+Collection+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a
Article:
http://www.tectonic.co.za/?p=4435
3. Utility Billing:
Google Search:
http://www.google.com/search?q=Utility+Billing+on+Linux&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a
Article:
http://www.capterra.com/utility-billing-software
Well, you do the rest. If you don’t find what is acceptable, develop your own by hiring FOSS developer. That is what I meant by IT Managers who ought to be Professionals with foresight to innovate and create.
Hot debate. What do you think?
7 
5
Let’s look at that list:
1. Majority targeted at manufacturing, not Local Gov’t. The needs are different.
2. What US Local Gov’t would, in their right mind, use a product with support based in South Africa? Heard of Time Zones much?
3. Has to integrate with the ERP.
As for hiring a developer, we’re talking about a local gov’t here who’s budget is based off of utilities and taxes. They don’t sell widgets to drive revenue and in the current economy, LG budgets are shrinking dramatically, not growing. Sure, they could hire a whole team of FOSS developers to build everything from the ground up, but are you willing to take a tax/utility increase to make that happen? COTS applications are the reality for LG IT because there’s simply not the funding to have a development team in house. COTS developers will target Windows because, let’s face it, they have the majority of the market share.
It’s a cold hard reality those IT Directors deal with on a daily basis, not the pie in the sky world you indicate where everything is all Linuxy, open sourcey, and smells like roses.
IT Budgets at the LG level are set by elected officials who generally don’t know a thin client from a hole in the ground. If it’s not something that stops crime, puts out a fire, fills a pot hole, or gets them re-elected, good luck getting in your budget.
Try this google search and come back when you finally understand it:
http://www.google.com/search?hl=en&q=define:reality&aq=f&aqi=&oq=
Hot debate. What do you think?
5 
4
Ok, let me point to the City of Munich, they moved the majority of the IT computer (14,000) to Linux.
http://news.cnet.com/Munich-fires-up-Linux-at-last/2100-7344_3-6119153.html
There are many other municipalities in Europe and all over the word who already using Linux in local governments reaping its benefits in cost, reliability and security. There are many more who already see what other have accomplished and moving ahead in phasing Linux into there IT operations. There must be something wrong with the US local governments, don’t you think?
Does that tell you something about Linux viability in government or what. There are many I can list for you, but again, I leave it to you. If you are too lazy to Google for them, then I am sorry, I am not going to waste my time on you.
The list was just random selections I picked to get you started, obviously you have one mind set and not willing to inform yourself.
Manufacturing and governments use computers in similar ways, but their data, applications and processes are deferent. Both use ERP, process transactions and generate outputs. Google and you shall find what you need for government.
I am not going to waste my time on you, but read this 2 parts article which was written on Jun 10, 2003 and Jun 19, 2003 By Tom Adelstein.
Part I
http://www.linuxjournal.com/article/6927
And make sure you read part II.
http://www.linuxjournal.com/article/6952
Open Standards in Massachusetts
http://www.desktoplinux.com/news/NS3926478427.html
If these links don’t enlighten you, nothing will.
Well-loved. Like or Dislike:
8 
1
Good post, Abe! But I fear you’re wasting your time here.
Well-loved. Like or Dislike:
6 
2
The reality is these systems are getting clobbered mercilessly. The reality is banks are admitting they lose hundreds of thousands every day (and they’re telling people to just get over it). The reality is nothing has and nothing can improve with such a security foundation.
The reality is also that no one is alone on the Internet. The reality is the situation today – read more Bk if you need a clue – is totally out of control, way beyond the pale, was years ago, and cybercrime is now worth billions.
Take that machinery company. It was their PC that screwed up. Oh gee, what expense to get a secure computer. Oh gee, what happened to Excel? What happened to my GAMES? Use the same box but with Linux – the same hardware takes you farther. So you save money. Opt for a turnkey Apple system and you’re out an extra thousand. One thousand bananas.
Now how much was it that company lost and will never recover?
Click your own link.
Well-loved. Like or Dislike:
8 
2
[quote]Open Standards in Massachusetts[/quote]
To quote the google:
[quote]Your search – Commonwealth of Massachusetts filetype:ODF – did not match any documents. [/quote]
Plenty of hits on .doc and .xls.
It would seem the Massachusetts Open Source initiative went nowhere. I work in that state and have never heard anyone mention that, and most of the hiring for state positions I see revolve around a Windows facing world with your normal variety of different backend systems.
Hot debate. What do you think?
2 
7
Here’s how you get your company understanding why backups are important.
If your company is too big to do all the computers, take the most effective one, like the guy in charge of calling the shots.
Take his harddrives out, replace it with another, install just the basic OS that your office is using.
Wait for him to call the IT department, freaking out.
Tell him it’s all cool, providing he’s been making the backups.
Let him suffer for a few.
Then tell him if he’s really lucky, you might beable to save his data.
This is best to do before a big meeting, presentation, etc. And never, never tell him the truth, you’ll get fired. Trust me, you’ll get fired, ’cause if he was smart enough to understand why you did it, he would of been making backups to begin with. Or her, guess just men aren’t stupid.
Most people won’t do extra crap unless they understand why they are doing that. And whats the best way to see how much you value something? yep, to lose it.
be seeing you…
Well-loved. Like or Dislike:
7 
1
I suppose it’s time to move to Linux and open source software also in Norfolk and stop burning money paid by taxes etc.
Like or Dislike:
5 
2
At least you “sort of” mentioned it only affects MICROSOFT systems.
Like or Dislike:
2 
3
Going on what the City is reported to have said, it seems to me that IT didn’ t have a disaster plan, and like badly trained detectives charged around the place tramping all over and destroying the evidence instead of preserving it – why not have just unplugged the blamed ‘print server’?
- now it’s isolated for examination.
The head of IT services should roll. Absolute amateur. Did far more damange than the reported trashing of system files. Unless … well here we go, conspiracy theories will abound.
Like or Dislike:
2 
1
Many of the comments recommend punishing the people who made an error by firing them.
Perhaps these IT guys are undereducated in security and instead of punishment, they should be sent to training.
As an IT guy, it is difficult to keep up on security, it is a lower priority. These are my sysadmin priorities:
1) immediate problems that keep people from working
2) creating/deleting accounts
3) backup
4) patching
5) upgrades
6) security
Like or Dislike:
3 
1
I can’t even get my staff of 5 people to even save their word files in the officially sanctioned network folder so they can be backed up. When yet ANOTHER seagate died, I shrugged when the sales weasel started crying about lost files.
Like or Dislike:
1 
0
quote:
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.
endquote
As I understand it, print servers have one or more hard drives. All the technicians needed to do was remove the hard drives from the print server and isolate them for forensic review. To make the print servers usable, install new-from-the-print-server-maker hard drives and begin the setup. That’s what spares are for.
Like or Dislike:
0 
0