The U.S. Department of Homeland Security today took aim at widespread media reports about a hacking incident that led to an equipment failure at a water system in Illinois, noting there was scant evidence to support any of the key details in those stories — including involvement by Russian hackers or that the outage at the facility was the result of a cyber incident.
Last week, portions of a report titled “Public Water District Cyber Intrusion” assembled by an Illinois terrorism early warning center were published online. Media outlets quickly picked up on the described incident, calling it the “first successful target of a cyber attack on a computer of a public utility.” But in an email dispatch sent to state, local and industry officials late today, DHS’s Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) said that after detailed analysis, DHS and the FBI “have found no evidence of a cyber intrusion into the SCADA system of the Curran-Gardner Public Water District in Springfield, Illinois.” The ICS-CERT continued:
“There is no evidence to support claims made in the initial Fusion Center report – which was based on raw, unconfirmed data and subsequently leaked to the media – that any credentials were stolen, or that the vendor was involved in any malicious activity that led to a pump failure at the water plant,” the ICS-CERT alert states. “In addition, DHS and FBI have concluded that there was no malicious or unauthorized traffic from Russia or any foreign entities, as previously reported. Analysis of the incident is ongoing and additional relevant information will be released as it becomes available.”
The statement is the most strongly worded yet from DHS refuting the alleged cyber incident in Illinois. The story broke on Nov. 17, when Joe Weiss, managing partner of Applied Control Solutions, a security consultant for the control systems industry, published a blog post about a disclosure he reported reading from a state terrorism intelligence center about a cyber intrusion into a local water plant that resulted in the burnout of a water pump. The break-in reportedly allowed intruders to manipulate the supervisory control and data acquisition system, or “SCADA” networks that let plant operators manage portions of the facility remotely over the Internet. Within hours of that post, media outlets covering the story had zeroed in on the Curran-Gardner Water District as the source of the report.
Weiss has repeatedly declined to share or publish the report, but he cited large portions of it in my story from last week. The language and details reported in it stand in stark contrast to the DHS’s version of events. According to Weiss, the report, marked sensitive but unclassified, stated:
“Sometime during the day of Nov. 8, 2011, a water district employee noticed problems with a SCADA system. An information technology service and repair company checked the computer logs of the SCADA system and determined the system had been remotely hacked into from an Internet provider address located in Russia. The SCADA system that was used by the water district was produced by a software company based in the US. It is believed the hackers had acquired unauthorized access to the software company’s database and retrieved the usernames and passwords of various SCADA systems, including the water district systems.”
“Over a period of 2-3 months, minor glitches have been observed in remote access to the water district’s SCADA system. Recently, the SCADA system would power on and off, resulting in the burnout of a water pump.”
“This network intrusion is the same method of attack recently used against the MIT Server,” the water district alert stated. “The water district’s attack and the MIT attack both had references to PHPMyAdmin in the log files of the computer systems. It is unknown at this time the number of SCADA usernames and passwords acquired from the software company’s database, and if any additional systems have been attacked as a result of this theft.”
Weiss blogged about the ICS-CERT statement, and said he can’t figure out how the two accounts could be so different. He notes that the day after his blog post, Don Craven, chairman of the Curran-Gardner Water District, was quoted on a local ABC News affiliate television interview saying that there was “some indication that there was a breach of some sort into a software program, a SCADA system, that allows remote access to the wells and the pumps and those sorts of things” (see video below).
“The real thing that bothers me is how could there be such substantial amount of information provided where a lot of it is really a simple yes or no situation,” Weiss said. “Was there a Russian [Internet] address involved or wasn’t there? The Illinois facility also said their technician had observed these abnormalities for 2-3 months. Well, either he did or he didn’t.”
The ICS-CERT communique also mentioned another alleged hacking incident of a water facility in Texas that was widely reported last week. In that incident, a hacker using the nickname “pr0f” claimed to have gained access to a water control systems plant, and posted a series of screen shots to prove his accomplishment.
Regarding the alleged hack in Texas, the ICS-CERT would only say it is still investigating:
“In a separate incident, a hacker recently claimed to have accessed an industrial control system responsible for water supply at another U.S. utility,” the ICS-CERT alert continued. “The hacker posted a series of images allegedly obtained from the system. ICS-CERT is assisting the FBI to gather more information about this incident. ICS-CERT has not received any additional reports of impacted manufacturers of ICS or other ICS related stakeholders related to these events. If DHS ICS-CERT identifies any information about possible impacts to additional entities, it will disseminate timely mitigation information as it becomes available. ICS-CERT encourages those in the industrial control systems community who suspect or detect any malicious activity against/involving control systems to contact ICS-CERT.”
My story from last week quoted Michael Assante, president and CEO of the National Board of Information Security Examiners and a former chief security officer for the North American Electric Reliability Corporation (NERC), expressing concern that initial reporting on cyber-related SCADA incidents often turns out to be inaccurate.
But Weiss said the complete reversal makes no sense, and that “something doesn’t smell right.” By way of example, he points to the fact that while media reports on the claimed hack of the Texas facility made today’s DHS Daily Infrastructure Report, the Illinois incident is noticeably absent from any of the recent editions of that report.
“What this is essentially saying is the state intelligence centers shouldn’t put anything out unless DHS approves it,” Weiss told KrebsOnSecurity. “It says either Illinois is incompetent or DHS is covering something up.”
State fusion centers, most of which were formed under a joint project between DHS and the Justice Department between 2003 and 2007, collect data from government and private sector sources. Some of the centers have produced warnings that have been a tad controversial. For example, a report in 2009 from the Virginia Fusion Center warned that certain historically black colleges were potential hubs for terror related activity, and identified hacktivism as a form of terrorism.
“It says either Illinois is incompetent or DHS is covering something up.”
My money is on the latter.
I really fail to see any motivation for DHS to cover anything up here.
It would seem to me that if anything making the story, if it were true, as public as possible would benefit them in terms of preventing further occurences, maintaining and securing more funding, showing that they are needed still as an organisation – and getting into the conspiracy nutjob territory: keeping the population scared and primed for conflict, hiding other activities, justifying a future conflict against a foreign agent and so on.
I hope you’re not just blindly making this claim to satisfy some baseless paranoia or conspiracy theory is all I’m saying. No offence mean’t BTW.
It is a common kneejerk reaction in the intelligence community to deny first and retain control. Why stir up potential panic that water systems can be attacked? Why confirm to the attackers they what they did was important and valuable? It’s quite possible an attacker with access may not even know what he is doing on that dashboard or where it may be happening.
Recall also that this collateral damage on the specific Illinois incident is pretty big, for instance it directly called into question a vendor and *their* security and the possibility of divulged username/passwords to presumably other remote access systems.
I recall earlier this year when RSA quite denied big issues, but slowly eased up until we later found out how deep the issue really was. No doubt only after they fixed the big fish customers…
This is also all public government stuff. There are jobs to protect and policies to protect and authority to protect. Unfortunately, too often the impression is not that someone needs more funding to do the job right, it’s that they don’t have proper control or competency to do it anyway, so replace them.
Not saying I agree with these viewpoints, but that’s often how it is.
Given how messed up Illinois is (I live there), it could very well be both.
Wired has a good story up today that appears to shed more light on this.
Comedy of Errors Led to False ‘Water-Pump Hack’ Report
http://bit.ly/teDmQ7
Brian it sure be great if you obtain this as stated below .
a hacker using the nickname “pr0f” claimed to have gained access to a water control systems plant, and posted a series of screen shots to prove his accomplishment.
That would shut everyone up and start thinking twice on what they say . 🙂
http://pastebin.com/Wx90LLum
Thanks for the links. pr0f may want to put his OpenPGP key on the key servers. I won’t import OpenPGP keys from any place other than the key servers.
“Was there a Russian [Internet] address involved or wasn’t there?” What the hell’s that supposed to prove? Russia’s a great source of open proxies. If I was doing something naughty, there would almost certainly be a Russian ip address in my proxy chain.
I will stick my money on it that was is correct, The man has been in the industry for years regarding SCADA contros and the security of them and what not.
This IMO seems more like DHS worrying about starting a domino effect and every one and their brother starting to hit SCADA controls and them having to try and stop it after the fact.
The goverment doesnt want utilities going into panic mode, butnI ould bet they have sent out an extremely hush hush advisory to watch out for abnormal behaviour in their SCADA systems… but we wont be hearing hat publicly I would be willing to bet.
Cover up or not…I am in agreement with DavidM. Maybe this will be a wake up call to all utility companies to beef up their security and make sure that ALL possible controls are in place to protect those systems.
It is CRITICAL that all of us who are in business to do the same in these times. More internal system controls and social engineering security awareness training for our employees….public and private.
Neej, in support of my opening comment, I give you three words: Fast and Furious. Aka a proven (and failed) government cover-up of a government-initiated and -executed exercise in lawbreaking. No theory there; just unadulterated (and incompetent) conspiracy.
I have no idea what you think this:
http://en.wikipedia.org/wiki/Operation_Fast_and_Furious
has to do with this story outlined on this page other than showing that various government agencies carry out clandestine and illegal operations.
I don’t think many would dispute this but just because it happens sometimes doesn’t mean it’s rational to jump to the conclusion that this is happening by default in a given situation.
Also I would have preferred it if you had replied in the thread you started but never mind.
You, my friend, are the one who “jumped to a conclusion” regarding the speculation in my opening post. I merely opined that I found the idea of DHS covering something up more credible than incompetence on the part of Illinois. (I might add, only slightly more credible; and I find the the two alternatives by no means mutually exclusive.) At least you accurately inferred this from my post: “that various government agencies carry out clandestine and illegal operations.” Nothing new there. Not for many decades.
Upon reading your original post I do think you are correct about me jumping to conclusions regarding what you meant – sorry regarding this.
I still stand by what I said though – it’s easy to jump to a particular conclusion when it’s simply unknown publically what real events took place.
Governmental transparency issues aside: Either way, the attackers know they were successful. They know DHS can’t control the information and they know we know they are out there.
What is funny or unnerving is that we were talking about exactly this scenario in my college sec course (in Illinois) just a few weeks prior. Instructor teaches a scada security specific course and demonstrated a mini watertower
Governmental transparency issues aside: Either way, the attackers know they were successful. They know DHS can’t control the information and they know we know they are out there.
What is funny or unnerving is that we were talking about exactly this scenario in my college sec course (in Illinois) just a few weeks prior. Instructor teaches a scada security specific course and demonstrated a mini watertower intentionally leaking water as all the controls read “normal”.
Maybe this will be the “Y2K” of the SCADA world?
Brian,
Is it possible that either Stuxnet, or a variation thereof, is responsible for the intrusion since I remember it affecting the same sort of systems? If so, it could be damage control from the government!
I am employed by the government and am quite familiar based on my own observations with how various government agencies are known to assume the best, rather than worst, has taken place in suspected activities until corroborated evidence surfaces to prove otherwise. Whether this is the result of downright laziness or apathy unknown to me. However, if a public water utility has a broken pump (let’s call it a dead human body), that became broken (let’s say, murdered) as a result of unusual activity such as turning off, on, off, on (let’s say, signs of asphyxiation), for the DHS to claim there was insufficient evidence (no killer was identified) to corroborate the utility being hacked is shameful and appalling. In my opinion, just because you don’t have any leads, much less evidence, on an incident doesn’t afford you the liberty to conclude it was merely due to an act of nature and that a crime hasn’t in fact been committed.
Anthony,
Although I don’t agree with your dead body view, you are correct in assuming the DHS significantly jumped the gun on their statment.
What you did not say but is pertinent to the argument is that the DHS Statment is very obviously politicaly not technicaly inspired.
From a technical viewpoint it is all to easy to come up with a pump killing scenario that would in all likelyhood not leave direct evidence of the attack only one of inference at best.
For those still in doubt consider two things,
An absence of evidence does not rule out in any way that an attack has happened.
Secondly only overriding evidence of a direct cause not in any way involving an attack can rule out an attack as a possability.
The primary reason for this is an attacker wishing at some point to repeate the attack will try to hide evidence of an attack and how the attack was carried out
I told my grandtomher how you helped. She said, “bake them a cake!”
This is interesting – I had received a tip a few days ago from a reputable source to expect an effort to cover up the attack. I am looking forward to an explanation of why the Illinois Statewide Terrorism and Intelligence Center was under the impression that an attack had occurred.
If there is a valid security reason for a cover up to protect other vulnerable systems, I do not mind. I hate that we, as a free society, always have to have our pants down (in comparison to say, China) – but they (DHS) needs to do a better job at it…
This would appear to be relevant: http://threatpost.com/en_us/blogs/hacker-says-texas-town-used-three-character-password-secure-internet-facing-scada-system-11201
Addendum to the pastbin link above: http://isc.sans.edu/diary.html?storyid=12088&rss
http://www.washingtonpost.com/world/national-security/water-pump-failure-in-illinois-wasnt-cyberattack-after-all/2011/11/25/gIQACgTewN_story.html?hpid=z4
It is never ending and I am sure someone will hack the “whole internet” one day and we will all be stuffed. Can you imagine the carnage if “www” went down?
The other explanation is that there was confusion about which facility/server was involved, and both sources are providing information they believe is accurate. It seems that a government agency would be more likely to issue a “non-denial denial,” rather than to say something that could come back to publicly haunt them if proved true later.
I am sorry but it just isn’t passing the smell test. It sounds fishy to me. Did the SCADA controller really wander off and just had to login from Volvograd Russia (hypothetical location) to do this or that thing and the pump fried itself? Be that as it may three things need to be said.
First, you cannot wall off these SCADA systems from the Internet. Most of them are running on Windows and they really do need an AV product on them and they really do need to get their OS, product, and AV updates. Even Linux and Macintosh systems have tremendous volatility these days.
Second, all of these SCADA systems need at least some sort of transparent firewall in front of them. A transparent firewall is like a glorified bridge – no IP addresses. Snort or other methods can be used to contol access. It should be configured to allow in only the locations (IPs or host / domain names) specified and what ports they can come in on. That way when a controller tries to login from Volvograd, Russia they would at least first have to call up and supply the needed information to be allowed in. After they finish doing what they need to do the temporary opening should be immediately closed. What does this do besides provide access control? It eliminates the unknowns down to a more manageable level.
DHS really needs to set up these transparent firewalls and help move them into place with best use outside access practices. Those outside systems would be much better if they have a reduced malware attack vector. What holes have I missed? Who cares? We will be lucky to get them to do this.
While I think a lot of people are jumping to conclusions and believe that Illinois made an honest mistake by jumping the gun on reporting a hack and didn’t fully investigate their logs before blasting the information out.
However, I think everybody is TOTALLY missing the point.
WHY WOULD YOU EVER WANT TO ALLOW EXTERNAL ACCESS TO YOUR SCADA SYSTEMS LIKE THIS? ESPECIALLY FROM FOREIGN COUNTRIES!!!!
If remote access is a requirement, you better have high levels of authentication for access, only permit access from known IP sources and even then only after interrogating said systems to be sure you know to be yours and properly configured and not compromised.
Have I missed anything? I just think its funny that people are only slamming them for an “early warning” and that it was OK some guy on vacation in some foreign country did something stupid.
Barry,
There is a problem with just white listing IP addresses or Blacklisting counties etc.
First of is that the “technical service” personnel “dialing in” may be using one of a myriad of almost constantly changing and even possibly shared IP addresses.
For instance “mobile connectivity” access to the Internet by either a Smart Phone or GSM etc mobile USB device has a significant issue.
The phone network provider has something like a thousand or so Internet users for every IP address they have…
This means that it is quite likely there are 50 or more mobile users behind a single IP address, and that the phone company may give you a different IP address for each new conection you make.
Secondly if you blacklist a country, an attacker is realy not going to find it difficult to get access to a host that can be used as a proxie gateway to put their effective IP address in the “home country”.
Thus both white listing and black listing of IP addresses are at best very very weak access control mechanisms. And some people view them as at worst a significant waste of resources.
Right on, Barry, re your “external access” point. However, plugging that hole in the dike will still leave ample opportunity for someone (else?) to do “something (else) stupid.”
“We have met the enemy, and he is us.” That is, assuming that this whole affair is nothing more than “something stupid.” Let’s all hope that Occam’s Razor applies here.