A recent cyber attack on a city water utility in Illinois may have destroyed a pump and appears to be part of a larger intrusion at a U.S. software provider, new information suggests. The incident is the latest to raise alarms about the security protecting so-called supervisory control and data acquisition system, or “SCADA” networks — increasingly Internet-connected systems designed to monitor and control complex industrial networks.
CNN is reporting that federal officials are investigating the attack, but quoted a Department of Homeland Security official downplaying the incident. Wired.com says the focus of the attack may be the Curran-Gardner Public Water District near Springfield, Ill. The Register quotes DHS’s Peter Boogaard saying the agency and the FBI are gathering facts surrounding the report of a water pump failure, but that “at this time there is no credible corroborated data that indicates a risk to critical infrastructure entities or a threat to public safety.”
The incident was first reported in a state cyber fusion notice dated Nov. 10, and soon was summarized on the blog by Joe Weiss, managing partner of Applied Control Solutions, a SCADA systems security firm. Weiss criticized the lack of response and alerting by the US-CERT, Department of Homeland Security, and the information sharing and analysis center (ISAC) run by the water industry.
Weiss read KrebsOnSecurity sections of the report, which traced the origin of the attack to Russian Internet addresses.
“Sometime during the day of Nov. 8, 2011, a water district employee noticed problems with a SCADA system. An information technology service and repair company checked the computer logs of the SCADA system and determined the system had been remotely hacked into from an Internet provider address located in Russia.”
The alert also indicates that this attack may be linked to a SCADA provider that also serves other industries, in addition to the water sector. From the alert:
“The SCADA system that was used by the water district was produced by a software company based in the US. It is believed the hackers had acquired unauthorized access to the software company’s database and retrieved the usernames and passwords of various SCADA systems, including the water district systems.”
The intrusions apparently took place over several months, during which time the attackers remotely logged into the water district’s SCADA networks and toggled systems off and on, eventually causing the failure of a water pump at the facility.
“Over a period of 2-3 months, minor glitches have been observed in remote access to the water district’s SCADA system. Recently, the SCADA system would power on and off, resulting in the burnout of a water pump.”
The notice also stated that the method of attack appears to be similar to the recent compromise of servers at the Massachusetts Institute of Technology (MIT), which involved security weaknesses around phpMyAdmin, a popular Web-based database administration tool.
“This network intrusion is the same method of attack recently used against the MIT Server,” the water district alert stated. “The water district’s attack and the MIT attack both had references to PHPMyAdmin in the log files of the computer systems. It is unknown at this time the number of SCADA usernames and passwords acquired from the software company’s database, and if any additional systems have been attacked as a result of this theft.”
Michael Assante, president and CEO of the National Board of Information Security Examiners and a former chief security officer for the North American Electric Reliability Corporation (NERC), said the attack highlights the potential pitfalls of utilities increasingly turning to off-the-shelf commercial solutions and remote access to trim costs in an era of tight state and local budgets.
“In smaller districts, you’re not going to have big network architectures [that allows you] to have restricted routing and VPN architecture,” Assante said. “But when we get to smaller water districts, the less infrastructure they can have to do their work, the cheaper it is. And with these current budget restraints for municipalities, Web remote access seems to be the way they want to do business.”
Assante said it was too early to assess the broader implications of this incident, and noted that the initial reporting on cyber-related SCADA incidents often turns out to be inaccurate. But he said that if most of the information in the original report is correct, then there are significant lessons to be learned from this incident.
“You have compromises occurring over remote access, and over months this had effects on the system that were anomalous or never coordinated to a cyber event,” Assante said. “If what really happened here turns out to be 80 percent close to what’s in the original reports, it will be very important to know what we can learn from this.”
The wait-and-see response from the DHS and FBI appears to have encouraged hackers to highlight similar vulnerabilities in other water districts. Several sites now are reporting on a claim by a hacker named “pr0f” who posted a series of images that appear to demonstrate remote access to a SCADA system that is responsible for the water supply in the City of South Houston, Nevada. It’s not clear whether the image noted here is supposed to be for a Texas or Nevada facility, but Fred Gonzalez, superintendent of the City of South Houston, Texas water and sewer division said his organization was still analyzing the information to determine its veracity.
A screen shot posted online today, allegedly from remote access to SCADA systems for the City of South Houston.
Excellent article (as always). I find it interesting that DHS and others are warning all of us that industrial control attacks are on the horizon and then go into denial mode when one actually happens within days of the warning. This all begins to feel like field testing for bigger and far more destructive attacks later. And no one should be surprised they are doing this field testing on the fringes where systems are bound to be less protected. They can hone their skills to get to the bigger prizes later.
You quote a superintendent from South Houston, Nevada.
I’m a bit confused about where that is. It’s not showing up in Google Maps.
Any information? Looks kind of like a bogus name to me.
Yes, it was confusing to me also, and I meant to add that earlier. I’ve updated the story with a clarification. Thanks.
Brian,
It looks like this is indeed South Houston and Nevada may be the location of the system / facility (i.e. on Nevada street). The other photos also what appeared to be location names in the title as well.
For example…
http://i42.tinypic.com/1znptuu.png
Texas and Pecan, most likely are referring to Pecan Drive and Texas Street.
What’s up with this article link, Brian? I click the ‘read more’ link and get redirected to the mobile version of the site’s main page. I was only able to read the rest of the article by clicking to view comments.
I’m not entirely sure, Jimmi. Been re-arranging some things on the site this past 24 hours, to counter a 20k+ bot -strong DDoS attack on my site. Apologies for any difficulty you may have had.
Brian,
It is of note that the attack was by the theft of improperly secured credentials at a third party (ie just like the RSA hack).
This realy should be a wake up call to the support and security industries that these databases of credentials whilst essential to carry out work, MUST not be accessable from any publicaly connected network.
I don’t know why anyone is suprised that this has occured. Dick Clark who used to be the Cyber Czar for the White House sounded the warning that the possibility of cyber attacks against critical infustructure of the U.S. was a real threat and that the Gov’t needed to take the initative to get ahead of it before the Gov’t had to find out after the fact. And the funny thing is Weiss was quoted in the same story by an interviewer about taking down the portion of the power grid. the interviewer asked him :
Interviewer: So it’s more whether in fact one could mess with a system so it basically could cause enough damage to shut it down?
Weiss:Between in the plant or in the switchyard.
Interviewer: So just put it in all perspective. What’s the worst-case power scenario, power we’re talking here — power lines, power grid?
Weiss: So just put it in all perspective. What’s the worst-case power scenario, power we’re talking here — power lines, power grid?
Interviewer:Over how big an area?
Weiss: Big as you want.
Interviewer: Is that a possibility?
Weiss:Yes.
Interviewer: How?
Weiss: I’d just as soon not go into it.
Interviewer: But you believe, as an expert, a man who understands these systems, that indeed that is a possibility?
Weiss: It’s possible.
Interviewer: Why isn’t Washington quaking in its shoes?
I can’t tell you. I don’t know. I don’t know.
Interviewer: Have you gone to Washington and raised your hand and said, “Folks, this is a problem?”
Weiss:I have tried — again, I have to be careful how I say this. When I was at EPRI, I was not supposed to have communication with those people. I did, because I thought it was important, and in the end, it didn’t help my career. I have contacted any number of Senate and congressional staffers that I thought were relevant. I’ve tried to talk to a few. I’ve invited them to our conferences, to not only see what was happening, but talk to people and gain an understanding of what they knew and felt.
The other thing that I think is important, too, is to realize as big as our industry is and how few people attend or are even aware of this. I was just at a conference two days ago in Houston. We had a security session. It was for the generators. We had two senior vice presidents of some of the largest insurance reinsurers in the country and another consultant and myself, and I think there were 20 people there.
Interviewer: A lack of concern?
Weiss: A lack of understanding. Lack of concern happens when you understand it and then decide it’s not important. We’re at the point where I don’t believe most people understand what can happen.
—–
Hence why Clark and several other sound the alarm about this years ago, The Gov’t always seems to see things in black and white, with no grey area…Well this is a big a grey area as you want to see. One would hope they learn a lesson and realize this is more than a credible threat, it is real.
It’s actually more complex than that. Clark & many others “sounding the alarm” are sounding both improbable & reasonable alarms. Clark is part of the movement drumming up fear of “cyberwar,” requiring government “cyberdefense” and “cyberweapons.” The truth is that a computer connected to the Internet with shoddy remote access or security can be hacked. It’s just a computer security problem. Additionally, the US government recently made the worst move of all by deciding to not classify certain SCADA weaknesses as vulnerabilities.
https://threatpost.com/en_us/blogs/dhs-thinks-some-scada-problems-are-too-big-call-bug-092611
The US government also continues to try to get infrastructure providers to let them backdoor & control their systems for “security.” Ominously called “Perfect Citizen.”
http://www.wired.com/threatlevel/tag/nsa/page/2/
To top it off, there are existing high assurance systems that can be used for SCADA security & others, but they’re restricted to the public (Boeing SNS) or effectively killed by red tape (GEMSOS). Common Criteria EAL6/7 systems are considered “munitions” with serious export restrictions. Many A1/EAL7-class vendors went out of business because of that and that US gov.t started buying insecure systems in mass.
Essentially, they’re forcing American individuals & businesses to use hackable OS’s. Most probably, so that *they* can hack us too if they feel it necessary for “national security.” That they’re also stockpiling zero-days through 3rd party contractors supports this idea.
Mark my words: the US government CANNOT be trusted to handle infrastructure security. Their actions so far suggest they want control & surveillance, not security. Regulations that mandate tried & true INFOSEC practices or liability legislation for damage caused by negligence is a better route. They should also remove export controls for EAL6/7 class systems. If they can sell our ally a F-35 jet, then I should be able to sell them a firewall or VPN.
If one actually looks at Government and its role in cyber security, you have to ask yourself how much control do you want your government to have?
Next explore the authorities that federal agencies have.
Finally consider that any organization involved in cyber is always asked, “what are you doing about it?” When one sole searches most incidents there is not much substantive or truly meaningful actions that can be taken.
Dick Clark, Really?
Why give him credit rather than DHS?
Thank you, Mr. Krebs, for your excellent article on this topic. The Water Infrastructure Security Enhancements (WISE) project of the ASCE has been working on this topic for the past 8 years. A similar water system hack was reported in Maroochy Shire, Australia in 2000. That event was precipitated by a frustrated employee who used a laptop and radio equipment to hack into a radio-controlled SCADA system. I agree with others who think this was a “probe,” and that other events will likely be detected. Be Safe, Chuck Stack, MPH
I’m not sure what to think of this… seriously, it’s always either the Chinese or the Russians. I think someone must get over something about their past if that someone want’s to have a present and a future. The nature of this kind of reports is always a bit fishy. In my view this is another small contribution to justify for the heist of Internet freedoms. I mean, anytime there is a hack in the U.S. me and my friend can flip a coin and choose tails for Chinese or heads for Russians. It’s 99% guaranteed one of us will be right.
You’ll soon need a coin/dice with three sides. They were recent reports over alleged iranian hackers successfully compromising some certificate authorities. But you have a valid point, nevertheless.
Sure, I agree. Anyway, it was not meant literally. Most of the time we deal with allegations which never convert to concrete proof and are never officially explained away but they sometimes slip into our subconsciousness and become something of a fact, whether we like it or not. Then, we (generally speaking) adopt ideas based on those nonfactual events.
Just to clarify, the article did not say that the Russians or Chinese perpetrated this attack. It said that the attack came “… from an Internet provider address located in Russia.”
It’s not beyond the realm of possibilities that the machine responsible for the attack was compromised and controlled elsewhere. It’s also not beyond the realm of possibility that the Russian IP space or machine were hired out to lord-only-knows-who.
So you’re basically saying that the geolocation of an IP address is basically useless. Thank you. I guess…
Because of HTran and numerous proxy products, Geo Location is almost useless.
Of course. I was just emphasizing the fact that the official report states that they traced the origins of the attack to a Russian internet address about which they don’t give (or have) any details, which is for some means everything but for others means nothing. As far as they “know”, and technically as you said, it may have originated from the nearest Wi-Fi connection to the Water Utility or any other place in the world.
It’s just that, in general, it’s simply understood as “Russian hackers hack into U.S. Water Utility” when in reality it is probably far from the truth. We already know that a serious hack is always bounced so that it traces to the wrong location and probably very far from it’s geographical origin.
Before I submitted my comment I did a simple Google Search with the keywords “russians+hack+water”, and of course that people should always be aware from where they get their news but media always twists and goes up a notch, for example, as a result a layman will read stuff like this:
http://www.popsci.com/technology/article/2011-11/russian-hackers-breach-springfield-illinois-water-utility-destroy-pump
Technically, we know it’s quite a step to put on a headline like that, but generally people will literally interpret that something like a Cyber-Cold War is going on and “we need to put a stop to those commies’ intrusions by any means necessary”. Anyway, that’s a totally different conversation so I’m not going to ramble any further.
The part that blows my my mind; is every day people read and watch on the news about some new cyber attack then turn around a go I don’t need any kind of computer security. We can’t afford to spend any money on this. It’s a lot cheaper to spend money to prevent this than after the fact.
John,
The cost is dependent on your viewpoint.
If you consider there are high probability events medium and low probability events and rare events you can asign a protection cost to them.
A sensible person devotes some time organising these events into types of outcome or classes of failure and then mitigates against the most likley single events and classes of event.
You then try to come up with a mitigation strategy to account for the rest of the events known or unknown.
Currently genuine attacks against SCADA systems are very very rare events even including Stuxnet.
Thus at this point it has not been cost effective to spend money on mitigating against such attacks on the equipment only on the perimiter.
I suspect that is now changing.
A previous article on SCADA vulnerabilities being hidden and played down instead of fixed.
https://www.schneier.com/blog/archives/2011/05/new_siemens_sca.html
Brian,
Good job reporting on this.
This sounds like a typical false flag operation. It seems too convenient that it was traced to Russian internet addresses. Our government says “Oh, watch out for cyber terrorism,” and then we get some.
And what do the alleged plan on doing with that control? Everything can be controlled manually right at the equipment itself. There’s very little likelihood of overfeeding a chemical, before you manually shut the system down.
I guess the threat is actually shutting the system down. But there are so many water plants (some cities having 3 or more), that would be quite an undertaking to make a big enough problem.
This sounds like our own government doing this to instill even more fear in the people and create more security measures and more restrictions on our freedoms.
Frank,
This was reported in the local media in Springpuddle, The Sj-r.com all the local rags like the New Berlin Bee. This happened in an area where people don’t keep secrets and where there is little federal presence.
The open source info is huge on this. There have been pictures of equipment and all kinds specific references to affirm that there was a cyber intruder; and there was an equipment failure. I suspect one could question about whether the pump failed solely due to intruder activity.
Brian, you wrote that you posted a clarification on the location of the South Houston incident, but I didn’t see it. Anyway, I think you will find that Nevada and Virginia are names of South Houston, TX water facilities based on the street names.
The Joe Weiss blog link seems to have disappeared on us.
I think you have an extra dash. The link:
http://community.controlglobal.com/content/water-system-hack-system-broken
Got it. Thanks, Hhhobbit.
My friends in the security sector (DHS etc.) have gone totally dark on this one. Wassup?
Methinks there is more to this story than is being let on. Keep your eyes peeled, I’ll be back if I hear anything. This is very strange.