Amid multiple recent reports of hackers breaking into and tampering with drinking water treatment systems comes a new industry survey with some sobering findings: A majority of the 52,000 separate drinking water systems in the United States still haven’t inventoried some or any of their information technology systems — a basic first step in protecting networks from cyberattacks.
The Water Sector Coordinating Council surveyed roughly 600 employees of water and wastewater treatment facilities nationwide, and found 37.9 percent of utilities have identified all IT-networked assets, with an additional 21.7 percent working toward that goal.
The Council found when it comes to IT systems tied to “operational technology” (OT) — systems responsible for monitoring and controlling the industrial operation of these utilities and their safety features — just 30.5 percent had identified all OT-networked assets, with an additional 22.5 percent working to do so.
“Identifying IT and OT assets is a critical first step in improving cybersecurity,” the report concluded. “An organization cannot protect what it cannot see.”
It’s also hard to see threats you’re not looking for: 67.9 percent of water systems reported no IT security incidents in the last 12 months, a somewhat unlikely scenario.
Michael Arceneaux, managing director of the WaterISAC — an industry group that tries to facilitate information sharing and the adoption of best practices among utilities in the water sector — said the survey shows much room for improvement and a need for support and resources.
“Threats are increasing, and the sector, EPA, CISA and USDA need to collaborate to help utilities prevent and recover from compromises,” Arceneaux said on Twitter.
While documenting each device that needs protection is a necessary first step, a number of recent cyberattacks on water treatment systems have been blamed on a failure to properly secure water treatment employee accounts that can be used for remote access.
In April, federal prosecutors unsealed an indictment against a 22-year-old from Kansas who’s accused of hacking into a public water system in 2019. The defendant in that case is a former employee of the water district he allegedly hacked.
In February, we learned that someone hacked into the water treatment plan in Oldsmar, Fla. and briefly increased the amount of sodium hydroxide (a.k.a. lye used to control acidity in the water) to 100 times the normal level. That incident stemmed from stolen or leaked employee credentials for TeamViewer, a popular program that lets users remotely control their computers.
In January, a hacker tried to poison a water treatment plant that served parts of the San Francisco Bay Area, reports Kevin Collier for NBCNews. The hacker in that case also had the username and password for a former employee’s TeamViewer account.
Image: WaterISAC.
Andrew Hildick-Smith is a consultant who served more than 15 years managing remote access systems for the Massachusetts Water Resources Authority. He said the percentage of companies that reported already having inventoried all of their IT systems is roughly equal to the number of larger water utilities (greater than 50,000 population) that recently had to certify to the Environmental Protection Agency (EPA) that they are compliant with the Water Infrastructure Act of 2018.
The water act gives utilities serving between 3,300 and 50,000 residents until the end of this month to complete a cybersecurity risk and resiliency assessment.
But Hildick-Smith said the vast majority of the nation’s water utilities — tens of thousands of them — serve fewer than 3,300 residents, and those utilities currently do not have to report to the EPA about their cybersecurity practices (or the lack thereof).
“A large number of utilities — probably close to 40,000 of them — are small enough that they haven’t been asked to do anything,” he said. “But some of those utilities are kind of doing cybersecurity based on self motivation rather than any requirement.”
According to the water sector report, a great many of the nation’s water utilities are subject to economic disadvantages typical of rural and urban communities.
“Others do not have access to a cybersecurity workforce,” the report explains. “Operating in the background is that these utilities are struggling to maintain and replace infrastructure, maintain revenues while addressing issues of affordability, and comply with safe and clean water regulations.”
The report makes the case for federal funding of state and local systems to provide cybersecurity training, tools and services for those in charge of maintaining IT systems, noting that 38 percent of water systems allocate less than 1 percent of their annual budgets to cybersecurity.
As the recent hacking incidents above can attest, enabling some form of multi-factor authentication for remote access can blunt many of these attacks.
However, the sharing of remote access credentials among water sector employees may be a contributing factor in these recent incidents, since organizations that let multiple employees use the same account also are less likely to have any form of multi-factor enabled.
A copy of the report is available here (PDF).
Update, 6:25 p.m. ET: Clarified that the report was issued by the Water Sector Coordinating Council (not the WaterISAC).
It is unfortunate that it will take a successful attack resulting in injuries and death to force the water/waste utilities to act to harden their sites and IT procedures. We always rush to lock the barn door after the horses have left.
The threat is so obvious, and yet serious action to thwart an attack is lacking.
The safeguards on our utilities (electrical, water, waste) should be as comprehensive as those on our nuclear weapons. They are of similar importance and consequence.
Everybody makes mistakes …
1) Marie Curie’s Lab in Paris is now a museum and may even see visitors when the radioactivity dies down.
2) Enrico Fermi’s nuclear reactor is no longer under a stadium in Chicago.
3) The Hydrogen Bomb ‘Bravo test’ yield was 250% and that’s the mistake which should never be allowed removal from history.
Agreed. Nothing against computer operators but when they sit at their desks, they could monitor one thing and take a short walk.
All valves for lye, ?, chorine, etc, have to be be analog. They have to be watched and adjusted by a human with redundant analog meters. That human has to be verified by another.
Not a real fix because the two could be of the same ideals. If you look at how exact the chlorine levels have to be an entire village could be wiped out by noon.
In another lifetime, I think it was in the spring of 1997, something completely bizarre happened on the West Coast of the United States. A satellite turned in the air unexpectedly and all of a sudden, gasoline pumps at stations and credit card processing machines at stores and ATMS throughout the western United States randomly didn’t work. At a local restaurant where I was a frequent customer, the owner let me pay for meals, tip the waiter and get extra cash back by writing a check. (In my mind, I recall that this went on for several weeks, but it might only have been days. I was probably 21 years old at the time)
I recall commenting once to this restaurant owner “You know, if someone really wanted to get us or control us, they could pull off something like this – just start randomly turning all of the master switches off and on.” Maybe it’s about to happen now because of all the holes we’ve left unpatched.
Here’s a link to a Washington Post archived news article about the situation I was mentioning. Happened in 1998, not 1997. https://www.washingtonpost.com/archive/politics/1998/05/21/satellite-glitch-cuts-off-data-flow/0502d42b-a46e-4efe-9b00-b23183779903/
Do you have the real internet? Do you know where it is located? Have you checked into ‘lo’ and arp network being http tracked with live nuke codes? Do you want to play A game? Anyways jamesjosephsandoval31@gmail.com if you find the real internet, people are paying for next n pragma cached no x’s
I have a degree in Chemistry and know lots of ways to purify water past any reasonable taste. In the February ice in Texas the dummies on the other side of the reservoir had to boil their water two days longer than I did, if they even had power. Easterners, what can I say …
What alarms me more about good luck and engineering standards is the report that Data Centers contribute to the western drought.
The Municipalities ask: “What did Silicon Valley (IT), Civil Engineers, Energy Companies, etc. know and when did they know it ?” … in 1914 when the Panama Canal opened self-sufficient in water (and therefore hydroelectric power). The Canal remains so to this day.
Not only water systems, look at what has already been touched (that is a diplomatic attacked) by hostile players. Our President has given the hackers a list of “preferred targets,” (those that will cause the most disruption), with a statement “don’t touch,” that has as much meaning as that said by the Vice-President who said “Don’t come.” Both statements, are ridiculous and mean nothing.
I think it will become the “electronic warfare” we have read about in the SF stories. You don’t have to get ready for it—It is here now.
It is unfortunate there are a lot of IS managers who, if you think about it, have not done “Job 1” for the organizations they work for, and just collected their paycheck. By the way, this comment goes up the latter as well as down.
For years, many years, we have heard “backups”. But can you ask how do they work under stress? Can you just plug it in and it will work. Is there a “Plan B” and has it been tried, how often, and how did it turn out?
The best recoveries that I witnessed are those performed by networking folks and not IS.
Based upon some of the larger downtimes we have seen, caused by improper testing, unplanned upgrades, not understanding the real network, and just poor coding—-we have already seen the writing on the wall.
The next few months should tell.
In 37160 middle tennessee they have been extremely busy with variant of algae that the Petro industry uses, modifying it and combining with silica gives the bacteria physical touch for lack of better words, this has been put into our water, just insidious, since it’s physical touch, it’s now possessible with A demonic entity making them so polymorphic and dangerous, this is human experimentation, this is used in conjunction with gov takeovers, of your friends look different and your hometown is under siege, they’re probably sedated in A cryogenic state(professional) or A yr into A hypnotic trance, forgotten about still in heaven
…well yes and no. many people forget or do not know about the power of the local Public Utility Commissions (PUCs) on water spend and thus rates to the consumer…
…many water utilities cannot spend on cyber things unless and until the PUC says they can if it affects rates…
…so even if they wanted to, they can’t…
Thanks for the article, Brian. I live in a small North Georgia county with its own water/sewer works. I’ve got absolutely no idea what the cyber security status is. I’ve forwarded your article to the chairman of the County Commission asking that he contact the water company for a report on where they stand regarding this vital issue. Hopefully the status is OK, but if not maybe your article will prompt some positive action. Clean water is at least as important as electricity, but the electric companies are monster utilities with deep pockets for IT and IT security. Our local water works, not so much.
Scarey! One thought, however…those small water systems might not have too much to worry about since they are small, and there wouldn’t be much bang for the buck for someone hacking them unless its a disgruntled employee.
Regards,
As one can see by some of the comments above, most of these thousands of water companies are way too small to mount, let alone devise and maintain, effective system-wide comprehensive and effective safeguards against a cyber attack of some sort. The only solution is a set of Federal government subsidies that support their implementation. Oh, and one can’t get the subsidies unless they adopt and implement Federal guidelines. What? There are none? Oh my. Well, we have to start somewhere. Thanks, Brian!
Why federal?
State governments have budgets to do this. States even have National Guard cyber protection teams trained and ready to help.
That’s false math and we need it done to a standard. It makes no sense to do it piecemeal.
State governments are the correct bodies to protect our utilities, because “The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people.”
Oh, God, here we go with the atavistic “State’s Right’s” crowd…hey, we had enough of you guys during Re/Deconstruction and the Jim Crow era; kindly go back to your Hoot or Holler.
I moved to a small village in NY in 1981. We have two huge water tanks for emergencies. These tanks had open tops.
It wasn’t fear of terrorism back then but too many people were swimming in the emergency water supply tanks at night. We are a fat nation. Long ago, prime of life in all respects was 30.
In order to swim, one had to scale the tank. No sophisticated gear back them just a hook and rope. Dangerous but a cool swim on a hot summer night was great.
The tops were eventually closed.
This article makes me want to contact our local authorities as it’s a very real threat. Thank you for sharing! Also, would love to connect regarding our Cybersecurity Instructor openings we have here at http://stormwindstudios.com – would love the opportunity to chat with you about full-time or part-time teaching! You can reach me at monica.brown@stormwindlive.com
As important as this is, why don’t some of the IT security companies (or even individual practitioners) volunteer some time/resources to help out the smaller water treatment facilities? It may be as simple as educating them about the information security related pitfalls of their current practices or it could be helping them actually secure their systems. Either way, I think it would be good public relations material for a company or resume material for an individual. (I realize there may be legal implications but those are usually handled by putting together solid written agreements between the involved parties.)
I’d be very nervous about this if I ran a small water treatment authority, or if I depended on one.
Someone shows up and says, “Sure, just let me inside. I’ll inventory all you cyber weaknesses and exposures for you. You can trust me.” I’m not at all suggesting that Dillon or anyone else is a bad guy, just saying it’s hard for Mr. Smalltown administrator to know whether the volunteers they’re letting in are there to help or to take advantage.
Dillon. That is a pretty good idea. I just forwarded the link to the city administrator where I live. I have a vested interest in the water coming out of my tap being clean, fresh, and present. Thinking I can leverage any time I spend on it for CEP as well. Win, win, win.
“The water act gives utilities serving between 3,300 and 50,000 residents until the end of this month to complete a cybersecurity risk and resiliency assessment..”
My suggestion is that these assessments start with reviews to determine if there are air gaps between the control systems and the internet. That remote access is even allowed is inexcusable.
Situation I heard described was one person in a rural community who’s responsible for controlling devices at different locations 30+ miles from their home. The needed to turn on the devices, then turn them back off when a process finished (about 3 hours later). Of course this is in addition to their normal work. So they *could* hire an employee whose only job is to spend a few hours driving to each location to activate the process, and then drive around to turn off the devices. Or they could allow remote access.
I agree that allowing remote access is problematic. The alternative costs money. Lots of it. How much extra taxes are you willing to pay?
Surely a set of Yubikeys is not too much of an expense.
https://www.yubico.com/ca/works-with-yubikey/catalog/teamviewer/
Good article! Where I live, in a crowded metropolis of may millions, our water comes from few sources, but goes through MANY hands in the distribution process until it hits the tap, where it always tastes awful, but that’s besides the point. (We all drink bottled here, and _it_ comes from sources ‘way up in the mountains.)
Now, I wonder what security steps are in place _for those sources_, IT or physical (armed guards, etc.)
Think I’ll drop them a note. Thanks, Brian!
I used to help manage remote access control systems years ago. It was so much more secure when remote access was only by dial-up to a modem, then username/password. Now anyone, anywhere in the world, can scan for certain well known control systems and try to brute force their way in.
If you can’t afford security, then maybe take your self off of anything that is connected to the net. Set up 2 systems, one connected and one internal that never touches anything outside.
Yes it’s in the public interest to safeguard the drinking water, but if you can’t handle it, give it to the Govt to take over. Subsidies? No. More often than not, those end up in the pockets of the higher ups and little goes to what it’s actually intended for.
Or one could make any fines/penalties actually mean something. What good is a $100K fine to a multi-million dollar company? Take it from them because they are putting the public at risk. It’s that simple.
In most (all?) cases, drinking water is already supplied by the government. I think that’s best, but it’s not a panacea.
Why oh why does everything need to be connected to the internet?!
I just finished Bruce Schneier’s “Click Here to Kill Everybody”. Scary stuff! And that was in 2018.
Really Great article!
https://portforwardinghub.com/
states should have come forward to safe life-saving resources.