In late May, KrebsOnSecurity alerted numerous officials in Florence, Ala. that their information technology systems had been infiltrated by hackers who specialize in deploying ransomware. Nevertheless, on Friday, June 5, the intruders sprang their attack, deploying ransomware and demanding nearly $300,000 worth of bitcoin. City officials now say they plan to pay the ransom demand, in hopes of keeping the personal data of their citizens off of the Internet.
Nestled in the northwest corner of Alabama, Florence is home to roughly 40,000 residents. It is part of a quad-city metropolitan area perhaps best known for the Muscle Shoals Sound Studio that recorded the dulcet tones of many big-name music acts in the 1960s and 70s.
Image: Florenceal.org
On May 26, acting on a tip from Milwaukee, Wisc.-based cybersecurity firm Hold Security, KrebsOnSecurity contacted the office of Florence’s mayor to alert them that a Windows 10 system in their IT environment had been commandeered by a ransomware gang.
Comparing the information shared by Hold Security dark web specialist Yuliana Bellini with the employee directory on the Florence website indicated the username for the computer that attackers had used to gain a foothold in the network on May 6 belonged to the city’s manager of information systems.
My call was transferred to no fewer than three different people, none of whom seemed eager to act on the information. Eventually, I was routed to the non-emergency line for the Florence police department. When that call went straight to voicemail, I left a message and called the city’s emergency response team.
That last effort prompted a gracious return call the following day from a system administrator for the city, who thanked me for the heads up and said he and his colleagues had isolated the computer and Windows network account Hold Security flagged as hacked.
“I can’t tell you how grateful we are that you helped us dodge this bullet,” the technician said in a voicemail message for this author. “We got everything taken care of now, and some different protocols are in place. Hopefully we won’t have another near scare like we did, and hopefully we won’t have to talk to each other again.”
But on Friday, Florence Mayor Steve Holt confirmed that a cyberattack had shut down the city’s email system. Holt told local news outlets at the time there wasn’t any indication that ransomware was involved.
However, in an interview with KrebsOnSecurity Tuesday, Holt acknowledged the city was being extorted by DoppelPaymer, a ransomware gang with a reputation for negotiating some of the highest extortion payments across dozens of known ransomware families.
The average ransomware payment by ransomware strain. Source: Chainalysis.
Holt said the same gang appears to have simultaneously compromised networks belonging to four other victims within an hour of Florence, including another municipality that he declined to name. Holt said the extortionists initially demanded 39 bitcoin (~USD $378,000), but that an outside security firm hired by the city had negotiated the price down to 30 bitcoin (~USD $291,000).
Like many other cybercrime gangs operating these days, DoppelPaymer will steal reams of data from victims prior to launching the ransomware, and then threaten to publish or sell the data unless a ransom demand is paid.
Holt told KrebsOnSecurity the city can’t afford to see its citizens’ personal and financial data jeopardized by not paying.
“Do they have our stuff? We don’t know, but that’s the roll of the dice,” Holt said.
Steve Price, the Florence IT manager whose Microsoft Windows credentials were stolen on May 6 by a DHL-themed phishing attack and used to further compromise the city’s network, explained that following my notification on May 26 the city immediately took a number of preventative measures to stave off a potential ransomware incident. Price said that when the ransomware hit, they were in the middle of trying to get city leaders to approve funds for a more thorough investigation and remediation.
“We were trying to get another [cybersecurity] response company involved, and that’s what we were trying to get through the city council on Friday when we got hit,” Price said. “We feel like we can build our network back, but we can’t undo things if peoples’ personal information is released.”
A DoppelPaymer ransom note. Image: Crowdstrike.
Fabian Wosar, chief technology officer at Emsisoft, said organizations need to understand that the only step which guarantees a malware infestation won’t turn into a full-on ransomware attack is completely rebuilding the compromised network — including email systems.
“There is a misguided belief that if you were compromised you can get away with anything but a complete rebuild of the affected networks and infrastructure,” Wosar said, noting that it’s not uncommon for threat actors to maintain control even as a ransomware victim organization is restoring their systems from backups.
“They often even demonstrate that they still ‘own’ the network by publishing screenshots of messages talking about the incident,” Wosar said.
Hold Security founder Alex Holden said Florence’s situation is all too common, and that very often ransomware purveyors are inside a victim’s network for weeks or months before launching their malware.
“We often get glimpses of the bad guys beginning their assaults against computer networks and we do our best to let the victims know about the attack,” Holden said. “Since we can’t see every aspect of the attack we advise victims to conduct a full investigation of the events, based on the evidence collected. But when we deal with sensitive situations like ransomware, timing and precision are critical. If the victim will listen and seek out expert opinions, they have a great chance of successfully stopping the breach before it turns into ransom.”
While I’m pretty sure that KrebsonSecurity cannot recommend specific companies or services, I wonder if a list of good questions would help leaders (including IT staff) of municipalities, NGOs, and other organizations to assess the services offered?
1. A quick web search found many (too many IMHO) articles about this, starting with a CERT article from Aug 11, 2019:
https://www.us-cert.gov/ncas/tips/ST19-001
To my inexperienced eyes, this looks purty good. Is it?
2. One point in ST19-001 bothers me. In answer to the question, “What do I do if my computer is infected with ransomware?” they offer the following advice for home users:
“Immediately contact your local FBI office or local U.S. Secret Service office to request assistance.”
To me, it seems unlikely that the FBI or the U.S. Secret Service would actually help a home user with a ransomware attack.
I think Mr. Wosar’s comments get to the root of the issue. Hopefully the data is backed up. But aside from that, the best advice has always been to nuke the systems and reinstall. That’s still good advice here, it’s just too much trouble for many organizations.
The feds generally are not interested in getting involved in warning individual organizations, and they certainly don’t do anything to help home users.
Can someone kindly explain to me how a security firm in Wisconsin can “see” what’s happening with regards to an attack inside a network in Alabama?
Bad OpSec by the criminals is one way. They get drunk they start bragging and think that because they’re doing it in their native tongue that nobody will know. Another way is to compromise the means of communication between members.
I’m not involved with this event and would love to know this too. In addition to the bad opsec comment mentioned, some security vendors have their own honeypots/honeynets that receive traffic from compromised machines. Some have also infiltrated environments used by bad actors.
@Jill, I’m curious too, but I think your question demonstrates why we won’t get an answer.
While it’d be fun for Brian to tell us where the flaws are in the bad guys’ OPSEC, there’s no way he can tell us without telling them what holes they need to patch or precautions they should be taking.
I didn’t see any indication that the WI security firm was on the Florence City network. They apparently monitor the dark web. (Perhaps hacker groups, data samples for sale, etc) With analysis, they likely deducted the city of Florence.
Most likely, they are monitoring C2 (Command and Control) traffic to known IP addresses being used by the criminals. They would have identified the “from” IP address and research would allow you to reach back and identify it as the Windows 10 machine back in Florence.
Most likely in forums/IRC/discord.
Well, thanks for tryin’, Brian!
Now, it’s likely IT matters will get a faster track through the bureaucracy.
Expensive lesson!
More likely the Florence leaders will treat this as a ‘one-time issue’, their IT managers will claim they have their act straightened-up and they’ll all go back to pretending that viruses are not real. We’re talking Alabama after all…
So now that we‘re talking about Alabama, instead of being a richardhead about it why don’t you explain yourself? Surely you’re not implying stupid IT guys only exist in Alabama. Correct? Or maybe you’re attempting to deflect your own stupidity.
I work for a government agency in the State of Florida and this kind of stupidity is rampant. They were warned and had a chance to stop the ransomware before it got started by unplugging their systems from the internet, but government bureaucrats can’t make bold decisions like that. When you are in government you are cowed into not making decisions without lots of meetings and various “permissions” given as doing otherwise could get you in trouble. I see it all the time. If you don’t want to be labeled a “troublemaker” or blackballed or fired, you keep your mouth shut and go along with decisions of senior management who have limited intelligence. Waiting for the city council to approve funds or make a decision is so typical. We waste millions because of that type of management. Nobody is allowed to make important quick decisions on their own and when someone does try it, they usually make stupid ones that are then covered up.
Best reply yet…
Steve. Is the private sector any better in the final analysis? The pressure and coercion aren’t identical, but haven’t there been even more spectacular screw-ups by corporate entities?
“Is the private sector any better in the final analysis? The pressure and coercion aren’t identical,”
The pressure and coercion I’ve seen in the private sector are identical to the public sector. If you tell the truth, you not only get fired, you get ____balled by other employers too.
(Off topic for this blog, but let’s be aware that words like blackballed maybe aren’t appropriate any more.)
The origin of the term blackballed has nothing to do with race, and so I see no reason you should suggest it is somehow no longer appropriate to be used.
What is “stupid” about not having that priority? Not knowing? You expect everyone to know what the manager or the head of the it department knows. Then there is no need for an it department, or overt security. Because everyone is on the same page. But, not everyone is on the same page. Some are fast reads, or skipped that page for the juicy document following, or they were more interested in the budget. For a city, what kind of a budget would be needed, for the”best” it? Could they even afford it? Remember, it has to be an A+ version, not a b or c grade. Like they had, and it has to be affordable to the taxbase, otherwise it will not be done. Kinda like, bad laws.
Seems to me that Krebs has covered this issue a while back. The root of the problem isn’t that the city has a limited budget that needs approvals issued in order to enact emergency responses, but has more to do with the structure of the administration. The IT manager does not sit on the board, and has to make requests of his superiors, when what should have been happening is that the IT manager should have already been in a position to approve transfer of funds. But having the job title include the word ‘manager’ tends to show how high up that position was placed. They need to create a position more like ‘chief cyber-security officer’ in charge of all IT managers. Basically, that fact the guy had to ask for funding was itself at the core of the problem. He was never given the authority to do his job 100%, and he doesn’t sit next to the mayor in order to report on the current state of IT health
You wouldn’t believe people go nuts about things like configuration control when the systems are crashing around their ears – but they do.
At least you tried!
I dunno why they didn’t use 2-factor authentication
The problem is that 2FA is already being compromised. If a user chooses to let a website or system remember their login in for a period of time (so that they don’t have to log in with 2FA every time), hackers are already able to use that “remember me” token to by-pass 2-factor.
Bad actors are good at what they do for sure.
That’s not really a 2FA flaw. That is authentication session management.
Remember your browser works just as well with passwords, OTP, hardware tokens, etc.
2FA doesn’t solve everything. And doesn’t even have anything to do with a lot of the issues.
I believe this was a phishing attack. Don’t know if it also involved credential stealing (which 2FA may protect against), or if a malicious attachment was involved.
@Steve, you hit the head on the nail. If you are the Subject Matter Expert and they don’t take your recommendations, they should expect to be owned.
“… by a DHL-themed phishing attack…”
Really? In this day and age supposedly smart people STILL click on those?
It’s not a one click compromise. It’s usually a zip file that needs to be opened. That situation is covered in the first 15 minutes of security training. For an “IT Manager” to fall for it is a sad statement on the qualifications of their IT team.
Not always. This month’s patches includes fixes to vulnerabilities that require nothing more than files opening in the preview pane.
I suspect the IT manager logs on to his workstation with the same credentials they use to manage the servers. Possibly also log on to their local workstation with admin privileges.
I’m the CEO of my company and we receive DHL packages all the time, since our main vendor is in Europe. Secondly, I was on vacation in New Caledonia once and wanted to ship something home and could only find a DHL office on the island.
Not everyone lives in the USA and DHL dominates in many areas. We actually HAVE to click on DHL emails to pay customs and import/duty/tax, and I have been warming my employees for months since we started seeing DHL related fake emails.
So yea, they’re kind of a big thing.
It wasn’t about being themed on a DHL styled email, but that it was a phishing attack
I would guess that DHL has protocols for how they contact people via email, and likely that they would never send files of any sort
Any IT security department should have verified how DHL would contact anyone in the organization, along with anyone else who has established repeat business
I’ll also add that basic email checks (SPF/DKIM) in their mail system would have prevented these emails from getting through in the first place if they were spoofing the DHL domain. And if it was a DHL email coming from a non-DHL domain, it should have been pretty obvious to an IT manager that it was spam.
I’m curious as well how Hold knew about the attack. They may have saw it on the Dark Web, it’s a service they offer.
https://holdsecurity.com/solutions/dark-web-monitoring.
Presumably Yuliana finagled their way into a Dark Web consortium that trades in these vulnerable accounts, and downloads compromised account lists to fire them against the relevant domain for activity.
Let me guess. The communication with the City Counsel and various security companies took place over the compromised email system. Bad guys knew exactly what steps were being taken and when.
Even more obvious when the trigger was pulled just before getting a second, and presumably more experienced, security company involved.
I have often wondered why ransomware attacks cannot be traced back to their source. If they can find us, why can’t we find them? Why is the Internet so vulnerable to bad actors?
It’s frequently less a matter of knowing who did it than bringing them to justice. The CrowdStrike report linked to in this story makes quite clear we know a great deal about the group behind this gang, and they have a long and storied history of stealing hundreds of millions of dollars from businesses working from the safety of their mother country Russia.
https://www.crowdstrike.com/blog/doppelpaymer-ransomware-and-dridex-2/
See also:
https://krebsonsecurity.com/?s=%22business+club%22&x=0&y=0
Perhaps if the US government ceased committing terrorist actions against the governments of countries like Russia, China, and North Korea, they might be more cooperative? Just a thought.
Haha. Hey, Brian, I didn’t know that you had Russian drones trolling here as well.
Oh, they can be traced quite easily. Most all of the strains of ransomware has to communicate with the command center that is controlled by an attacker. The problem is that even if you traced it to some city in Russia, what are you gonna do next? Authorities there wouldn’t care, or they may be even encouraging this behavior.
Hope word gets around that when Brian Krebs reaches out by telephone to give a heads-up about your IT system being compromised, the warning won’t be ignored or rolled over into voicemail.
Half of governments notified would just call their cops on whoever was trying to warn them.
@simkin said: “Half of governments notified would just call their cops on whoever was trying to warn them.” What cops? They were “Defunded”.
Of course there are resources that specialize in helping government response to cyberattacks, Infrguard and regional cyber security partnerships with CISA.
Emr -isac is a nice start to finding dedicated staff and free training for tight budgets. Sometimes additional voices can help persuade the decision makers.
Well when Brian speaks they should listen. It was down right very irresponsible of them to pass the buck. I know maybe not everyone reads Brian’s blog but if someone is trying to help you at least listen.
The best and brightest works for city government! Remember the Tuttle city manager?
Years ago (before I knew what I was doing), one of my Linux boxes was broken into. It was a unprivileged account so no lasting damage was done but, during the forensics, I discovered a file with the results from a scanner the criminals ran on my system. This file contained several IP addresses of what I later determined to be routers for a major hotel chain. No big deal until I discovered the routers had no password on the admin account. I was able to log in with ease.
I tried to contact the IT department for the hotel chain but they wanted nothing to do with me. Even their ISP wasn’t interested in the information. It was frustrating as heck but there is little a conscientious person can do in the face of dimwitted apathy. Sometimes you just have to throw up your hands and concentrate on the stuff you can do.
“. . . credentials were stolen on May 6 by a DHL-themed phishing attack” Some peonic minions will just never learn, will they? Oh, sorry, it was the Florence IT manager. My bad.
just because they have a title of ” it manager ” does not mean that he/she know anything about it .Could be a friend of someone in the government who know how to turn on the pc and log in to facebook
I’m actually surprised that those idiots didn’t start blaming Krebs for this ransomware infection. It’s pretty dangerous for him to call city officials like that too.
Man, it’s so maddening to read. When will they learn that once your data has been compromised, it is almost 100% guaranteed that it will show up on the dark web whether you pay the ransom or not.
If I was a tax paying citizen of this municipality I’d be questioning my government’s misuse of public funds. Tax money is not collected to send to oversea (or other entity) black mailers. There is liability and responsibility that the city/local government has and is not taking. That is just one important question that should be asked.
…”but i could have a delivery coming from dhl that i didn’t expect”…
…i believe pt barnum said “there’s a sucker born every minute”…
I bet Steve price gets to keep his job.
He got phished by a DHL Shipment spam email?
He should be fired on principle for that alone…
Honestly, that was my biggest reaction here. The IT Manager got phished by a DHL attachment? Seriously?
If it were a low level employee, I’d call him/her ignorant and probably blame management for not stressing how incredibly stupid it is to open attachments from delivery services. But an IT manager?
He should be fired and never allowed to work in IT again.
Unless he routinely got DHL notices, and part of his job was to review them.
To the best of my knowledge DHL does not communicate to clients via attachment. They use links. I believe this is true for FedEx, UPS, etc. In any event, an attachment would be very rare, and require additional confirmation before opening.
So, the attacker can also add an obfuscated hyperlink to the message to simulate the DHL links he would be used to.
The message should have probably been flagged by an email security system as spoofed since UPS, FedEx and DHL emails are so often phishing attempts.
It should be easy enough to detect that the message was not actually sent from. DHL.
Lots of city government organizations, etc, in various states, running on a mail service provider address–184.154.61.162–used by the Florance AL government. The address is running governments, including sheriff departments, etc from WI to FL to CA.
Malicious communications:
http://www.virustotal.com/gui/ip-address/184.154.61.162/relations
Many or most of these hacks only work because of phishing or social engineering of Managers or employees.
So it seems that one way to avoid this would be to lockdown emails. All emails would be routed to a trained phishing team on a secure site that would review them before passing any legitimate ones on to the recipient.
What?
That would literally destroy any business.
There are automated solutions provided by large cybersecurity firms. No humans, so they analyze email fast. But still not perfect.
In any business, for email, there is really no tolerance for false positives. Missing emails is a quick way for people to get fired and to scrap an expensive solution that was just installed.
I received a patent last fall for a method that will stop an attack before it starts—believe it or not. But I am having a very hard time getting money to complete a product or any interest in licensing. Hard to contact the people in the security business, and they don’t respond at all. No interest in even asking to read the patent. And I have had no more luck than Krebs talking to governments.
They won’t tslk to you on the phone, so, send them an email stating you are a King, and need some money tranferred to their account. Nice way to get a conversation going…….
@Bob:
What is the patent #?
Patents are public records and can be viewed online. So if you’d share it (and save the trouble of searching for it), interested parties could view it and assess its value to them, either as an investment or a product/service to offer.
That is, of course, if this is for real. Right now, yes, your post sounds like it’s coming from Nigeria or Eastern Europe.
Well, I’m in Oakland CA . I only provide the patent number to the people I contact if they ask. A simple test of actual interest. Also, first patent, almost 30 years ago, which was the first intuitive method of inputting Chinese characters without resorting to western alphabet was
Well, it’s coming from Oakland CA. My attitude is that if they are not even interested in responding, they would not read it. And it’s not like I usually am so lucky as to have their email address. They are well fortified these days, unlike 25 years ago when I managed to talk face to face with two of the largest companies in the industry about licensing my first patent, which was for the first intuitive method for inputting Chinese characters without having to resort to western alphabet. They stole it. I made some money anyway, but not a lot. Happy to send the patent or patent number to anyone who can give me their email.
You might consider that this is NOT the way things are done in cybersecurity. Companies want to buy an actual product or service… not help you develop one.
Are you calling venture capitalist firms, or other investors? End user companies aren’t investors… and if you need help to develop a good idea, then you’ll have to give someone else a piece.
There is a phrase in security, “don’t roll your own”. Specifically referring to cryptography, but applicable across the industry. New ideas that have not been tested for decades are extremely dangerous
Is your patent for a piece of hardware? As in the circuitry or mechanical components itself. Or some “Process”?
Or it is software, code, or a design implementation… if so, why don’t you have a “copyright”??? That is the type of intellectual property that would protect your idea. Not a patent.
Also, there are a lot of fraudsters who talk about patents. They work on the general public’s ignorance of the patent process. There is a misconception that patents validate the working function or efficacy of the idea. Having a patent is just having a detailed record of an idea. The existence of a patent is no indication of whether it is a good idea, or even works. In the relatively young and dynamic field of cybersecurity, the US Patent Office often cannot even properly screen patent applications for ideas already in common use.
“Patents do not actually grant a right to practice the claimed invention. Rather, patents just give you a right to exclude somebody else from practicing the claimed invention.”
This is a very important point. This means that some disreputable people make a business out of patenting common ideas that they can sneak past the patent office, then find companies who already implement this common idea, and sue for infringement or extort for licensing.
This might be the reason nobody will talk to you Bob.
I had grown weary of all these ransom attack stories in the news, but this one took the cake! Thanks Brian for making this tired old subject way more interesting!
Hi JCitizen,
Can you please email me at
djstuder@gmail.com
I’ll respond directly.
As mail admin, I’ve seen this DHL stuff for years. I finally filled out DHLs submission form complaining about using a form to report phishing. They provided me with this address to send phishing emails to:
Security.Support@dhl.com
I received a response today from one I sent yesterday. Not too impressed with their enthusiasm/interest in investigating or stopping them but at least I can put the phish in their hands.
Now the problem is my mail server (Icewarp) AV intercepts in/out infected attachments (incoming lands in my ‘bucket’ account for review) so I have to carefully c/p the mail header info to get it to DHL.
DHL.com has appropriate SPF/DMARC records set up. There’s nothing more they can do to stop other people from spoofing their email address. It’s now up to you to ensure your email system checks DMARC and SPF records to filter out spoofed phishing emails.
…it’s not dh’ls problem…they never send attachments…never…
Bob: suggest you try a Go Fund Me account to make the patent operational.
Once again, a lack of training/good security procedures started the infection–even with a high level employee. I learned long ago to right click on a link and see if it agreed with the written link, right click again and scan it with my antivirus and another right click on the link on the link and scan it at Virus Total. Another right click and Save As will often even save the malware file for further security analysis unless there is a redirection involved or the malware guys have prevented a download. Is that so hard?
Regards,
The worst part of this story is hearing that they now plan to pay the ransom “in hopes of keeping the personal data of their citizens off of the Internet”. Talk about false hope.
My personal and banking information was in that system that they compromised. What steps do I need to take to make sure that I am protected? I do not trust the hackers! They still might sell our information to others even though Florence city is paying the ransom. That’s just doubling up their profits.
My personal and banking information was in the system that they compromised. What steps should I take to protect myself?? I do not trust that the hacker still won’t sell our information to others, even after Florence city pays the ransom. That’s just doubling up their profits!
Security 101 – lock the door at night. (prior to the Internet)
102 – when you allow access to computers, someone can rob you blind from anywhere.
103 – drilling a hole in the bottom of the boat to let out the water can back fire. relates to 102.
Like a reply above, I tried in vain to get 2 companies to act; Ransomware, aka Firefox blue screen. Company 1 would only delete the URL registered name daily without acting on the person doing the registering (went on for months)(foreign company with office in Mass) and company 2, the ISP providing the ransomware software server space (in Florida), would not release any “confidential” information.
I’m confident a lot of people paid a lot of money.
Me, I haven’t been able to do a restore for 3 years now (no shadow copy of register) cause I too did a ‘update your browser’ from a phony web popup.
This story is about Florence. A good story. If you want to go off on a tangent, the Equfax theft had Congressional hearings. Outcome? Free credit monitoring for a year. (cost? 3 cents a person. LOL) Yet the stolen data is now used to apply for unemployment and stimulus and 1040 tax refunds. How does the government verify the data entered is real? They check it matches Equifax database. Millions have gone out on debit cards (same verification upon receipt/use.)
I know what could have been done, but Congress is no smarter than the employees in Florence, and listens about the same.
30 years in the business, but something about a dead horse…. Oh how I’d like an email someone who cared, or accept that it’s not too late to plug that hole in the boat which let the water in. Equifax will resist cause they make $ selling boats.