In late May, KrebsOnSecurity alerted numerous officials in Florence, Ala. that their information technology systems had been infiltrated by hackers who specialize in deploying ransomware. Nevertheless, on Friday, June 5, the intruders sprang their attack, deploying ransomware and demanding nearly $300,000 worth of bitcoin. City officials now say they plan to pay the ransom demand, in hopes of keeping the personal data of their citizens off of the Internet.
Nestled in the northwest corner of Alabama, Florence is home to roughly 40,000 residents. It is part of a quad-city metropolitan area perhaps best known for the Muscle Shoals Sound Studio that recorded the dulcet tones of many big-name music acts in the 1960s and 70s.
Image: Florenceal.org
On May 26, acting on a tip from Milwaukee, Wisc.-based cybersecurity firm Hold Security, KrebsOnSecurity contacted the office of Florence’s mayor to alert them that a Windows 10 system in their IT environment had been commandeered by a ransomware gang.
Comparing the information shared by Hold Security dark web specialist Yuliana Bellini with the employee directory on the Florence website indicated the username for the computer that attackers had used to gain a foothold in the network on May 6 belonged to the city’s manager of information systems.
My call was transferred to no fewer than three different people, none of whom seemed eager to act on the information. Eventually, I was routed to the non-emergency line for the Florence police department. When that call went straight to voicemail, I left a message and called the city’s emergency response team.
That last effort prompted a gracious return call the following day from a system administrator for the city, who thanked me for the heads up and said he and his colleagues had isolated the computer and Windows network account Hold Security flagged as hacked.
“I can’t tell you how grateful we are that you helped us dodge this bullet,” the technician said in a voicemail message for this author. “We got everything taken care of now, and some different protocols are in place. Hopefully we won’t have another near scare like we did, and hopefully we won’t have to talk to each other again.”
But on Friday, Florence Mayor Steve Holt confirmed that a cyberattack had shut down the city’s email system. Holt told local news outlets at the time there wasn’t any indication that ransomware was involved.
However, in an interview with KrebsOnSecurity Tuesday, Holt acknowledged the city was being extorted by DoppelPaymer, a ransomware gang with a reputation for negotiating some of the highest extortion payments across dozens of known ransomware families.
The average ransomware payment by ransomware strain. Source: Chainalysis.
Holt said the same gang appears to have simultaneously compromised networks belonging to four other victims within an hour of Florence, including another municipality that he declined to name. Holt said the extortionists initially demanded 39 bitcoin (~USD $378,000), but that an outside security firm hired by the city had negotiated the price down to 30 bitcoin (~USD $291,000).
Like many other cybercrime gangs operating these days, DoppelPaymer will steal reams of data from victims prior to launching the ransomware, and then threaten to publish or sell the data unless a ransom demand is paid.
Holt told KrebsOnSecurity the city can’t afford to see its citizens’ personal and financial data jeopardized by not paying.
“Do they have our stuff? We don’t know, but that’s the roll of the dice,” Holt said.
Steve Price, the Florence IT manager whose Microsoft Windows credentials were stolen on May 6 by a DHL-themed phishing attack and used to further compromise the city’s network, explained that following my notification on May 26 the city immediately took a number of preventative measures to stave off a potential ransomware incident. Price said that when the ransomware hit, they were in the middle of trying to get city leaders to approve funds for a more thorough investigation and remediation.
“We were trying to get another [cybersecurity] response company involved, and that’s what we were trying to get through the city council on Friday when we got hit,” Price said. “We feel like we can build our network back, but we can’t undo things if peoples’ personal information is released.”
A DoppelPaymer ransom note. Image: Crowdstrike.
Fabian Wosar, chief technology officer at Emsisoft, said organizations need to understand that the only step which guarantees a malware infestation won’t turn into a full-on ransomware attack is completely rebuilding the compromised network — including email systems.
“There is a misguided belief that if you were compromised you can get away with anything but a complete rebuild of the affected networks and infrastructure,” Wosar said, noting that it’s not uncommon for threat actors to maintain control even as a ransomware victim organization is restoring their systems from backups.
“They often even demonstrate that they still ‘own’ the network by publishing screenshots of messages talking about the incident,” Wosar said.
Hold Security founder Alex Holden said Florence’s situation is all too common, and that very often ransomware purveyors are inside a victim’s network for weeks or months before launching their malware.
“We often get glimpses of the bad guys beginning their assaults against computer networks and we do our best to let the victims know about the attack,” Holden said. “Since we can’t see every aspect of the attack we advise victims to conduct a full investigation of the events, based on the evidence collected. But when we deal with sensitive situations like ransomware, timing and precision are critical. If the victim will listen and seek out expert opinions, they have a great chance of successfully stopping the breach before it turns into ransom.”
Aw, this was an incredibly nice post. Finding the timme and actual effort to create a
superb article… buut what can I say… I put things offf a
whole lot and never manage too get nearlyy anything done.
Cyberhero Krebs I’ll say it once again:
You are indeed the man…
The city I work for has monthly computer training about events like this. You know training on how to hover over links before you click, etc.
Then they blast everyone with HUNDREDS of emails a week including “what’s happening” emails in the city that don’t affect 90% 0f the employees.
It is only a matter of time before the fatigue of clicking on all those emails will put my city in the same spot. How can you be discerning and inspect every email when you have 2, 3, 4 dozen emails to get through.
1. Block links in email.
2. Block attachments to email.
HTML and MIME are security nightmares.
Well, good luck doing business with NO links or attachments.
Any reasonably competent IT person should be able to find out if the link/attachment is legit and safe.
It’s not about competence, it is about cost.
Nobody is going to have human beings look at every email. There are many Email Security solutions out there. All the good ones cost money.
Probably less than $300,000…
And local/state governments have a very hard time justifying even that cost.
Even many private companies don’t want to spend the money on a security solution when they don’t understand how it directly makes or saves them money. Cybersecurity spending a big discussion, and you need a good Risk Officer to work out the numbers regarding loss and probability.
Many companies, and governments, simply don’t bother, and just purchase insurance. It’s good to have insurance, but many places use that as cost/benefit reasoning to NOT buy security solutions.
Sad thing is, email is horribly insecure and is also probably the most relied on system in most companies. Companies spend all their time trying to make it more safe but it is like trying to make tight-rope walking safe.
Hovering over links is good practice for emails, as the email client is handling the links.
However, be careful using this trick on websites that are less than trustworthy. Websites run javascript, and javascript can be used to dynamically change the URL link as you click on it… so hovering might tell you one think, but clicking on it takes you somewhere else.
You’d have to inspect the DOM to see it… look for functions like “onmousedown”, “onclick”, etc.
Google does this with their search results too. If they let users click on links and go directly to those websites, they can’t collect analytics on what you clicked on. So they fake/spoof their links when you hover over them to look like you’ll go straight there. But they run a javascript function “onmousedown” to dynamically change the link to a google controlled URL that would track and then redirect your browser.
Try it for yourself. Do a Google search, hover over the link for one of the results… then right-click that link. Hover over it again, and it’ll be a Google URL.
Google just uses it for tracking, but any website can do this for any purpose.
If a ransomware target does nightly backups and keeps seven days of backups, would that work for restoring after a DoppelPaymer attack?
No. As this article demonstrates the network was compromised for months before the ransomeware took effect. probably copied lots O’ data.
Yes, for restoration, this will work since IT would just need to know about the ransom within a week of when it starts to encrypt.
Note that the ransom isn’t to restore a system. That should be childs play with proper backups, and is only a convenience item to get back. The ransom is to avoid the release of personal information that the public would get their nickers in a twist over. The same public that will pony up the tax payments to make up for the ransom.
Do not confuse “ransomWARE” with ransom demands for data breaches.
Ransomware is malicious software that encrypts the data to deny its access from its rightful owners. That is what the ransom is for.
Most large ransomware campaigns use this denial of data as their leverage for ransom. They may tack on a premium for data that is sensitive or contains PII, as something else to demand. But it hasn’t been the primary ransom demand.
Now, with GDPR, CCPA and other privacy regulations, the penalty for breach is much higher. So some attackers may try to extort payment from companies thinking they may pay up because they don’t want to be fined.
But this doesn’t really work in practice, as it is criminal to not disclose the breach just because the hackers were paid and promised not to release the data to the public. The companies have to pay the data breach fines anyway.
Further, there are no guarantees that hackers won’t keep a copy and re-extort later. Companies know this, and rarely give into ransom demands for keeping private data out of the public domain.
In contrast, ransomware hackers DO make guarantees and build up reputations for decrypting data. And companies (and governments) take notice, and do wind up paying these ransoms.
They probably have some Boss Hogg types in there that figure if your call couldn’t help them get money or re-elected, there was no need to pay it any mind.
Bingo. I live in Florence and you pegged it.
The state government agency I worked for a year ago had an IT Vice President who would send around emails explaining security basics. Don’t click links was a common suggestion. The emails from IT VP would also have further reading and an ingenious security device, a link that says: “click this link, it is safe.”
Well. Should you ever decide to relocate, Mr. Krebs, the Land of 10000 Lakes would welcome you with open arms.
Our Senate website was hit a few weeks ago. Parts of it are still down. Not much information has been offered to the public, but I suspect the breach took place via a third party , (i.e. Zoom, FB, or others), and that my tech savvy congressmen (Kiffmeyer, Lucero, Nash, et al.) are having trouble auditing the trail.
Thank you, as always, for doing what you do.
So they are gonna pay and trust that their extortioners won’t sell/release the data anyway? Or demand more $ after payment?
IMHO, the mayor plus ought to be replaced. Incompetence reeks with peepz who have no insight and think IT Security is a drain must be held accountable..
We’ve seen a huge increase in phishing attempts the last 2-3 months. Anything with an attachment from outside a few trusted domains with correct SPF get dropped into spam. We train everyone to be fearful of all spam.
if the FROM and REPLY don’t match – spam.
We show our people how to setup their email clients to show only ASCii which is mandatory by the SMTP standards. Empty ASCii parts are spam.
Unexpected attachments are quarantined by the system.
Email is one of the systems i love to hate, but we’d never outsource it because we like our clients to have privacy, not feed some huge privacy-sucking marketing company with even more company+personal data.
Is this why I have no data on my phone anymore. My Verizon data had been gone for a week now and I’ve talked to 8 different people at at Verizon trying to get my data restored but it’s just gone. I have to use Wifi now! I’m suppose to have unlimited data……was their a cyber hack as well?
wow, omg..