Notifying people and companies about data breaches often can be a frustrating and thankless job. Despite my best efforts, sometimes a breach victim I’m alerting will come away convinced that I am not an investigative journalist but instead a scammer. This happened most recently this week, when I told a California credit union that its online banking site was compromised and apparently had been for nearly two months.
On Feb. 23, I contacted Coast Central Credit Union, a financial institution based in Eureka, Calif. that serves more than 60,000 customers. I explained who I was, how they’d likely been hacked, how they could verify the hack, and how they could fix the problem. Two days later when I noticed the site was still hacked, I contacted the credit union again, only to find they still didn’t believe me.
News of the compromise came to me via Alex Holden, a fellow lurker in the cybercrime underground and founder of Hold Security [full disclosure: While Holden’s site lists me as an advisor to his company, I receive zero compensation for that role]. Holden told me that crooks had hacked the credit union’s site and retrofitted it with a “Web shell,” a simple backdoor program that allows an attacker to remotely control the Web site and server using nothing more than a Web browser.
A screen shot of the credit union’s hacked Web site via the Web shell.
The credit union’s switchboard transferred me to a person in Coast Central’s tech department who gave his name only as “Vincent.” I told Vincent that the credit union’s site was very likely compromised, how he could verify it, etc. I also gave him my contact information, and urged him to escalate the issue. After all, I said, the intruders could use the Web shell program to upload malicious software that steals customer passwords directly from the credit union’s Web site. Vincent didn’t seem terribly alarmed about the news, and assured me that someone would be contacting me for more information.
This afternoon I happened to reload the login page for the Web shell on the credit union’s site and noticed it was still available. A call to the main number revealed that Vincent wasn’t in, but that Patrick in IT would take my call. For better or worse, Patrick was deeply skeptical that I was not impersonating the author of this site.
I commended him on his wariness and suggested several different ways he could independently verify my identity. When asked for a contact at the credit union that could speak to the media, Patrick said that person was him but declined to tell me his last name. He also refused to type in a Web address on his own employer’s Web site to verify the Web shell login page.
“I hope you do write about this,” Patrick said doubtfully, after I told him that I’d probably put something up on the site today about the hack. “That would be funny.”
The login page for the Web shell that was removed today from Coast Central Credit Union’s Web site.
Exasperated, I told Patrick good luck and hung up. Thankfully, I did later hear from Ed Christians, vice president of information systems at Coast Central. Christians apologized for the runaround and said everyone in his department were regular readers of KrebsOnSecurity. “I was hoping I’d never get a call from you, but I guess I can cross that one off my list,” Christians said. “We’re going to get this thing taken down immediately.”
The credit union has since disabled the Web shell and is continuing to investigate the extent and source of the breach. There is some evidence to suggest the site may have been hacked via an outdated version of Akeeba Backup — a Joomla component that allows users to create and manage complete backups of a Joomla-based website. Screen shots of the files listed by the Web shell planted on Coast Central Credit Union indeed indicate the presence of Akeeba Backup on the financial institution’s Web server.
A Web search on one backdoor component that the intruders appear to have dropped on the credit union’s site on Dec. 29, 2015 — a file called “sfx.php” — turns up this blog post in which Swiss systems engineer Claudio Marcel Kuenzler described his investigation of a site that was hacked through the Akeeba Backup function.
“The file was uploaded with a simple GET request by using a vulnerability in the com_joomlaupdate (which is part of Akeeba Backup) component,” Kuenzler wrote, noting that there is a patch available for the vulnerability.
These Web shell components are extremely common, have been around for years, and are used by online miscreants for a variety of tasks — from selling ad traffic and spreading malware to promoting malicious and spammy Web sites.
It’s not clear yet whether the hackers who hit the credit union’s site did anything other than install the backdoor, but Kuenzler wrote that in his case the intruders indeed used their access to relay spam. The attackers could just have easily booby-trapped the credit union’s site to foist malicious software disguised as a security update when customers tried to log in at the site.
Holden said he’s discovered more than 13,000 sites that are currently infected with Web shells just like the one that hit Coast Central Credit Union, and that the vast majority of them are Joomla and WordPress blogs that get compromised through outdated and insecure third-party plugins for these popular content management systems. Worse yet, all of the 13,000+ backdoored sites are being remotely controlled with the same username and password.
“It’s a bot,” he said of the self-replicating malware used to deploy the Web shell that infested the credit union’s site. “It goes and exploits vulnerable sites and installs a backdoor with the same credentials.”
Holden said his company has been reaching out to the affected site owners, but that it hasn’t had much luck getting responses. In any case, Holden said he doesn’t relish the idea of dealing with pushback and suspicion from tons of victims.
“To be fair, most vulnerable sites belong to individuals or small companies that do not have contacts, and a good portion of them are outside of US,” Holden said. “We try to find owners for some but very few reply.”
If you run a Web site, please make sure to keep your content management system up to date with the latest patches, and don’t put off patching or disabling outdated third-party plugins. And if anyone wants to verify who I am going forward, please feel free contact me through this site, via encrypted email, or through Wickr (I’m “krebswickr”).
no good deed goes unpunished…I know thousands of readers are thankful you are taking the risks you do to try to keep us safe…soldier on!
Well, paranoia seems to thrive in these uncertain times where cybercriminals unleash one threat after the other. I talked about it just yesterday with one of my colleagues : We train our users to not take anything for granted, and then we have to face the consequences of our own acts, just as what happened to Brian. I suppose there is no easy answer to this one, and I would be very happy to learn about thoughts from other members of the infosec community. As for me, I try to talk to my users in person so they know there’s an actual face behind the e-mails and the newsletters and the measures the company has to take in order to protect its assets, and I encourage them to come to my office and talk in person each time there’s a problem.
Anyhow, thanks to you Brian for your invaluable work !
Ideally, it shouldn’t even matter whether the caller is Krebs or Crab or whoever. You call, you tell them there’s a breach and how to verify it, they fix it and (hopefully) thank you (instead of trying to shoot the messenger).
Yes, I know … we don’t live in an ideal world – if we did, the problem wouldn’t exist in the first place.
most of these so-called IT desk ppls that answer phones are too busy at work on their own personal business to care, hey they just fill a void as they see it and get a pay-check to boot. I tell ya, they don’t teach much about ethics in any schools these days, not even in technology. these are very ideal conditions for the types of social engineering that the so-called IT professionals, if that’s what they want to be referred to as, get bamboozled with and the fun part is to watch them take their whole company down with them. but hey, they seem to know it all and if they don’t they’ll tell you anyway.
Then again, maybe they need to be out of business!
“Of course, that’s just my opinion, I could be wrong.”
Couldn’t agree more with this. As a webdev, I take all user reported problems seriously even if they turn out to be PEBCAK. They are usually PEBKAC but that’s ok, sometimes we get real issues, mostly edge cases.
The problem with that solution is the requisite “how would a dick abuse this” question.
Sure – the first time they get that phone call, they do all the labor and due diligence to track down the problem and correct it.
What about the 17th time? How responsive would they be on the 18th call if the first 17 were hoaxes phoned in by some bored kid.
Thanks! It will put others on notice to check their sites.
I am going to print t-shirts that simply say “Be like Patrick” and a link to this article and give it to clients I consult for. Good for you Patrick, that guy deserves a raise!
You want to teach your clients not to think for themselves as to how they might independently verify a claim that 60,000 people’s bank accounts are at risk?
Patrick has more in common with anti-vacciners than with security minded employees.
Heh – ya, I got through a bachelors and masters in physics and not a word about ethics, consequences for the future, or the travesties of past physicists. Our school system is failing big time.
+1. I work for a public university in California that seems much better at teaching anti-ethics than ethics.
Periodically, I come into possession of troves of stolen credentials (last one was ~150,000 lines) through my role at work and have the same experience as Brian when it comes to trying to notify victims.
Email is hopeless – I might hear back from maybe 1 in 50 after I send a clear & concise message advising of the issue. Presumably the rest just think it’s a scam.
Calling people gets mixed results. Some think you’re a scammer and give you an earful of abuse, some just can’t seem to grasp the severity of the issue. Only a very small monitory actually understand what you’re telling them and do something about it.
I get enough satisfaction from that last group to keep persisting…but it really is a thankless task!
I report at least 30 of these every month. The lack of response, and the slowness of response, is just baffling.
Also a lot of people who *do* update their wordpress installs fail to update old themes, which is the source of a lot of these upload exploits as well. If you have wordpress installed and you don’t use any of the older pre-installed themes: remove them!
I have a feeling that the number of servers these criminals have access to via this kind of exploit must number in the hundreds of thousands.
SiL / IKS / concerned citizen
Working for a health authority IT (unix admin), I am frustrated that many IT professionals still do not take infrastructure security seriously in this day and age. Many times I forward security articles related to my team and most of them don’t appear to be concerned about the increasing threats to our networks and systems that contain private patient data. I strongly feel anyone in the IT industry has an obligation to make efforts to become aware of potential cyberthreats that could compromise the systems they manage.
I hear, “that’s above my pay grade” a lot, even from brilliant PhD scientists. What they’re saying, I think, is that their own career (read: obtaining money) is much more important to them than their community. But this is the heart of capitalism, right?
Security needs to be on the management team’s list of business priorities to have any chance of a security culture being driven through the organization. Without that employees may not be that bothered about the consequences of a security breach.
Until the pain is felt by the people who are in a position to stop it from happening, instead of others that had no control, it will continue. With the entitlement mentality in the US toward the “right” to all kinds of other people’s information, regardless of relationship or no, I don’t see that change coming any time soon.
Being in an IT Role is a ‘thank-less’ position. We all try to focus on security and always try to have a level of forward thinking. (Pro-Active) Which can be huge challenging. Having discussions with a VP’s or Owners on ‘why’ we need to spend the money to fix something.
Unfortunately it’s like the plumbing inside the walls to most Execs. “If I don’t see water or sewage spraying or standing water, it’s not broken…why are we spending $ to ‘fix’ it?” And if I don’t spend the money and it breaks and you have to fix it. I’ve already paid for your time.” Sound Familiar?
It seems it takes “an event” to make your point and when they feel the pain of said ‘event’ then they start listening with a different set of ears. (I guess it’s also part of earning your credibility).
So for all of the IT Folks out there. THANK YOU!! Stay the course and the continued pride in what you do and delivery daily. If it wasn’t for our skills these companies wouldn’t be able to do what they do on a daily basis. Do what’s right and do no harm.
Remember: If you make it “idiot-proof”…they will always build a better idiot.
The answer to the dilemma of clueless management is the IT Security Audit. The framework isn’t as important as just getting it done. If management wont pay for it, perform your own, internal, informal audit. Present the finding/weaknesses along with remediation plans & costs. On any particular finding, if the CFO or CEO wants to sign-off on the risk then your covered. Once I started doing this the CFO opened up the company coffers because he did not want to sign off on the risk. That’s why its called a risk management system. Risk can be a beautiful thing if its wielded properly.
Hi,
Thank you for your article. As one of your goals is to help people and organisation protect their IT assets against cyberattack, I will really appreciate if you could add one more information to this article about how to detect “Web shell” into a website that uses WordPress or Joomla as CMS.
Thanks.
Sorry, but the link to ‘Hold Security’ is coded to:
http://Coast Central Credit Union
which is both bad HTML code and the wrong target.
You are doing God’s work
Though I read Brian Krebs regularly, I would be skeptical of someone who claims he is him contacting me out of the blue, and then just blindly following any instructions to “confirm” that something was breached. Just thinking back to the fake microsoft help desk calls that may people got, and they gave instructions that was to help confirm they were legit, at least to people who really didn’t know what they were looking at. But, at the same time, I would get contact information, and then when I was done with the call, I’d try to find out other information independently (google can be a good place to start) and to start performing some basic investigation to confirm something may not be right.
How do deal with such calls is likely an area of training that many organizations need to have, including who to contact internally, types of actions to take, and what information to get during the call, among others.
I can see that this is exactly how it played out. And I don’t have a really good general answer for the problem.
I suppose Brian could put a public key of some kind up on this site. But as others have noted, it is tough to get anyone to read the email to begin with, and it doesn’t help when you are on the phone trying to convince someone to take you seriously.
The only really effective thing I can see is posting a story – just like this. The only problem is that there are far too many compromised sites out there.
use keybase.io and get a public key.
verify they key then it can be independently confirmend by each person who wants to communicate with him
Regardless of whether or not you believe the caller, you should take every notification of potential threat seriously. They could have easily checked the site in sandbox setup that wouldn’t endanger the main system.
You really think spammers start calling people claiming to be Krebs? That’ll be the day. And then Krebs tells them how to verify him, and he gives links to your OWN website. He’s not asking them to type in random urls. These guys acted like they got calls from fake Krebs all the time. Yes sure.
I’d probably have done the same thing. A ton of malware, sites, and people try to imitate Brian and taking anything someone says as proof they are someone else is always a gamble. If people don’t believe you, a lot of it is due to your growing reputation and celebrity status – I wouldn’t believe it if anybody famous called out of the blue.
Ask to setup a video call.
The threats from vulnerable third-party components are increasingly common and Internet wide. The practice of using third-party components on public facing assets like websites is ubiquitous across all industry verticals, including FIs. The dangers extend beyond the firewall and the range of end point sensors. Executives really need to be asking their tech teams what measures are in place to guard against this.
Have you considered a Google Voice number on your site that you only use for work related calls? The call screening feature works great.
Very good read, as I expect nothing less coming from your site, Brian. I have a question for you –
You always hear stories about IT security admins or freelance cyber crime fighters finding an exploit or vulnerability in a company’s website. Then the next thing you know, the company is pressing charges on them, by simply being virtuous and hoping to make a difference.
Do you realize going into these kinds of situations that the odds may be against you? They may see it as, “How do you know all of this? Are you the one that hacked into my site?” Can you simply be implicated for trying to help? Should you have a good lawyer ready before you approach a company about the vulnerabilities that you know exist in their website?
These situations seem like risks worth taking, in the sense that you become a little more iconic in the security world for saving the day. Also, that company may compensate you – or they may try to throw you in jail. I think a lot of these companies could care less. They might even say, “thanks,” and that is the end of it.
that’s like me telling you that your shoes are untied. and if you don’t tie them you’re going to fall flat on your face. I tell you this because i see they’re untied, all the while you’re walking and texting, or tweeting, or whatever it is you do as you take a long stroll on a short jetty during high tide and rip currents. i know you can see because you’re walking and texting, or tweeting, or whatever, oblivious to the threats around you. and you’re going to sue me for that? if so, then you need to hit something real hard, perhaps that’ll knock some sense into your head.
i mean it’s not like i untied your shoes, now is it?
oh my, pride certainly does come before the fall.
“Of course, that’s just my opinion, I could be wrong.”
I completely agree with your analogy,
but if i slip on a soda I spill in Walmart I can still sue Walmart, for selling me the soda and slipping on their floor, so ha!
People will find any loop hole to screw you over if they really want to. Especially companies who have to really comply with HIPAA and other confidentiality policies. You try and tell them their shoes are untied and they’ll blame you for untieing them.
Well, isn’t that a shame. Hold Security’s own website is not protected: http://holdsecurity.com/wp-login.php
I can’t imagine a security company exposing the admin panel of their website like that.
What was it about … the shoemaker’s kids go barefoot?
Good catch!
Pulled from my own TCP trace:
log=test&pwd=test&wp-submit=Log+In&redirect_to=http%3A%2F%2Fholdsecurity.com%2Fwp-admin%2F&testcookie=1HTTP/1.1 200 OK
Common, it doesn’t take a genius to look at any sites robots.txt file…….
Common, it doesn’t take a genius to look at any sites robots.txt file….
A lot of WP-Admin web portals are indeed blocked by web crawlers.
That’s pretty obvious….
So? Security by obscurity is good then?
I feel for you Ed. On the one hand, a relatively small CU has an IT staff and a VP in charge. Great. But sadly, the response to this incident is all to indicative of the state of small and medium sized companies. @eric, sure, an article will get attention, but yeah, that won’t scale. What is amazing to me is how many colleagues are hustling for business while at the same time, many, many more companies have got such a poor handle on security. How do we more properly align the supply and demand signals?
We connect the pain of causing a harm to the people with the responsibility not to allow it to happen. And we make the pain proportionate to the entity allowing the harm, NOT in relation to the one harmed, so that they care.
If you drink and drive in Europe, the fine will be a percentage of your income. Consequently, they just don’t do it.
If companies had meaningful consequences for harm they caused, we wouldn’t be having this conversation. And you can bet we wouldn’t see nearly as much data collection as we do, if it all had to be safeguarded.
The issue is accountability.
I am a previous customer of this credit union and an IT Professional of 15 years (6 in the financial sector). When I first moved to the Eureka area, I dropped off my resume to the IT department at Coast Central Credit Union. I was told..
“We only promote MSRs to the IT Department, we generally don’t hire outside IT people.”
!!!!!
I promptly closed my account, KNOWING they would be hacked eventually. I am NOT surprised.
haha, smart man josh. and they don’t hire from outside because they’re union.
now they’re exposed for their stupidity:
Coast Central Credit Union Website Hacked; Management ‘Confident’ Online Banking Was Not Compromised.
haha – Enjoy the Lime Light CCCU, you’ve earned it.
Management ‘Confident’ = network security oxymoron
Member Service Representative?
Member service representative = customer service representative.
Credit union’s are non-profit cooperatives owned by their members (versus being privately held or owned by stock holders). When you open an account at a credit union you are becoming a member of the cooperative. That’s where the difference in terminology comes from.
The union part of the name doesn’t refer to labor unions. Though labor unions have credit unions (and presumably members of the credit union are also members of the labor union) they don’t inherently mean the same thing.
I am not surprised by the response. I occasionally try to inform companies of a bug I discovered on a site and am routinely ignored. They don’t deem credible bugs discovered “in the wild.” Or they want more information, like I am supposed to debug their system for them. And this is for companies I have a documented relationship with, not some anonymous phisher.
Um… What are the folders “Kids” and “Teens” for at a bank???
We try to get the youngsters saving early. Teaches them about the moneys and so forth.
They’re redirects.
But they redirect to insecure servers which is really not a good example for kids.
@Brian, if you still have their ear, please encourage them to get their partners to pay for TLS certificates…
https://coastccu.org/kids
https://coastccu.org/teens
In the right environment, links to secure sites which redirect to insecure sites means a MITM attacker can trick you into going to a page it controls while having you think it’s trusted by the party identified (e.g. coast ccu)
Hello, I am the author of Akeeba Backup and I want to point out some obvious factual inaccuracies in your post.
com_joomlaupdate (i.e. the Joomla! Update component) is NOT part of Akeeba Backup, it’s part of Joomla! itself since version 2.5.4, with the current Joomla! version being 3.4.8. Joomla! Update does use Akeeba Restore, i.e. the self-contained restore.php script used to extract the update ZIP package on the site during the update process.
About two years ago (August 2014) a security researcher notified us of a cryptographic attack against restore.php which could only take place if a site administrator started an update and deliberately halted it while the extraction was taking place, leaving behind a file necessary for the attack to take place. I personally spent the next 40 out of 48 hours awake, coming up with a solution. A Joomla! version containing the fix was released immediately. The POC wasn’t released for another 2 1/2 months and there were no indications that anyone was successful at reproducing the security issue before the release of the POC.
Therefore the attack was NOT made possible by Akeeba Backup but by a severely outdated version of Joomla!. The Joomla! project publishes security advisories on its site and files for CVEs for every single security issue. Moreover update notifications for both core Joomla! and third party extensions are VERY VISIBLE in the Joomla! back-end (there’s a huge red warning with an update counter and update instructions in this case). The site administrator in question is at fault for deliberately ignoring the update messages for more than 18 months.
I would appreciate it if you put things in perspective instead of painting CMS and third party developers as the problem. We do release security updates and shout from the rooftop that there is an update. All the site administrators need to do is press the freaking update button. My first security advice every single time I’ve been giving my Joomla! site security presentation the last seven years all over the world is “update, yesterday”. I’m so sad that people are STILL not updating and exposing their sensitive client data to hackers. I guess you just can’t fix stupid 🙁
Trying in vain to notify the responsible website owners of breaches reminds me of Clifford Stoll’s plight in his book “The Cuckoo’s Egg”. So many people don’t care! That book was the one that piqued my interest in Information Security.
+1
Akeeba Backup – why would a bank with an in-house IT department use a plugin to do backups? That’s just stupid!
Trying to report any problem with any website is a losing battle and is pointless. Public shaming is the only way to go.
I once couldn’t change my email on a site. The Ajax request was reporting a 500 error from the server. I reported the problem, gave all info to reproduce the problem, included screenshots, and said forward to your web developers. I got a response back saying “clear your cookies and cache.” I responded saying my cookies and cache we’re fine (they are cleared several times a day, I browse under ‘tinfoil hat settings’) and I’ve verified with 3 separate browsers, I am a professional web developer (true), and the problem is on the server. Just forward my email to the web devs and they will know what to do. I got a very nasty email in response which included the lines “there is nothing wrong with the website.” Really nasty stuff was included telling me how wrong I was and insulting me personally. Never was able to change my email address….
Another great post Kreb. Sometimes even when you’re trying to do a good deed it’s not well received. Unfortunately it’s just the way the world is moving with fewer people doing good deeds and more people trying to take advantage of online vulnerabilities. Good job blogs like yours exist!
A large regional convenience store chain here in SW MO had something on their pumps that captured my Debit Card info (which I hadn’t used in weeks) and within 2 hours made a purchase in Ca to ship to Colorado. I called the warehouse of the shipper and stopped the order and ordered a new Debit Card.I’m proactive. When I called their corporate HQ in Iowa (twice in a week) to make sure they were away got “denials of the obvious” (as you did) and ended each call with “I will be reading about you on Krebs in 8 months”. That was 6 months ago. They have 2 more months for their incompetence to come to light. Only until heads start rolling, “Would you like fries with that?” will this purposeful incompetence stop.
Your card details could have been obtained much earlier in an unrelated breach and warehoused for a while. Tracking card compromises involves triangulating more than a single fraud.
Sadly I get the same thing all the time. Maybe next time I should tell them my name is Brian. Usually it’s several emails, phone calls, multiple contacts to IT, Administrative staff, and if possible someone with authority to do something. Patience really pays off. Sometimes you even get a thank-you. Yup, next time my name will be Brian. Actually got a thanks the other day, walked past a bank ATM that had its doors open right in the mall walkway. I walked in the bank and told them, they were surprised and happy. We all do what we can.
Patrick and Vincent should be immediately fired.
Many credit unions outsource their I.T. to third parties, such as Fiserv. The support folks at the credit union itself, may not have the ability to test or even see the flaws pointed out to them. They have to make judgment calls about what trouble tickets to submit to their contractor.