Earlier this month, I published a story about a criminal hacking gang using Adobe ColdFusion vulnerabilities to build a botnet of hacked e-commerce sites that were milked for customer credit card data. Today’s post examines the impact that this botnet has had on several businesses, as well as the important and costly lessons these companies learned from the intrusions.
Last Tuesday’s story looked at two victims; the jam and jelly maker Smucker’s, and SecurePay, a credit card processor based in Georgia. Most of the companies contacted for this story did not respond to requests for comment. The few business listed that did respond had remarkably similar stories to tell about the ordeal of trying to keep their businesses up and running in the face of such intrusions. Each of them learned important lessons that any small online business would be wise to heed going forward.
The two companies that agreed to talk with me were both lighting firms, and both first learned of their site compromises after the credit card firm Discover alerted their card processors to a pattern of fraudulent activity on cards that were recently used at the stores.
Elightbulbs.com, a Maple Grove, Minn. based company that sells lighting products, was among those listed in the ColdFusion botnet panel. Elightbulbs.com Vice President Paul McLellan said he first learned of the breach on Nov. 7, 2013 from his company’s processor — Heartland Payment Systems.
McLellan said the unpatched ColdFusion vulnerabilities on the company’s site was certainly a glaring oversight. But he said he’s frustrated that his company was paying a third-party security compliance firm upwards of $6,000 a year to test Elightbulbs.com for vulnerabilities and that the firm also missed the ColdFusion flaws.
“Shortly before we were told by Heartland, we paid $6,000 a year for a company to brutalize our server, for protection and peace of mind,” McLellan said. “Turns out this flaw had existed for two years and they never saw it.
McLellan said the company received a visit from the FBI last year, and the agent said the group responsible for hitting Elightbulbs had compromised much more high-profile targets.
“The FBI investigator said, ‘Hey, don’t beat yourself up. We’ve got credit card processors and government institutions that run ColdFusion who were breached, this is small potatoes’,” McLellan said. “That was a small consolation.”
Ultimately, elightbulbs.com opted to remove the target from its back by outsourcing the processing of credit cards on its site to authorize.net, a third-party processing firm that specializes in securing e-commerce transactions.
“Myself and my IT director made a pact that we’re never going to back to charging cards on our server, that we were going to take the site out of the equation,” McLellan said. “At first I thought it would turn away customers, but people don’t seem to mind the extra step. And for me, I get to sleep at night knowing I’m protecting my customers data. Personally, I’ll never go back to taking [credit cards] on the site. It’s hard enough running a small business, and I don’t want credit card theft being one of the things I have to constantly worry about.”
Kichlerlightinglights.com was another lighting store ensnared by the ColdFusion botnet. Company owner Gary Fitterman said the breach cost his company a tremendous amount of money and time.
“It was like being attacked by terrorists,” Fitterman said. “When we learned what had happened, we immediately went into a frenzy, spent a ton of money to get [forensics experts] in to take a look.”
In the end, Fitterman and his team also opted to outsource the credit card processing to a third party, deciding it wasn’t worth the risk of continuing to handle it in-house.
“Now we can just concentrate on making our business grow, rather than always playing catch-up to make sure we have latest and greatest,” Fitterman said. “It’s not worth the risk. I don’t think there’s that much information out there to make small businesses like me aware of everything you should be aware of before this happens to you.”
Also among the four dozen or so sites enslaved in the ColdFusion botnet was the Web storefront for LaCie, a hardware company that specializes in external hard drives.
Clive Over, director of corporate communications for LaCie owner Seagate, said the company has investigated the incident and has so far found no indication that any customer data was compromised in the attack.
“This week, the Company received information indicating a server hosting LaCie.com may have been maliciously targeted and possibly breached at some point during calendar 2013,” Over said in an emailed statement. “Privacy and security is of utmost importance to the Company, and we therefore took immediate action to investigate this matter as soon as we became aware of it. The Company has conducted a preliminary investigation and, at this time, we are not aware that company or third party information was improperly accessed. The Company is currently working closely with third party experts to do a deeper forensic analysis.”
Adobe ColdFusion vulnerabilities have given rise to a number of high profile attacks in the past. In February, a hacker in the U.K. was charged with accessing computers at the Federal Reserve Bank of New York in October 2012 and stealing names, phone numbers and email addresses using ColdFusion flaws. According to this Business Week story, Lauri Love was arrested in connection with a sealed case which claims that between October 2012 and August 2013, Love hacked into computers belonging to the U.S. Department of Health and Human Services, the U.S. Sentencing Commission, Regional Computer Forensics Laboratory and the U.S. Department of Energy.
Update, 12:15 p.m. ET: The Guardian reported today about another apparent victim of ColdFusion fail: the carmaker Citroen.
When people don’t update their software what do they expect? I know the gentleman said small businesses don’t have enough information but I would argue that the IT director just plain didn’t update his software, it was that simple and a free fix. I don’t see how that can be hard?
I agree. This is less of a “ColdFusion Fail” and more of a “SysAdmin fail”. It sucks that CF had a couple big exploits that came out last year, but they were only only applicable to un-hardened servers and were patched within weeks by Adobe. ColdFusion *E-mails* you when new patches come out and visually alerts you in its admin interface. No need to pay a consulting company $6K, just check your mail and click the Update button!
Even so, I agree with outsourcing CC processing. Most companies just aren’t up to it. Stripe allows you to collect CC data right “in your site”, but it’s submitted straight to their servers over secure Ajax and never touches your servers.
Outsourcing is the right idea for them since there is no reason to take on that risk if their IT department is unable to address it. What also seems to be a common them with ALL of these breaches is the lack of a basic WAF. You can get a wonderful free WAF and rule feed yet the companies feel their data is not important enough to secure I assume.
I have to agree that many small businesses do not think that the data they gather is important enough to protect. I pointed out some serious flaws in data security to a friend who owns his own business, and his reply to me was: What do I care? What can an attacker do with the data?
He does not process the credit cards through the same equipment as he keeps his customer data, but misses the value of the data he does collect. He never even bothers to back up the data. I had to give up trying to explain the importance of security because he just did not care.
The small business tends not to value the data, and the larger ones value it so they can monetize the information. I’m not sure which is worse, having someone lose it to hackers that exploit it, or selling it to folks that exploit it in their own ways.
The way folks think about information will need to shift away from unilateral valuation of other people’s information.
I suspect if the liability of holding the data had A) big penalties (e.g. $1,000,000.00 per person effected) and B) required insurance to hold that data (e.g. 10 customer company would require a $10,000,000 insurance coverage) then the insurance companies would likely require much better handling practices.
We are not anywhere near creating a token economy for information. It could address folks treating the same data as anywhere from zero (worthless) to a pure asset with unlimited leverage available.
Some mechanism, whether the one described above, or regulations like Dodd/Frank with derivatives valuation will happen. Perhaps A) to establish a valuation in GAAP, and B) regulations based on that valuation. The present valuation scheme of whatever it’s worth to the person holding it (rather than to the person that person(s) represent) clearly needs to change.
The ‘when’ is another matter. I suspect it’s after we have Great Recession 2.0 from IT’s bright idea of Big Data. But that’s just because I’m quite sure the IT folks think they are the smartest folks in the room (smarter than those financial rocket scientist derivative folks, who were.. wait for it.. literally the smartest folks on the planet).
So frankly, I’m cynical, but history suggests it’s well founded cynicism.
The idea of folks holding data being caretakers seems out of fashion in an environment that stresses leveraging and monetizing the data.
One last, thread (I promise!). Cultural disrespect in general leads to not caring about others and is related to not caring about what information could do to others. When it hurts them personally (or when they inadvertently hurt someone they care for) they consider information with an existential view. It’s unfortunate that technology’s de-socialization isolates folks from caring about others while at the same time allowing them to leverage that apathy.
The above may help explain how a small business, thinks (“I did this myself”.. and “everyone should look out for themselves”) would view data harming others as ‘not my problem’, or ‘worthless’, while a large business sees the leveraged monetized information value and legal liability. The only difference is their valuation of the same data.
I’m not saying anything here that folks don’t already know, but how we think about handling someone else’s information is a worthwhile topic before folks can really come up with how to handle it.
Don’t forget that a lot of these small businesses being targeted don’t have any staff who is IT literate… let alone a “sys admin”. What seems like a cultural IT given (update your apps/systems) may not be so obvious to people who aren’t immersed in that culture.
I can’t really disagree with that, but it’s hardly a fail on the part of ColdFusion though. If any of these companies were using PHP, I’m sure it would be out of date and insecure as well. PHP does have more vulns per year than CF:
I can Brad, I worked for someone a long time ago, this person refused to acknowledge having someone Admin the site on a 24/7 basis. In that time they where attacked and hacked about 4 times a year.
I got sick of being called in to fix things up, now this person hates ColdFusion with a passion because they refused to pay for SysAdmin support.
Simpling comparing an arbitrary count of vulnerabilities identified tells you NOTHING about the relative security of two platforms, just as comparing the number of service bulletins tells you nothing about the relative safety of two cars.
This same error is made constantly when comparing the relative security of various products, and it’s always an error.
1. Some vulnerabilities are more serious than others. A single vulnerability that allows arbitrary code execution is far more serious than a dozen denial of service vulnerabilities.
2. Open source products are more likely to have their vulnerabilities made public precisely because the source code is available to the public.
3. How long does it take for a known vulnerability to get patched?
I’ll readily agree that relative security is a complex thing and based on many factors, but but to completely discount the amount of holes coming out in a piece of software is disingenuous. To me, the fact that PHP has had SIX times the number of holes means something. I’ll also point out that 53% of PHP vulns are code ex and overflow, while that only comprises 8% of CF vulns. Don’t know about you, but I’m glad I have CF installed on my servers.
None the less, that wasn’t the point– I was originally saying that companies that fail to patch servers are the core issue. And as a platform, CF has a pretty good security record according to most statistics and looking at the number of vulns AND the number of exploits to have been released.
Brad, that is only for ColdFusion 10 and up. ColdFusion 9 and below have a manual patch system that is rather convoluted and complex. Some of these servers that were exploited were ColdFusion 8 which is long past End of Life for updates. Many times the hosting company isn’t even applying the updates unless specifically asked and even then many hosting companies don’t have the technical expertise to apply the patches correctly. I know this from our experience in fixing servers that have been ‘patched’ by a hosting provider. The short story is that you should hire an expert to manage your servers no matter what platform you are using. ColdFusion has been taking a few punches to the face this past year, but so have all other platforms.
Don’t forget that there is a huge majority of people going on grabbing cheap VPS solutions running their sites, which in turn is not patched on the CFML level, but even the OS level.
They may think that they have nothing worth anything on the server, but these servers also get used to proxy scams and hacks as well. If you actually look up the IPs of a lot of these attacks they are proxied through other sites on a lot of hosting companies.
I send them an email every now and then, but I think these guys just don’t even care that there servers are being used in this manner, as they don’t understand how it works.
Most hosting providers have a policy against that. If you report them to the abuse center they *should* shut down compromised VMs until they are cleaned.
Problem is people don’t report them..
This attack was first discovered in late of 2012, some of my servers or customers where attacked in Jan 2013, but due to the sites running ContentBox. They only got as far as the IIS exploit allowed them, but could not run any of their codes on these sites.
ColdFusion has been unlucky to get the attention it has, but it is only small in comparison to the other more serious attacks that go on well before we even know about them.
But there is no excuse to keep security patches and servers well up to date.
To help put Brad’s comment’s in context, he’s a self-described “evangelist” for ColdFusion products, so take that for what it’s worth. He has a vested interest in making CF products seem as shiny and new as possible — even if half of its userbase is on way older versions of the software.
It’s true, I get paid to help spread the adoption of professionally-backed open source frameworks built in CFML. However, no one is paying me to contribute to this discussion. I am simply someone who’s used CF for well over a decade and have my share of security experience as well.
I feel that CF as a technology has received a bad reputation in the last year or so that it doesn’t (entirely) deserve based on the shortfalls of a portion of the install base as well as old, unfounded prejudices. That’s why I spend a lot of my free time sharing about the language and helping provide more “balanced” views on the current merits of the platform.
The first two companies were refreshing to hear what their leaders said – LaCie just sounded like PR CYA doublespeak.
It’s always good to point out that (I believe) the source code to Cold Fusion was stolen in that huge breach at Adobe this last year – this would seemingly allow the bad guys to more intelligently identify and target weak points in the product going forward.
Thanks for another well-researched
and clear article.
And congratulations to mgmt. at
‘for coming forth and sharing their experience
and future course of action.
This, in contrast to other companies,
which when compromised,
deny it or do not comment.
“…we take the security and privacy of our customers
Read that headline first as “the long fail”, which would also be appropriate.
Note that @Brad says that ” ColdFusion *E-mails* you when new patches come out and visually alerts you in its admin interface” is only true for ColdFusion server version 10, which also has a much easier way to update.
Prior versions of CF server do not notify you of security or other updates, and the update process in those versions was not easy, which may explain why some have not updated those prior versions.
And, it is a great effort to move everything to a new CF server, which is probably another reason for not updating.
CF10, however, has a nice interface for alerting and updating versions.
ColdFusion 10 came out back in 2012. CF 9 originally shipped in 2009– 5 years ago! So yeah, you’ve got to stay current and if you’re on some old, outdated version of CF I don’t know what to tell you other than to keep your stuff up to snuff. If you can’t afford the licensing, switch to Railo. It’s a professionally supported free open source CFML engine.
Even if you have your payment processing outsourced, isn’t having your web server compromised still putting car data at risk?
Yes, you can have an iframe to a 3rd party that handles the payment but if I own your server can I not still scrape RAM and sniff traffic?
No, because if done properly, the card data never touches your server or even crosses your firewall. Yes, a hacker might still be able to get E-mails, names, and addresses off your server, but they can’t get CC numbers if you don’t touch them. That’s why i use Stripe for my stuff– it’s embedded right in my page, but ajaxed over SSL straight to Stripe and then the form fields are cleared out. It never touches my network cards, RAM, etc. One less thing to worry about.
What you say is technically true, the CC info will never touch your system. However, if you care for your customers credit card info, you are still as vulnerable to attacks against them. Someone compromising your web server can point the customer anywhere he wants for payment. If done properly, he/she will only act as a MITM and payments will still go through as expected with the CC info stolen meanwhile.
@PG What you say is true, but you’re changing the subject. The original comment by “Security Guy” simply asked the the card data specifically could be hacked from your server if you were using a 3rd party processing center. I was only answering that specific question, and the answer is “No, not if you’re doing it correctly”.
That in no way absolves the server admin from still keeping the primary server secure to protect against a myriad of other vectors, of course. It just just reduces the sensitivity of the information stored there.
His question was if card data was still at risk when the webserver is compromised. The answer to that question is for me yes but using other vectors than what he mentions like memory scraping.
As a countermeasure, stripe could recognise the ips assigned to your server. If it started seeing requests from one of them, that means your server is now doing a MITM for CC handing and potentially compromised.
With a name like security guy, that is a fairly stupid question. The concept is that you take a secure connection and pass them onto the desired 3rd party, as this is a new session with them the browser is connected there and you will not see anything come into your servers until the request for payment has either failed or succeeded.
But with a name like the Security Guy we would think you should know that!!
Wow, Andrew why all the anger?
If someone has owned your server operating system, all bets are off. Either redirecting users to an evil twin payment site or simply doing a man in the middle SSL attack all become possible when you have complete control of the underlying OS that the web site runs on.
The key phrase is ‘to the desired 3rd party’.
And it all depends on who is doing the ‘desiring’ 🙂
If the server isn’t compromised, then sure, white hat draws (white hats never win.. it’s draw or lose only. 🙂
However, as SecurityGuy points out, if the server is compromised, then the ‘desired 3rd party’ can then be designated by the black hat.
The whole ‘store and forward’ thing seems like a problem, if you ask me, because 1) it stores, and 2) it forwards 🙂 It’s a good actor man in the middle.. nothing more.
anyway, carry on.
If you get a server that is owned as you put it, then you have more problems than what is being discussed here.
I would have to agree with Brad that this is mostly a failure of the sysadmins of properly configuring and hardening the software. Adobe has published lockdown guides for ColdFusion 9 and 10. When followed, they mitigate virtually all of the exploits that occurred in the past year. If organizations do not use them they run the risk of this happening to them.
There is also a service, HackMyCF, which will report security issues on a given ColdFusion server, which I would highly recommend.
Especially since the attack on CF came through a flaw in IIS….
Confused… didn’t we see breaches earlier this year from well known stores and their response was something like…. it wasn’t us, it was our third party processor… passing the buck because you can’t manage the security yourself isn’t a solution as far as I’m concerned.
Sometimes you update to the latest & safest version of java, Adobe, etc, only have you mission critical application break.
What do you do? Run older versions of the helper apps OR stop doing business for a while.
These uncoordinated interdependencies are the biggest issue with cloud computing, SaaS, Etc.
You do the same thing all of us do, you upgrade a staging server first and workout any upgrade compatibility issues. Once you have upgrade compatibility issues worked out then you upgrade the production server. Upgrade compatibility is something everyone has to deal with on every platform and in every language.
I’m the person at CF Webtools that was called in to investigate what happened on the eLightbulbs.com server. The short story is ColdFusion was not fully patched. However, what I found on the server was far more interesting. There was a rogue IIS Module installed that was sniffing every HTTP POST looking for credit card data. So even if the server had been patched later the theft of data would still continue.
Especially since the IIS flaw was patched back in Jan 2013 or there about’s and CF not long after.
I see what you did there…
Another interesting article
I would blame the programmer before I blame ColdFusion. I have had sites running for literally years and never had a site hacked. Any programming language is insecure if the person writing the code doesn’t have any idea how to secure their code.
True, but in most cases the actual flaw is from other installed services on the server. How do you blame the programmer in that case!
No, in this case it’s the “SysAdmin” for not properly installing and locking down ColdFusion.
SysAdmin for this discussion could be a developer that claims he/she “knows how to install ColdFusion”. Attacks on flaws in IIS and ColdFusion were stoppable with proper layered security and a good solid ColdFusion lockdown. I’ve investigated numerous servers that have been hit by this and many more that had the attack attempted and failed because a SysAdmin did it right.
In some cases it came down to no one knowing who was responsible for maintaining the server and thus everyone thought someone else was doing it. Which is why I asked on my own blog “Who Patches Your ColdFusion Servers?”.
Wil, I stand by my statement. The reason behind that is that my servers had attempts on them as well, they succeeded in getting into the servers via the IIS exploit, they then used ColdFusion and a known exploit in that, to put files on the server.
However unlike everyone else, these files could never be run on these servers.
Again, how does that fall onto the developer?
I said “No, in this case it’s the “SysAdmin” for not properly installing and locking down ColdFusion.”
How does that translate into “I am blaming the Developer” as the problem?
Wil, that might be the case for you. But I was asking the other person and not you.
This article is complete rubbish. There is zero information in here besides inflammatory remarks and name dropping. I had to follow the story to The Gaurdian’s article which despite being equally trash actually included a quote that described an SQL injection attack. You’re talking about the most common attack on the internet as if it were exclusive to a single application server.
What a joke.
It one case it may have been SQL injection. I know for a fact in other cases it was not SQL injection. Google “IIS coldfusion exploit” and have a good read.
Sorry you didn’t like the article, Patrick. So more technical details, yes? This story was written for small business owners who have relied on technology (in this case ColdFusion) without totally understanding what’s necessary to maintain it. IMO, that’s a lesson worth repeating 100x a month, given how many companies still need to learn it.
You seem to have an issue because it seems to single out ColdFusion. Perhaps you have an emotional or financial connection to CF? Zero information? Trashy? Inflammatory? Who’s being inflammatory, I wonder?
You want more technical information about how these attacks went down? Spiderlabs has the best synopsis yet.
What you describe applies too every known language, why single out ColdFusion here?
I agree with Andrew and Brad. This article seems like an attack on ColdFusion when it is really a case of companies not keeping up to date which happens no matter what language you are using. Because of that, it lessens the validity of the article in my eyes. Stay true to the facts and leave your personal opinion of what products you like out of it to keep all readers interested in coming back.
It comes down to being educated or lazy in performing the actions needed to prevent these things. Hopefully it is due to being uneducated and they strive constantly (with the rest of us) to stay educated to keep up with the data terrorists that plague our professions.
@Brad and others pointing finger at “sysadmin” of the site… it is easy to blame them but step back a few.
Where I worked last (label industry), all of their customers were in the Windows XP (stone)age (and would not upgrade) their desktops. So it made sense to the owner not to spend more on being ahead of of the “norm”. Imagine a small business owner that denies expenses to upgrade, costs for salaries to patch, etc… They would side with the ease of use (cheap)… no passwords for the desktop operators…it was too bothersome for them to manage. (I am serious!).
Now imagine, you have a sysadmin that cannot do his/her job: every upgrade, update, request is scrutinized and denied. No one wants to send the employee(s) for training. No management wants to funnel money to server upgrades, software, security or any “sound” policy. They want it cheap. Heck, after leaving, they gave the empty position to some other employee because “they knew filemaker”. And the hardware firewall service agreement was allowed to lapse.
They get what they pay for. So point fingers at them all: the company for having no security policy; the admins; the software companies for their extortion-like licensing (the tiers systems is antiquated and bias against the little guy); the hosting company for not being proactive; the web programmer for not being secure. Some of us have our hands tied: we tell them what is wrong…they complain it costs too much. “Pennywise, pound foolish”.
Eddie, good points all around. Attack vectors are often left open due to company-wide failures.
As the interviewee for the eLightBulbs.com reference, I thought I would weigh in here.; I think there is more that could/should be said regarding our scenario.
We had been under the strong impression that our CF8 server was properly locked down as we had our hosting provider scheduled to perform these tasks on a Managed Services contract. We were licensing a VDS machine of theirs for our server and up to that point, they had properly patched CF, Windows, IIS, etc… additionally, they were aware of, and implemented proper security procedures.
We had also been contracing a PCI compliance penetration test from another third party firm to properly identify any holes in our website/server. This was a manual process, and they implemented various known ColdFusion hacks to attempt to break our site/server. They never once revealed any holes; this, after spending $6,000 per penetration test.
After the discovery of the issue on our server by Wil Genovese (mentioned in comments), we parted ways with the penetration test firm and had some strong sit-down meetings with our hosting provider. They indicated that while they previously had patched ColdFusion for us (since 2007), this wasn’t in the SLA, so they were not contracted to do it on a regular basis. Well, now we know… :/
We are now running on CF10 and we are setup with Wil Genovese’s firm, CF WebTools, to keep our server patched and locked down on an on-going basis. Further, we use Authorize.net DPM protocol to handle all of our credit card transactions, so the data is now being passed between the customer’s browser and AuthNet directly.
This truly was a frustrating and exasperating experience, especially since we thought we had all bases covered with our Managed Services plan through our hosting provider and our annual Penetration Testing. We have learned our lesson and now are more deeply involved in the oversight of our server’s security and have hired CF experts (read: this is what they do on a daily basis) to keep everything as secure as it needs to be.
Paul, kudos for sharing publicly about the attack. It sounds like you guys did more due diligence than most companies do.
I’m surprised your pen testers didn’t know about the vulns. There are auto-pown and metasploit scripts all over the Internet. Do you mind sharing when the actual attack occurred?
it did my heart good to see what I think is one of your best pieces, The Long Tail of ColdFusion Fail. Having been through a data breach (Giang Hoang Vu was finally arrainged Monday in Atlanta) for The Adober breaches), the quotes from the victims stirred long-dormant emotions. I can say it feels like your home was robbed by breaking & entering, but you found out by reading a headline in the national press, and you don’t know if the robbers are still there for weeks afterwards, or even how they got in.’
I think this is the type of piece that can and should glean a larger audience – not to say your regular stuff should be dropped in lieu of this, or that deep articles doesn’t get coverage, nor that you should change to get better coverage – there are enough pop-tech writers out there – just that i found this eminently readable, and enjoyable as an article.
@Paul McLellan, thanks for being willing to provide more info and details regarding your incident. Hopefully someone else will read, and make them verify that their provider is obligated to patch the server.
I hope you gave the pentester an earful.
Brian – interesting response from the Lacie guy, I had my MasterCard popped a week or so after buying a USB key from Lacie.
It would appear that Lacie may well have been compromised and credit card data stolen.
There is sadly a preconceived stereotype attached to ColdFusion, which is why you see these articles with such negative titles. The reality falls down to organizations not upgrading, patching, and locking down their products appropriately.
For a comparison, see the article posted for the Microsoft zero day vulnerability that has been around apparently since Word 2003 (11 years?). There is no ‘long tail’ and ‘fail’ in that title.