October 3, 2013

Adobe Systems Inc. is expected to announce today that hackers broke into its network and stole source code for an as-yet undetermined number of software titles, including its ColdFusion Web application platform, and possibly its Acrobat family of products. The company said hackers also accessed nearly three million customer credit card records, and stole login data for an undetermined number of Adobe user accounts.

A screen shot of purloined source code stolen from Adobe, shared with the company by KrebsOnSec

A screen shot of purloined source code stolen from Adobe, shared with the company by KrebsOnSec

KrebsOnSecurity first became aware of the source code leak roughly one week ago, when this author — working in conjunction with fellow researcher Alex Holden, CISO of Hold Security LLC — discovered a massive 40 GB source code trove stashed on a server used by the same cyber criminals believed to have hacked into major data aggregators earlier this year, including LexisNexis, Dun & Bradstreet and Kroll. The hacking team’s server contained huge repositories of uncompiled and compiled code that appeared to be source code for ColdFusion and Adobe Acrobat.

Shortly after that discovery, KrebsOnSecurity shared several screen shots of the code repositories with Adobe. Today, Adobe responded with confirmation that it has been working on an investigation into a potentially broad-ranging breach into its networks since Sept. 17, 2013.

In an interview with this publication earlier today, Adobe confirmed that the company believes that hackers accessed a source code repository sometime in mid-August 2013, after breaking into a portion of Adobe’s network that handled credit card transactions for customers. Adobe believes the attackers stole credit card and other data on approximately 2.9 million customers, and that the bad guys also accessed an as-yet-undetermined number of user names and passwords that customers use to access various parts of the Adobe customer network.

ColdFusion source code repository found on hacker's server.

ColdFusion source code repository found on hacker’s server.

Adobe said the credit card numbers were encrypted and that the company does not believe decrypted credit card numbers left its network. Nevertheless, the company said that later today it will begin the process of notifying affected customers — which include many Revel and Creative Cloud account users —  via email that they need to reset their passwords.

In an interview prior to sending out a news alert on the company’s findings, Adobe’s Chief Security Officer Brad Arkin said the information shared by this publication “helped steer our investigation in a new direction.” Arkin said the company has undertaken a rigorous review of the ColdFusion code shipped since the code archive was compromised, and that it is confident that the source code for ColdFusion code that shipped following the incident “maintained its integrity.”

“We are in the early days of what we expect will be an extremely long and thorough response to this incident,” Arkin said. The company is expected to publish an official statement this afternoon outlining the broad points of its investigation so far.

Arkin said Adobe is still in the process of determining what source code for other products may have been accessed by the attackers, and conceded that Adobe Acrobat may have been among the products the bad guys touched. Indeed, one of the screen shots this publication shared with Adobe indicates that the attackers also had access to Acrobat code, including what appears to be code for as-yet unreleased Acrobat components (see screen grab above).

“We’re still at the brainstorming phase to come up with ways to provide higher level of assurance for the integrity of our products, and that’s going to be a key part of our response,” Arkin said. He noted that the company was in the process of looking for anomalous check-in activity on its code repositories and for other things that might seem out of place.

“We are looking at malware analysis and exploring the different digital assets we have. Right now the investigation is really into the trail of breadcrumbs of where the bad guys touched.”

The revelations come just two days after KrebsOnSecurity published a story indicating that the same attackers apparently responsible for this breach were also involved in the intrusions into the  networks of the National White Collar Crime Center (NW3C), a congressionally-funded non-profit organization that provides training, investigative support and research to agencies and entities involved in the prevention, investigation and prosecution of cybercrime. As noted in that story, the attackers appear to have initiated the intrusion into the NW3C using a set of attack tools that leveraged security vulnerabilities in Adobe’s ColdFusion Web application server.

While Adobe many months ago issued security updates to plug all of the ColdFusion vulnerabilities used by the attackers, many networks apparently run outdated versions of the software, leaving them vulnerable to compromise. This indeed may have also been the vector that attackers used to infiltrate Adobe’s own networks; Arkin said the company has not yet determined whether the servers that were breached were running ColdFusion, but acknowledged that the attackers appear to have gotten their foot in the door through “some type of out-of-date” software.

Stay tuned for further updates on this rapidly-moving story.

Update 4:38 p.m. ET: Adobe has released a statement about these incidents here and here. A separate customer security alert for users affected by this breach is here. Also, in a hopefully unrelated announcement, Adobe says it will be releasing critical security updates next Tuesday for Adobe Acrobat and Adobe Reader.

Update, Oct. 5, 4:35 p.m. ET: Rakshith Naresh, a product manager at Adobe, said in a Tweet yesterday that the breach did not involve ColdFusion vulnerabilities.

Update, Oct. 9, 12:50 p.m. ET: Naresh’s Tweet stating that the breach didn’t involve ColdFusion servers was deleted at some point. I followed up with Adobe via email: An Adobe spokesperson said the company’s investigation is still ongoing, and that “at this time we have not identified the initial attack vector to include or exclude a ColdFusion server.”


140 thoughts on “Adobe To Announce Source Code, Customer Data Breach

  1. unknown

    visa issued an alert yesterday of adobe being a compromised company. If they were hacked because they were running an old version of their own software, that would be EPIC.

    1. fahlamn

      If they were hacked because they were running an old current version of their own software, that would be EPIC.

      1. meh

        I’m sure it had an auto updater – it was just waiting around for the 3 month window to self update or the lone admin to go find the machine.

  2. Phil

    NOW it makes sense why their products cost so much money: They’ve been padding their legal settlement accounts for just such an event! #anythingbutadobe

  3. brokenlinks

    link to Adobe’s statement is broken and I need lulz. halp!

  4. DefendOurFree

    The court docket systems used Adobe PDF. I think there was a breach of the source code well before August 2013.

  5. Rick

    The first link in your addendum links back to your story, not the Adobe statement.

    BTW, can you stop with the gray text on white? Difficult to read with that lower contrast. Thanks.

    1. nikol

      hit Firefox….options…content…colors…background…AND Wow, choose a color, any color.Turquoise looks killer on this site, then …..uncheck allow pages to bla bla….ok ….and ok

  6. Rick

    was alerted by a co-worker that Adobe is advertising for a “Director of Security” job….

  7. QHoster

    Amazing again how everybody is keeping everything online unprotected. Source code and more.

  8. PFM

    Brian: you are on a roll! Whats next? Keep up the excellent work.

  9. Jonathan

    There are unconfirmed reports that Adobe’s firewall was made out of actual bricks of adobe.

  10. stratosphervert

    Guess it’s MoldFusion at this point… hold my beer.

  11. pblakez

    Its scary to think how many breaches have been through ColdFusion over the years

    I some ways worse than java / flash / windows as it is deployed by big companies and governments on their servers where as the others are mainly targeted at end users on their workstations

  12. Diane Trefethen

    They say you can’t fix stupid and boy does this case prove the point. Does anyone know why large corporations persist in leaving their confidential and valuable information on servers that are online? Convenience? What’s convenient about picking up the pieces in a situation like this? Perhaps LAZY might be more accurate.

    1. anymouse

      I have no idea if the following is valid for Adobe, but here’s an example of why a corporation would put sensitive data on servers accessible via the Internet: a company has software developers in Mountain View, software testers in Delhi, e-commerce and website support in Boston, customer support in Bangalore, legal in NYC, manufacturing in Shenyang, and a main warehouse in Omaha.

      There are ways to mitigate this risk, e.g. VPNs, secure servers, encryption, complex passwords, etc, but management often demands unrealistic deadlines so employees sometimes cut corners. Not to mention incompetence, corporate espionage, or internal theft.

      1. anymouse

        Oops, I forgot to include the ever-growing trend of BYOD. Many employees demand to use their personal devices, yet refuse to allow the company’s IT department to ensure that the device will not leak data or contaminate the company intranet. I’ll bet Adobe is chock-full of BYODs.

      2. noway

        you have hit the hammer on the nail with that statement!

    2. Janantha

      I bet it wasn’t stored on a directly connected server(s) but the defense in depth wasn’t placed well enough at core and after the initial insertion using what ever vulnerability, escalating to secure areas might have been easy.

      1. v

        (disclaimer: I work at ADBE, but I’m just a peon, don’t know much more than any of you about the incident).

        I don’t know how they breached the system, but people here are not completely stupid. To access the source code you need to be on the intranet and to use two-factor authentication (pin + physical token – different than your LDAP password); plus, not all people can access all repositories. For production data it’s even harder…. you have to gain access to a “bridge” machine that is the only machine with connection to the production system databases, and then from there you can authenticate. And even if you get on to the production system and gain access to customer data (emails, encrypted passwords – yes, they are salted), you still don’t have access to credit cards – everything related to payments is on a separate production machine that is isolated from most of the system; I’ve never had access to one, don’t know exactly how they are secured. I’m pretty sure that the breach didn’t happen via the web servers – those are isolated machines with pretty much zero access rights.

        I’m not saying Adobe didn’t fuckup the security of the data – obviously it did. But don’t assume that people here are stupid and don’t even follow the minimal security best practices. This was most likely a sophisticated attack, not a simple webserver exploit.

        (disclaimer: obviously, being a just a peon, I don’t know how all production systems are set up. But if a relatively small one is set up like I described above, it’s pretty hard to imagine that the security of the “crown jewels”is more lax).

        1. Jack Cain

          With all due respect to the hard working folks at Adobe, I know of a situation where someone left passwords to the “crown jewels” of a major company on a wiki. Sometimes normally intelligent people take risks through personal justification and others suffer the consequences.

          The attack could have been as sophisticated as someone being lazy and sharing a password with a supposedly trustworthy co-worker. I’ve seen passwords shared with overseas contractors in order to meet due dates. If you insist on safe practices, you are viewed as trying to hurt your company, division or department because it “CERTAINLY can’t happen to US!”

          Even two-factor is easily beat by this fallacy inherent in human nature – “Here, just enter this, but I need that fix by tomorrow morning” is a phrase uttered by too many developers.

  13. The Utah Data Center/N.S.A./ Area 51/Room 641A/XKeyscore/PRISM/Mainway/Marina/Meta Data

    I just read about this on a Zdnet newsletter. Nice how you make the connection between the LexisNexis, Dun & Bradstreet and Kroll database hack with Adobe.

    Was the encrypted hash information salted ?

    I wonder if B.K. is getting a security bounty from Adobe in Bitcoins or maybe a nice t-shirt with a company logo on it :–)

  14. anymouse

    I have no idea if it is relevant, but a week ago I wiped a W-7 PC and reinstalled everything. Adobe Flashplayer installed as it has for months. But today I wiped and reinstalled another W-7 PC, with Adobe Flashplayer having a completely different screen (I entered the URL by hand and verified it, so that was not the issue). Try it and see. Maybe I will rewipe and reinstall this last one.

  15. andy1

    Wow, great reporting! You are all over everything current in security. I heard your interview on NPR on the Silk Road bust. Nice job keeping at the fore.

  16. john

    ha ha ha, we knew this about adobe 20 years ago, that this would happen, so a lot of people back then went unto free systems where you could get abode for free, because you knew that you credit card would be taped, etc, no surprise here

  17. Warrior

    You would think that a company like Adobe with so much on the line would have state of the art, cutting edge security protocols in place to prevent this very thing from happening. They have no one to blame but themselves. Why wasn’t this highly sensitive info encrypted so in the event of a breach, the invader would be met with random nonsense impossible to descramble?

  18. john

    What idiot leaves source code on a server open to the outside?
    Fire that security manager TODAY

    1. bob

      Any company with more than one office. Or are you assuming VPNs, etc are infallible?

      1. anymouse

        Yeah, no kidding. It is amazing how many people still think corporations house all employees on one campus. Outsourcing has a major security flaw. Not to mention allowing people to work from home.

        VPNs and intranets all run on the Internet. If they are not set up properly, the bad guys will be able to read your data.

        If you want to be able to outsource yet still have 100% security, pay the big bucks for T1 lines between all users.

  19. Fill

    Funny how chilling the source code release is, yet open source doesn’t seem to have these sorts of security-through-obscurity breach problems.

  20. Janantha

    @Diane Trefethen, I bet it wasn’t stored on online but the defense in depth wasn’t placed well and after the initial insertion , escalating to secure areas might have been easy.

  21. Jack Cain

    I wondered why I suddenly started getting massive spam and odd network performance about 3 weeks ago.

    It’s interesting that I did not see anyone concerned for the 2.9 million customers (like me) who now get to spend untold hours checking records, bank accounts, credit reports and reloading systems.

    Glad someone outside of Adobe is checking for lost data…

    Topic 2: Creative Cloud is a requirement if you wish to buy Adobe’s consumer software. How are we supposed to keep our accounts & systems secure if we have to subscribe in order to have the latest software & patches that are supposed to keep our systems secure…or should I call the Post and just give everyone my private info?

  22. Mike

    Hey Brian,
    Thanks to you, I was able to change my Adobe passwords 12 hours -before- I was notified by Adobe.
    Many thanks and keep up the great work.
    Mike

  23. Peter

    Thanks for the warning!

    I notified my family in the States. They did not hear about this breach until I warned them by mobile phone. I wrote a small article here in the Netherlands on this site (https://www.security.nl)

    Kind regards,
    Peter.

  24. Andre Richards

    You know, every company that wants to jump on the SAS bandwagon or try to figure out how to wring recurring payments out of their customers needs to make security a top concern but I don’t see that happening. How is it that every company that handles credit card and financial transactions have to comply to mind-numbingly strict PCI standards and companies like Adobe appear to leave the door wide open?

    1. bob

      Because compliance with PCI standards is fairly meaningless. I’m sure Adobe were fully PCI DSS compliant.

  25. DefendOurFree

    Just adding for reference:

    http://en.wikipedia.org/wiki/Dmitry_Sklyarov

    Dmitry Sklyarov, a Russian citizen employed by the Russian company ElcomSoft visited the United States to give a presentation called “eBook’s Security – Theory and Practice” at the DEF CON convention in Las Vegas, Nevada. On July 16, 2001, as he was about to return to Moscow, Skylarov was arrested by the FBI and jailed for allegedly violating the United States’ Digital Millennium Copyright Act (of 1998) by writing ElcomSoft’s Advanced eBook Processor software.[1]

    The original issue came to the attention of prosecutors when Adobe Systems, a US company, complained that copy protection arrangements in its e-book file format were being illegally circumvented by ElcomSoft’s product. Adobe withdrew its complaint, but United States Department of Justice prosecutors (under the authority of local U.S. Attorney Robert S. Mueller, now Director of the FBI) declined to likewise drop the charges. ElcomSoft’s product, and thus presumably the efforts of its employees including Sklyarov, were entirely legal in Russia. Sklyarov was eventually released on bail, but forced to remain in California, separated from his family, until his case concluded.

    The day after his arrest several web sites, coordinated from the website freesklyarov.org, and mailing lists started to organize protests against his arrest, many of them under the slogan “Free Dmitry” or “Free Sklyarov”. The main point of these campaigns was that no DMCA violations were committed at DEF CON, and the DMCA does not apply in Russia, so Sklyarov was being arrested for something that was perfectly legal in his jurisdiction. A campaign to boycott Adobe products was also launched.

    On July 19, 2001, the Association of American Publishers issued a press release announcing their support of his arrest. Adobe initially supported the arrest, but after a meeting with the Electronic Frontier Foundation, they issued a joint press release on July 23, 2001, recommending his release. However, Adobe still supports the case against ElcomSoft.

    After Sklyarov was arrested he was held briefly at the North Las Vegas Detention Center; then he was held in the Oklahoma City Federal Prisoner Transfer Center until August 3, 2001, when he was transferred to the Federal building in San Jose, California. On August 6, 2001, Sklyarov was released on a US $50,000 bail and was not allowed to leave Northern California.

    The US government agreed to drop all charges filed against Sklyarov, provided that he testify at the trial of his company. He was permitted to return to Russia on December 13, 2001.

    On December 17, 2002, after a two-week trial in San Jose, California, a federal jury found ElcomSoft not guilty of all four charges under the DMCA.

    The case raised some concerns particularly since it involved an individual being prosecuted for activities that were fully legal in the country where they occurred.

  26. noone

    I find it very funny, how times again nobody questions the practise of storing creditcard numbers in the first place.

    Yes, if companies did not store our cc-numbers, we would have to freshly enter them every time we make a purchase. That means typing like 16 digits. Every single time. No, that’s just too much to ask.

    Those lazy companies shall employ better security measures. Store our cc-numbers, so we don’t have to type, but make them 100% safe.

    100% safe doesn’t exist.

    Companies should not store any data on you, except for the absolute minimum required.

    Storing cc-details is not required. Ever. It’s only for convenience.

  27. Peter

    You have to be kidding me. The attackers got in from software THEY develop that had not been patched that they themselves advised people to do.

    1. Just Curious

      Nah. Probably java exploit to some developer at Adobe’s laptop, payload, ‘oh I have something at Adobe, neat’, RAT, VPN, attach to process after dev completes sign-on (maybe leaves it on; many don’t time out, don’t know about Adobe) or else they just grabbed a local checkout on the dev’s poorly-maintained box.

      BTW Brian, not that my opinion counts for much, and I’m only going to make a suggestion, but my guess from what I’ve seen of these guys is that it probably wasn’t the ssndob people that actually breached. More likely it’s something someone shared with them. Did I miss a conclusive connection between the source being there and them doing the deed? If so I apologize. Equally if it is something to remain private, I shan’t prod, just curious if there has been a definitive connection. Thanks.

  28. Madmax

    Funny, Adobe says in an email that they have reset my password. Just logged onto my CC account with my existing password. Hmmmm

    1. peter

      not even received an email advising yet. Changed the password myself.

Comments are closed.