Nationwide restaurant chain P.F. Chang’s Chinese Bistro on Thursday confirmed news first reported on this blog: That customer credit and debit card data had been stolen in a cybercrime attack on its stores. The company had few additional details to share about the breach, other than to say that it would temporarily be switching to a manual credit card imprinting system for all P.F. Chang’s restaurants in the United States.
In statement released to this reporter this evening, P.F. Chang’s said it first learned of the breach on June 10, the same day this publication pointed to evidence that the eatery chain may have been compromised. Their complete statement is as follows:
“On Tuesday, June 10, P.F. Chang’s learned of a security compromise that involves credit and debit card data reportedly stolen from some of our restaurants. Immediately, we initiated an investigation with the United States Secret Service and a team of third-party forensics experts to understand the nature and scope of the incident, and while the investigation is still ongoing, we have concluded that data has been compromised.
At P.F. Chang’s, the safety and security of our guests’ payment information is a top priority. Therefore, we have moved to a manual credit card imprinting system for all P.F. Chang’s China Bistro branded restaurants located in the continental United States. This ensures our guests can still use their credit and debit cards safely in our restaurants as our investigation continues.
We have also established a dedicated public website, pfchangs.com/security, for guests to receive updates and answers to their questions.
Because we are still in the preliminary stages of our investigation, we encourage our guests to be vigilant about checking their credit card and bank statements. Any suspected fraudulent activity should be immediately reported to their card company.
We sincerely regret the inconvenience and concern this may cause for our guests.”
Asked for clarification on what manual credit card processing means, a spokesperson for P.F. Chang’s said “all domestic P.F. Chang’s branded restaurants in the Continental U.S. will be retaining the carbon copies. P.F. Chang’s is also deploying dial-up card readers to restaurants that will be plugged in via the PSTN fax line and used to process the slips.”
This manual check-out process was actually witnessed today by an incident handler at the SANS Internet Storm Center, who reported that “the bartender placed the bill down along with a manually run credit card from one of the ole’school card imprinters.”
Well, maybe there is something to be said for security by obscurity.
they should switch to bitcoin. they can take both bitcoin and credit card. i guess they had to break out the old school credit card machine and carbon paper
“they should switch to bitcoin”
Because Bitcoin is SOOOOOO secure and never stolen…..
You know… this proselytization is getting old. When the topic is about credit card breaches, the last thing security minded IT people need is some evangelist coming around to preach the salvation of bitcoin. What we need is some rational discussion about the security issue itself because our management isn’t going to come to a meeting and say “So, we’re switching away from the near-universally accepted payment card paradigm towards something hardly anyone else ever uses. What should we go to?”
Speaking for myself, but guessing that others will agree: It would be nice if advocates would stay on topic and not bring bitcoin up when the issue isn’t directly about cryptocurrency. It’s relevance is ***well** down the list on a topic like this.
As a security professional, I *COMPLETELY* agree.
Cryptocurrency is neither universally accepted nor is it terribly secure. Enough with the preaching!
The best solution to prevent breaches from happening would be end-to-end encryption on the point of sale it. End-to-end means the card information is encrypted at the swipe versus the POS software encrypting it. These hackers inject software into the POS server(s) that ports to all locations then captures the card information before it hits the POS software. With end-to-end the only information that could be seen is encryption keys. This isn’t the same as the card being encrypted into a token (that’s how companies store cards on file for recurring billing).
Hear, Hear!
(I hope that’s how you spell the cliche, because people should hear what you said.)
You put it well. Switching to bitcoin doesn’t treat the problem. Thanks!
bitcoin is not guranteed by any regulation so you are on your own
not carbon paper it’s a piece of paper that when you run it over has carbon built in so it’s only 1 piece now not like before with 3 parts and carbons
> carbon paper
Just remember, the Ruskies went back to using using typewriters.
Rocking it old school with the imprinter. Priceless!
Waiting for the first card that doesn’t get seated properly and bent in half when the cashier tries to slide it.
—-
On a serious note: I wonder how they handle disposal of old carbons. I honestly don’t know how stores did it in the past because back then I simply didn’t pay attention.
A large portion of deployed Debit and some Credit Cards are now flat. Nothing to imprint… If you have one of these cards you won’t be eating at PF Changs.
Dial up will be their best short-term option. Retailers need to isolate their POS Sale Systems. The cost in lost business and reputation damage will eventually force retailers to invest in better security. Market and bottom-line brings about change faster than anything.
A side note, Vaughn,
When a card is flat, the cashier copies the number onto the carbon form. Press firmly. 🙂
they will use the swipe app like square or something similar
Brian –
I am in no way a computer expert – i couldn’t define the difference between shockwave and flash if you paid me to – yet your bio in the New York Times a while back turned me on to your blog. Keep up the good work and keep looking out for the little guys like us.
edit – little guys like me! sorry about that 🙂
Yeah – let ’em sniff those analog fax “packets”, HA!HA! It is way cheaper than mailing them! Geeze – when I remember those bad old days, it seems incomprehensible the way business used to work! :p
So credit card imprinters and PSTN lines are considered secure? Wow.
I agree, manual slips floating around is just as dangerous.
Jack — what makes you think that the manual slips are going to be “floating around?”. When you go into a restaurant and the terminal issues you a receipt, does the restaurant copy “float around?”… my guess is that the slips would be stored along with the merchant copies of the POS receipts and all the other confidential merchant documents in a secure location. Seriously, the things people say!
Seriously. Its bad enough that the credit card leaves the customers sight. But a PAN in the clear until the manual slip is secured is just waiting for it to be copied or photographed with a cell phone. You would be surprised what could be accomplished in the organized chaos of a high volume restaurant.
So you take a picture of the carbon. What are you going to do with it? You can’t create a card from it, because you didn’t get a read from the magstrip. Therefore you can’t make purchases with it in person. You don’t have a picture of the CVV2 on the back of the card, therefore you can’t make a purchase with it online.
Unless you’re going to go back to PF Changs with a stamped card and charge some more meals before they fix their system, this really isn’t going to net you a whole lot.
It is not that the CC slips cannot be stolen, it is that they cannot be stolen in the same quantity as infecting a chain’s entire point of sale system. In addition to far less potential reward, stealing the slips would require a huge increase in risk to the thief as he/she would have to be local to the area and not hiding in another country.
It’s more secure than a compromised system, no doubt.
BING! We have a winner.
+1 Not to many ID thieves are going to want to do theft the old fashion way – too much work!
There’s something to be said for taking a pretty radical security step that could decrease sales numbers by being slower. It shows they care for their reputation and customers over profit. <>
Also, just goes to show that having everything computerized and networked isn’t always a great idea. First financial data, then identity theft, next the “Internet of Things.” Connectivity is the enemy of security.
I believe they are using what we call “knuckle busters” for the credit card processing.
They are call credit card imprinters, and if a card does not swipe or a merchant’s system is down, they are a essential business tool which allows the merchant to prove that the credit card was present at the time of the transaction. As with any important document, the onus lies on the merchant to ensure the merchant copy is stored in a secure environment.
The carbon copies represent a huge security risk.
Not really.
Yes, it’s relatively easy for a person (who’l be fairly easy to find once the fraud become apparent) to photocopy the lot and walk out with them, but that’s only one store.
However, it’s not on the same scale as 282,000 cards stolen anonymously.
I would say that CC carbon copies are just as good as cash. So they are just as valuable as cash and they should handled that way.
I’ve had four banks in the last 5 years, and none of the debit cards would work with imprinters. The numbers were printed on them (and in one case, printed using a card printer on the manager’s desk while opening my account)
Thing is, you should never use a debit card outside of a trusted ATM / bank branch. The risk / liability is much worse for debit card-counting than credit cards.
That said, I’m more interested in how they’ll deal with Credit Cards that don’t have raised numbers:
http://www.oes.org/page2/39731~My_new_credit_card_doesnt_have_raised_numbers_anymore.html
They do it on purpose so that when you go to a restaurant/etc the waiter can’t do a rubbing and copy the info. Imprinting is very rarely used legitimately these days so they figure the fraud from unwanted ‘imprinting’ could be lowered by not embossing it.
Is that why they do it? I can’t imagine the sole purpose of a card issuer to not emboss cards is to prevent a server at a restaurant from rubbing it. Any facts to support this claim…
No embossed card production costs more. That is why you are seeing more non embossed cards. You don’t have to have an imprint of the card to make a transaction. The card network will accept a hand written copy or touch tone phone. Card imprinters (knuckle busters) are no longer a certified way of card acceptance. However they can be used if you have a communication outage or in this case a breach of your card terminals. The real problem lies in the way the current payment system is designed. The problem will not be going away anytime soon despite what you may read or hear on the news.
Once you remove software (code) and Internet access from the credit card transaction at the store, the criminals in Eastern Europe will need to change their tactics. I do not believe their malware will run on physical card swipe machines. Kudos to PF Chang for taking this measure albeit it is a stop gap approach 🙂
You charge something there, an hour later you’re hungry again (so you get charged again )
Dang, that was funny!
HA! :v
This almost makes me think they would have been better off with a card swipe reader plugged into an iPad. Fewer common points that might have been corrupted.
Even with that suggestion, the information would have still been in jeopardy: all of those numbers have to go back somewhere in the back office for processing.
Don’t the manually imprinted card slips still have to be inputted into some system for authorization with a card processor? If so, does this indicate that Chang’s believes that it’s the POS that’s compromised, not their authorization servers?
Naw, that would just change the attack target from Windows POS systems to iPad based POS systems. Remember, you are still relying on a software vendor to write the code to read and transmit the CC information. If that vendor screws up, it won’t matter that it was running on an iPad.
Thank you Brian for informing everyone about these criminals and their crimes.
Do you have any thoughts about your revelations compare with Snowon’s disclosures?
The credit card carbon copies will be stolen from a employee and the thefts will continues . Are the people who run this company total idiots?
Nope they run a business that requires folks to give them compensation. They try to do what is best for the customer and still remain open. We all lived for years with paper based machines and saw smaller amounts or just less reported amounts of fraud from credit card thefts, But with paper you can track it down to a person quite quickly. They get caught. With the electronic versions we see that our CC info ends up in Russia where no one gets caught and they get to continue stealing.
I’ll take the paper version any day, but I pay with cash most of the time. Always nervous using the network attached solutions.
I have trouble calling then idiots for staying in business and trying to accept what we use for money these days.
I have to agree. I’m old enough to remember those stupid carbons so the first thing that popped into my head was them swirling around in the alley behind the restaurant. But your remark makes sense, and I do think PF Chang’s is showing a bit of courage going that route. I never eat there but have friends and family who adore the place. I’m telling them take cash.
I also remember the olden days of manual card processing. In the later years of that “technology,” when identity security was first becoming a topic, most merchants automatically asked, “Would you like your carbon?” If you said yes, the merchant tore off their original and handed you both the customer copy and the carbon, still attached. It wasn’t messy if you paid attention to how you folded it.
And the only way to check stolen cards was a credit card number book that was as thick as the phone directory for New York city!
Man! I’m really showing my age here! 🙂
LMAO.I remember those books !!!!!!!
Likewise!
Those fat books were not replaced with updated copies very frequently… it would be months before a stolen card would show up in there!
The use of carbon copies with using those credit card slide machines was before the main stream use of the internet and the wide scale use of online commerce.
No they are not idiots, but just like you buy a car from a manufacturer because you don’t have the equipment and skills to build your own, these companies buy their POS systems from a POS manufacturer. Obviously, these POS systems along with the fact that we are using 40 year old CC magnetic strip technology is not a secure as we would have hoped.
note on complexity, obscurity and scale:
The obscurity is a secondary characteristic in use of manual and POTS card readers. what this does is essentially air gap the readers from each other. it is de-scaling, if you will.
They may have more incidents of carbon and pots reader hacks, but the scale of those incidents is constrained by the physical presence necessary to do the hacks (assuming physical interaction to initiate reader software changes).
Also interesting is that people can see the need to handle, store and dispose the carbon(viewing it like cash) with great care. No one would suggest copying names, amounts, store loyalty numbers etc with this type of system, yet big data collection does this. in fact, the whole carbon thing is viewed more as a liability rather than an asset.
The store based systems collectively are in some ways a more complex system (carbon, pots machine maintenance,etc. ), yet it is any more obscure (many know how it works and where it is located ). The removal of the network does de-aggregate, or de-scale so it’s arguably less complex AND smaller scale.
my sense is that infosec is still struggling to handle the problems of scale, as evidenced by infosec’s use of cute catch phrases like ‘security through obscurity’ and ‘complexity is the enemy of security’. It’s as if scale is a gorilla in the room that nobody wants to admit is a multiplier.
Given the concern they should change to cash. I’m told its difficult to get credit card information off the bills.
Good point mbi! The sad truth is that there will always be people out there trying to take something that doesn’t belong to them. I think PF Chang’s should be commended for doing their part to protect themselves and their customers
If PF Chang’s did there part to protect customers in the first place we would not be reading about them today.
I believe Lee Church has understood and characterized the P.F. Chang’s response correctly.
Yes, the carbon imprints do open up another exposure to possible theft/fraud, but that exposure is a tiny fraction of the possible exposure if their electronic systems are compromised.
Also, the PSTN-connected credit auth terminals are, and always have been, much more secure from a transaction-sniffing on a mass-scale. With a direct dial-up call to the processor, you have to tap the phone line, which is far more difficult than sniffing IP traffic.
Of interest here to me is the speed with which P.F. Chang’s has responded and rolled back to paper/voice auth and non-integrated dial-up credit auth terminals.
It is not a method/idea that necessarily scales well to large multi-lane retail (think Target, Wal-Mart, grocery stores, etc), but for a restaurant or specialty retailer (Sally Beauty comes to mind… and potentially even something as large as Michaels with 4 – 6 check out lanes being fairly typical) it is a very viable way to provide business continuity while eliminating the possibility of further breach of customer data.
To have moved so quickly, it would appear that they must have had a contingency plan in place for this type of event.
Was with friends at PF Chang’s yesterday (6/12) and the excuse they used when they returned with the imprinted receipt was, ‘our credit card system is down’. Thanks to BK now I know why their ‘system was down’. I agree with mbi, cash should be king in the majority of our purchases.
You would think that after Target, everyone would be checking their systems, so they wouldn’t become the next Target. Guess it would cost too much money and effect executive bonuses.
Evidently the losses from all these breaches have only reached the “unavoidable annoyance stage”, from the macro-economic P&L point of view. Sure, Target has lost $10 billion in market value since this time last year. Sure the problem is costing many additional billions annually in direct losses, legal and consulting fees, lost customers, etc. etc. But hey, chip and pin will save us as long as people don’t mind a little inconvenience. Yep, that’ll work.
What does PF stand for? Payment Fraud? Well, at least they now can Chang-e their setup and CIO/CISO … 😉
Once again the discussion is how the data (the card number) is being transmitted.
I agree that separation of the card processing from the [internet connected] POS terminal is a good thing. Whenever I see a merchant slide a card into a separate, POTS-connected card processing terminal- I smile a little inside.
However the data is transmitted or secured, the fact remains that it’s a static number that is vulnerable to unauthorized replay attacks by its very nature.
The answer: dynamically changing, one-time-use, account numbers that are mathematically linked to a “primary” account.
Would merchants need to incur the costs of changing their Point-of-Sale (POS) systems significantly?
Not necessarily.
A company named Dynamics Inc. based in Pennsylvania has a product that can encode [one-time-use card] numbers onto the magnetic stripe(s) on the back of the card.
This enables *standard*, *existing* POS card readers to work seamlessly with the newer [card] technology.
See Dynamics Inc.’s webpage here: http://bit.ly/19fbXKb
(last archived by archive.org on Oct. 1st, 2013).
A card presenting a number on its mag-stripe that is only good for one transaction at a time, cannot be [re-]sold by criminals. Whether or not card data is stored at (or scrapped from) the POS terminal is irrelevant if the data itself (the card number) changes with every transaction.
Hold the Payment Card Industry (including issuing banks) responsible for not (years ago) embracing technology that could all but eliminate skimming/re-use fraud.
Discover Card Services had the best idea I’d seen in a while. They used random numbers issued to a specific vendor, if anyone but that vendor ID tried to use the card number, it would not process, denying the criminal a dime!!
The only trouble is, they dropped it!! Apparently they would rather put up with the losses than pay for one of the best security systems I’d ever seen! They called it “Online Secure Account Numbers” – but I see no reason why the same thing couldn’t be done through a POS device. Once again I realize POS devices can be cracked, but so can online merchants – the problem for the criminal is he can’t use the information unless he can fully pose as the merchant with whatever ID system the card industry uses for that – I’m sure the crooks would figure that out too – but the way I see it, is it would just be a matter of reprogramming instead of adopting a ridiculously expensive system like Cow-Chip-N-Pen just to have that compromised as well.
If I was the guy that hacked their system, I would tell everyone I did it just so I could say “I hit PF Chang’s so hard I knocked them back into the 20th century.”
*rimshot*
Now that is a hoot!
Just an aside, many years ago, I was the victim of credit card theft via a stolen carbon. The thief still needed my address, however, in order to use the purloined credit card and she got it by masquerading as an employee for the store where this happened. She said they were going to start a catalogue and would I like one? Unfortunately, to get the HSN and Talbott goods she ordered, she had them delivered to her at her home address. I reported it to American Express, the police and the store. I was happy when carbons were discontinued.
VeriFone has Verishield Protect or VSP on select pin pads. It is end to end encryption and they carry the liability. Reduces the PCI scope tremendously. Not sure how to apply to table service though.
Using cash avoids the worry of compromised debit/credit cards. We use all cash and have one debit card for online shopping and travel. Works great, and won’t have to worry about a retailer or restaurant being compromised and someone swiping the money in our checking account.
Last Wednesday my wife and I celebrated our 31st anniversary at P. F. Changs. We had a nice dinner and they ran our card through a manual imprint system. Today I noticed that they wrote my CVV on the receipt. That is not necessary if these are actually manual credit card transactions. The CVV means that they plan to process the charges later as card not present transactions. For those of you who have jumped through the PCI hoops, you have to wonder what measures they are using to protect these receipts since they include the CVV. I am really annoyed. Did they replace a high tech security breach with a low tech security breach?
In UK we use chip and pin, that provides some additional security, but does not protect against a cyber attack CC’s internal systems that is clearly not secure enough.
Just a heads up: Manual credit card imprinting will not work for some types of cards. Mine does not have raised numbers on it; they are laser-etched into the metal. If you have a Chase Sapphire Preferred card, better take cash or another method of payment if you plan to dine at PF Chang’s.
Mr. Krebs: Thanks for this followup.
Are there well-known best practices for post-breach crisis management?
It would be instructive to be able to evaluate how well these companies are actually responding.