February 4, 2014

Ever since news broke that thieves stole more than 40 million debit and credit card accounts from Target using a strain of Point-Of-Sale malware known as BlackPOS, much speculation has swirled around unanswered questions, such as how this malware was introduced into the network, and what mechanisms were used to infect thousands of Target’s cash registers.

BLACKPOS copyRecently, I spoke at length with  Tom Arnold and Paul Guthrie, co-founders of PSC, a security firm that consults for businesses on payment security and compliance. In early 2013, these two experts worked directly on a retail data breach that involved a version of BlackPOS. They agreed to talk about their knowledge of this malware, and how the attackers worked to defeat the security of the retail client (not named in this story).

While some of this discussion may be geektacular at times (what I affectionately like to call “Geek Factor 5”), there’s something in here for everyone. Their observations about the methods and approaches used in this attack point to an adversary that is skilled, organized, patient and thorough.

So you first saw BlackPOS at a retailer in early January 2013?

Tom: Yes, it did seem like a game changer at that point because of the way it hooked into the POS system. By that I mean the fact that it hooks into the POS process, and it’s not just a general memory scraper like some people have stated.

Help me understand the distinction between a memory scraper and malware that runs completely in memory?

Tom: Well, it’s very specific. It’s what’s called an inter-process communications hook. If you look at a memory scraper — so if you were to dump the memory on your laptop right now — you would get this big old blob of information and you would have to filter through it to find what you’re looking for. But this [malware] is very specific: It’s very much designed for the POS system it’s running on, because it knows exactly where to hook and where the memory location is going to be when the data it’s looking for is flowing through it. But if you look at what it captures, it captures only the track data [information stored on the magnetic strip on the backs of cards]. So, it’s actually very, very sophisticated and that’s why I think this isn’t just a teenage hacker who put this together in his basement. I think this is a more sophisticated development effort. [HP last week published some interesting, uber-geeky details about the memory behavior of the version of BlackPOS found at Target].

Paul:  It certainly hooks into a specific process, but we did not figure out if it was just good at scraping the memory of that process, or whether it actually altered the process to hook into the code somehow.  That part of the code was obfuscated and not reviewed during the engagement.

What did you think of the iSight paper?

Tom: The iSight paper was good, and what it described was very similar to what we saw in the first variants of BlackPOS. But it didn’t talk about how it appeared on the network or where it came from or how a retailer might defend themselves against it.

In your prior experience with BlackPOS, has it been used against the same POS that Target uses? A source who seemed to have a clue told me that their setup was somewhat homegrown.

Tom: That I don’t know.  I haven’t really looked at what Target uses. With a homegrown system, the problem is if you’re going to build a process hook, you have to have a test environment, you have to know what you’re looking at and what memory addresses to go after, and that’s not exactly something that gets published.

Paul: Target has a homegrown POS.

So you think it’s likely they were using some off-the-shelf equipment and software? Wouldn’t it be enough for the attackers in this case to have obtained a physical point of sale device that was once used at Target?

Tom: If they have one, sure, that would be different. I don’t know if they’re using a commercial product. A lot of the big retailers use commercial products and will customize those with their back end system. But at the front end and what happens at front of the house…a lot of those are just retail off-the-shelf applications. A lot of those retailers, when you have a hard disk that breaks on one of the [checkout] lanes, they’ve got an outsourced service provider of that POS that comes out there with a new hard disk and fixes it.

How do the bad guys break in, and how do they actually get the malware pushed out to the point-of-sale?

Tom:  A lot of the POS systems use whitelisting software of some kind, such as Bit9 or SolidCore. Those two companies are the two you see most out there in the industry.

Paul: It could also come in through the point-of-sale application update process.

By whitelisting, you mean sort of the opposite approach of antivirus, right? As in, if the file or program isn’t approved by the whitelisting software, it simply won’t run on the system?

Tom: The software update processes at a point-of-sale that is running one of those [whitelisting applications] has to come through one of the software update channels and has to be reviewed for the update and approved. And when it’s approved, the whitelisting software says okay this patch is approved to come online.

It’s probably a good idea at this point to make sure we’re defining the terminology in a uniform way. When you think of point-of-sale device, are you talking about the cash register, or the card swipe terminal, or…

Tom: I’m using point-of-sale to mean the payment application that is running on the cash register. The vast majority of those devices, when you check out at the grocery store or large retailer, those are just PCs, and yes they’re mostly running Windows XP or WEPOS as their operating system. But above that, you have what I call the point-of-sale, or the point-of-sale application, and that’s the stuff that the cashier is interfacing with at the time you’re checking out. It’s a software application, running as multiple pieces of software inside the register itself. And whitelisting is put on to protect the register from any sort of unsolicited modification. A lot of the attacks before this [whitelisting became more broadly used] involved where you corrupt a sales clerk to put a USB stick in the cash register and infect the PC with some malware. But by using a whitelisting software, the USB stick will not work and the operations personnel back in the head office will get notified that something isn’t right with that register.

If they get past a whitelisting system, doesn’t that suggest that someone on the inside would have to be involved?

Tom: No, not really. You have to also consider the distribution server that distributes the point-of-sale software.

Paul: Right… three possible ways… it could come through a legitimate update channel, or the retailer was lax in their update procedures, or the attackers hacked the console of the whitelisting software and just whitelisted it themselves.  

So  when you look at a hacked system, what can you determine about the state of the POS software on the affected systems? 

Tom: Well, there can be huge problems if the retail software vendor does not provide  SHA-1 hashes or even MD5 hashes of their software. There would be no way to tell whether what you were receiving had not been tampered with..

How does the update process typically work?

Tom: Basically the retailer would receive notification of a patch, and go off to an outside server and pull it down. Very similar to you getting notified of a patch being available on a Windows machine. Except in these situations, when you’re dealing with critical retail software and installations, patching is very formal process, because uptime and reliability is very important. So they test it. They would install the patch on their internal lab servers, run and test it to verify that everything would run smoothly. Then they would then certify the patch when it was ready to run, and then distribute it to stores 1-5 tonight, and then go 6-10 then next day, or whatever their mechanism was to distribute them. But it would typically take them 2-3 months to finish the distribution to all the stores.

One of the questions still outstanding in the Target breach, which is where did this stuff come from and how did it distribute itself across the network? Do you have insights or best guesses on that? And wouldn’t the mechanism like the one you just described be the most likely way…some kind of POS update mechanism?

Tom: Well, if they were not running whitelisting software…maybe, yes. But you know the other thing was that the malicious software we have dealt with has sort of a worm-like characteristic to it. It would keep trying to worm its way around the organization. If it found itself already installed on a machine, it was bright enough to elect that machine and it would start seeking a way out of the organization. When we looked at this in the past, it was very difficult for us to determine how they tried to exfiltrate the information. I think what they did was…since they could find a host — remember, this is an early variant of what was seen at Target — they sort of elected a machine that became the exfiltration point and started doing the exfiltration. Worked exactly the same way, in that the data was encrypted, and moved off to an exfiltration point, and from there it was pulled off to Mother Russia.

Paul: Once they have a password that works on every system, they can use that to spread the malware. There would be nothing that that would stop a POS device in store 12 from talking to a register in store 23.

What type of encryption did they use?

Paul: RSA-512 bit.

But this information wasn’t encrypted by the point of sale or card swipe device at that point?

Tom: Well, remember this was taken off the memory, so it was encrypted from the card terminal, but when it got internally so the [cash register] could handle it, the system itself would decrypt it, and pass it to another process that used for communications, and then it would go off network to get the authorization. And the process hook would grab the data at the point that it gets decrypted out of memory. Then the malware used its own key to encrypt it.

So at that point are there two parallel streams of data here?

Tom: Yes. One good, one bad. And then the malware itself, would then encrypt that data, and pass it to where it was going to aggregate it from all these infected point-of-sale devices, and then the aggregation server would encrypt it again for exfiltration. So it’s pretty sophisticated stuff.

So the bad guys in the scenario that you describe were double-encrypting their stolen data?

Tom: Yes. And that’s why….a lot of people ask the question, hold on a second. Wouldn’t  intrusion detection systems flag this; if they see credit card numbers floating around the internal network and moving outbound, they classically raise the alarm. Yet no alarms would go off, if  it was encrypted. In the case of Target, it sounds like they were using those encrypted packets to hide from data loss prevention and intrusion detection systems.

Did the forensic report from iSight say the data stolen from them was encrypted in the Target breach? I don’t recall reading that. 

Tom: Well, if they were using BlackPOS, I’m pretty certain it was encrypted. Because, if Target is following PCI best practices, there’s no outbound traffic moving at that point, and I would expect that they would have an intrusion detection system in there, because that would be required. And in that case, the traffic would have become visible as card data leaving the network. Now there’s nothing that says you have to do data loss prevention like that [in the PCI standard] but a lot of big organizations do.

We were never able to figure out how they egressed the data. The only protocols that were allowed outbound were DNS. So if they wrapped the stolen data up in outbound DNS packets, maybe. We never solved that problem, we were still looking when the engagement ended. We could see it was doing a lot of the work, but we couldn’t see it was actually leaving the organization.

How is that even possible? That you would not have been able to see how they offloaded the card data?

Tom: There were a lot of modules inside of BlackPOS that were anti-forensics modules, so it cleaned up a lot after itself. It did look for a specific time window for when it would actually move data. There was a whole timing vector as to when it would turn itself on, and then it would start seeking things. We put the software in a lab, but it never [sent data outbound] so we couldn’t quite figure out how. In my client’s case, the firewall rules outbound were very strict, and we couldn’t figure out how it was getting the data out of the organization.

You said earlier that this malware had something of a worm-like capability. Can you talk more about that?

Tom: The iSight report alluded to this when it said there are a whole bunch of hacking and forensics tools in there, and it really did have everything but the kitchen sink in there in terms of tools. The actual transfer mechanism seemed to be a VBSCript. When it got involved, it would very gently start mapping the network, and so that it became aware of what was in the network with it. And then it would start seeking out other point of sale devices. It was very careful, and it took its time doing this. It would attempt to communicate with those other POS devices. And if one of those devices was already infected, the two would form a handshake and a bond, and the two of them would look for the third, fourth and fifth, and so on.

If it would see like a manager’s laptop, it would try to push itself there, but then when it got to that laptop – if it successfully infected it — and it didn’t see the point of sale software, it would run its anti-forensics software and completely destroy itself, so that there was no trace of it at all. We actually took the software and forced an infection onto one of our test laptops, and it only lived for about 15 seconds before it completely destroyed itself. And then it went through and did a whole bunch of anti-forensics work before it destroyed the anti-forensics module. Very cute program. It cleaned up after itself pretty nicely.

So this is why I think as an old software engineer, I was looking at it saying someone did a lot of work to make this thing run properly, and they did a lot of testing. This is not just one guy. If it is the work of just one guy, he’s absolutely brilliant.

ANALYSIS

From talking with Paul and Tom — and in discussing the Target case with security experts from other retailers — a few things seem clear (these are my personal takeaways, not theirs). Firstly, many retailers only update their point-of-sale systems according to a well-planned schedule (a schedule, by the way, which typically happens well before Black Friday — the busiest and most profitable day of the year for most retailers). As a result, depending on the size of the retailer, that update process can take weeks, even months. Their experience suggests  that whoever broke into Target was inside Target’s network for quite some time before the point-of-sale malware started stealing card data on Nov. 27, 2013.

Secondly, many reporters and readers have been asking what retailers like Target could have and should have been doing from a security perspective. I don’t have much information on what Target or other retailers were or were not doing, but assuming the attackers typically take months to orchestrate and execute these attacks, it stands to reason that more quickly detecting these intrusions would help quite a bit. According to Verizon’s 2013 Data Breach Investigations Report,  66 percent of breaches that Verizon responded to in 2012 took months or more to be detected. Too often, the breaches aren’t even first detected internally. It’s worth noting that Target’s case, the company’s CEO acknowledged being alerted to the breach on Dec. 15 by law enforcement. In the case of the breach at Neiman Marcus, which exposed some 1.1 million credit and debit cards, the company said it learned of the breach on Jan. 1, 2014, although the actual card thefts occurred between July 16 and Oct. 30, 2013.

Finally, I have to wonder how many times this scene played out in 2013: An individual forensics firm analyzes a sophisticated retail breach involving point-of-sale malware – collecting mountains of interesting and useful data about the threats, threat actors and their methods — but at the end of the day has no mechanism by which to share that information with others in the retail and security community. It’s telling that most of the details about the malware and methods used in the Target breach were first published here on this blog.  That’s not a brag: That’s me being incredulous at how the industry as a whole still sucks at sharing important information.

I hope that this interview helps other digital first responders, and that it fosters a more public debate about the malware and miscreants responsible for what appear to be a large number of very similar data breaches that will no doubt continue to come to light this year via additional victim disclosures (willing or otherwise).


149 thoughts on “These Guys Battled BlackPOS at a Retailer

  1. Ralph Daugherty

    If there’s anything out there remotely as useful as Kreb’s ongoing reporting and analysis, I haven’t seen it. Great work.

    I am an IBMi programmer (the IBMi has run a lot of retail and I’ve worked in it) but spent the 80’s writing low level PC software. I love the 8086 but… these businesses saying they can’t compete with the bad guys (today, in testimony to Congress) aren’t even trying. They use vanilla technology the bad guys have pwnwd and spend their time in useless finger pointing exercises.

    If the POS’s have to be general purpose PC’s, however lazy that is, they should be running hardened Linux on non-x86 compatible CPU’s. Even a casual reader of hacking forensics such as myself knows how much taking Windows and x86 out of the equation breaks so many tricks they depend on. Obviously gaining root on Linux and installing custom malware to run with the POS process is possible, but the PC’s really should be locked down from regular “maintenance” software updates.

    As far as servers go, be it database, FTP, web, you name it, again hardened Linux on non-x86 CPU’s or IBMi will not be broken into any time soon. The businesses could compete with the bad guys, they just like paying less for cheap systems. Not that the Windows infrastructure is cheap, but what they get out of it is.

    Lastly the comment on encrypting card data end to end was right on. But the retailers marketing want to know who you are and risk yours and my information by taking unencrypted information which exposes it to their incompetent security holes because of the perceived marketing value of our data.

    The handwringing is by IT incompetents who only know how to install Windows system and software. I like and use Windows myself for a personal PC, but I have IBMi and Linux for servers. Businesses should do the same, and we wouldn’t be having this discussion.

    1. Bill

      Ralph,
      I like your ideas, and don’t get me wrong, but isn’t this just another lateral move that falls pretty much into the “security by obscurity” pool? What you’re proposing would be a top down re-tool/train. Not that it wouldn’t be effective, for a time.

      Also, didn’t IBM (don’t know much about IBMi, and presumably neither do the bad actors) just sell off their entire server product line to Lennovo? The Lennovo that’s owned by Chinese concerns?

      1. Bruce Hobbs

        Bill, when talking about the IBM i, think of “security through design” and not “security through obscurity.” The big difference between IBM i and Intel is that, on the IBM i, data can not be run as a program. Program objects must can only be created with IBM compilers. Sure, I can write a program that contains malware but I would have a hard time sneaking into a system.

        And, no, IBM only sold the x86 servers to Lenovo, perhaps only 10% of IBM’s server business. The IBM i, the AIX boxes and the System Z mainframes will remain with IBM.

    2. Moike

      “Obviously gaining root on Linux and installing custom malware to run with the POS process is possible, but the PC’s really should be locked down from regular “maintenance” software updates.”

      It doesn’t matter what they run: if it can be updated, they’ll be able to figure out how to get malware loaded, even if it means getting their stuff into a custom embedded microprocessor boot image.

      If you claim there must be no regular maintenance to avoid malware, the company will be forever doomed to live with the first installed version, along with bugs and limitations, while competitors continue to improve their systems.

    3. Jeffery

      Ralph, you make a very excellent point. Switching to a hardened minimal Linux-driven POS software would utterly snuff out many of the attack vectors the bad guys are counting on. The benefits would be manifold. There are tons of security features native to Linux( e.g. easily configured iptables firewall at the device level, file integrity checking in each device, a Host-based IDS in each device, and much more.) Each POS device would become much smarter from a security perspective. The retailers would save millions given that the Linux-based POS software would surely run on their existing hardware. I only wish I had the time to build that distro!

  2. spyderman

    Nothing can be successfully defended. There’s only attack, attack and more attack. General Patton think tank. With the honeypots they had the attackers demonstrate two attacks. One to access files and another to execute files. I think you might want to monitor rather than battle them. Good luck.

  3. Felix Uribe

    I used to build POS system back in the 90s. As mentioned by Ralph and as I saw in the testimony today, we are not even trying, it costs too much!, the malsubjects will continue to have the upper hand in this game,unless we think OUT-OF-THE-BOX when it comes to making business decisions that have become useless in fighting malsubjects and malware.

  4. Ralph Daugherty

    Felix is right. spyderman, I suggest you grab you a copy of a book on hacking forensics and get a good feel with what is used to break into systems. Most of it depends on Windows and x86 CPU, and for SQL injection against a database, SQL Server (the ; statement continuation separator and SQL Server specifics).

    Bill, I understand your point about security through obscurity, certainly that’s a part of it. However, the OS is a big part of the security no matter how well one studies it to make it unobscure. For example, on the IBMi, programs are separately identified from the database. A lot of the tricks on Windows is based on tricking the system into running as a program something that was taken into the system as data because files are files. It’s more than obscurity, it’s actual real security.

    The server business that IBM sold off was their x86 servers. It’s marginal, commodity stuff best left to commodity people and hackers.

    Thanks for your thoughts. I am not into this stuff heavy, just need toi know how to secure my own systems. And I made one purchase at Target in my entire life and it was the week after Thanksgiving so I am reading on this breach and seeing so much useful information here on Krebs.

    1. eddie ski

      I recall that IBM’s early AS/400 line was used primarily for dB and banking industry. No one broke into that mid/mini iron. It was Microsoft and windows that knocked on the banks and practically gave their OS and tools away to get the banks onboard (see the ATMs still running XP).
      FWIW, I was the comsec for several AS/400s and programming.
      BTW, like Ralph stated, IBM sold off the x86 line to Lenovo, not the larger iron. I predict Lenovo will go head to head with Apple and Dell.
      (too bad I worked for an IBM business partner owner whose larceny killed his company)

      1. qsysopr

        I too recall the AS/400. It was rock solid..DB and transaction processing.

        It has a distinct UI and menu system. Its so recognizable that I have seen it out while shopping in large retail stores. There are terminals out on the floor and even saw the screen at the register. Next time you’re out in a big store, have a look. They’re still around.

        Just out of curiosity, I found myself standing next to one of these terminals. I logged in with qsysopr / qsysopr (the AS/400 “root” account equivalent) – WTF?!?!

        Forget about monitoring vendor remote access and encrypted network traffic on stealth channels when you just leave the keys to the castle set to the default creds. Sheesh.

  5. Ralph Daugherty

    ” It doesn’t matter what they run: if it can be updated, they’ll be able to figure out how to get malware loaded, even if it means getting their stuff into a custom embedded microprocessor boot image.

    If you claim there must be no regular maintenance to avoid malware, the company will be forever doomed to live with the first installed version, along with bugs and limitations, while competitors continue to improve their systems.”

    I said regular “maintenance” software updates, such as Windows patches and these commercial product updates. A POS is a simple thing. If there is a major change, it’s worth making an image and refreshing the POS. With a stable OS like SELinux and the extremely limited requirements of a POS, there is no need for a “maintenence’ update as Windows requires.

    Of course there are data downloads such as prices etc., but they go into configured files that the software reads and uses. There is no “patching”.

    I’ve been programming in big business for 30 years, guess what, we don’t sit around worrying about someone getting malware into our systems because they can be updated. When I was with a 1600+ store chain processing sales and doing sales analysis, we didn’t worry about it there either. When you start running your business on Windows, then you’d better start worrying, and even that won’t do you any good.

    1. Moike

      “I’ve been programming in big business for 30 years, guess what, we don’t sit around worrying about someone getting malware into our systems because they can be updated. When I was with a 1600+ store chain processing sales and doing sales analysis, we didn’t worry about it there either. When you start running your business on Windows, then you’d better start worrying, and even that won’t do you any good.”

      Your systems haven’t been targeted by a determined and capable adversary. Let’s say that Target converts to iSeries POS and green screen card swipers. They wish to install an update that improves the application. How can they determine that the application developer has not been subverted, and the application update now contains code to leak critical information?

  6. hhhobbit

    Thanks for the good article. This is huge. Maybe it explains the grocery store I went to replacing their POS systems about a year ago. The new POS systems prominently display Google. I hope that means no Windows POS. Something else is needed, it is needed now, and it needs to be available to Mom and Pop businesses as well.

  7. Chris

    The other missing piece in Target’s security is file integrity monitoring like Tripwire. When new files were written to or modified on the POS this could have been detected.

    Also a question for anyone familiar with WEPOS or POS terminals in general, are these systems similar to thin clients in that the file system is write protected? If not, perhaps this thin client capability might have prevented the attack. Of course a compromise of the patch management system could still spread the malware to all POS systems. But with good checks and balances and file integrity monitoring sneaking a piece of malware into the annual or quarterly patch might be a bit harder. Also patching thin clients makes a bigger disturbance that would likely be noticed if it happend outside the regular schedule.

    Patching thin clients you must first unlock the file system, the thin client has to be rebooted so the file system is writeable, then the file system is locked again and rebooted once more to make the file system read only.

    Combine file integrity monitoring with thin clients and it is much harder target. Though thin clients are pretty limited on storage and memory space, perhaps it is not enough resources for most commercial POS systems.

    1. Fred

      How is Tripwire or any other FIM solution going to protect against a random threat placed anywhere on a largish (>1m Elements) file system?

      Simple answer: It isn’t.

      These tools were designed to target specific files in known directories (via concisely defined rules) and any bad actor worth his or her salt knows how to code around potential rule constraints. In the case of BlackPOS and variants, this malware wasn’t a replacement for any commonly tracked system file. Are you going to monitor the entire hard disk in real time for evidence of an intrusion? Probably not, especially considering the performance impact that such a detection scheme would impose.

      But for the sake of argument, let’s say your enterprise does just that and employs POS systems which are powerful enough to accomplish the feat. Good malware morphs; (dark)CompSci 101. Squared-away malware authors will design their constructs to leverage this fact: What looks like random noise on several terminals will soon be globally dismissed by the beleaguered Tripwire admin whose SOA and CRM application Changed Elements reports for the morning staff review are long overdue.

      Agentless FIM solutions (those which use WMI services, for example) fare no better in this regard – and the access mechanisms themselves might conceivably be used as a malware propagation vector.

      An FIM suite is not an anti-virus suite, much as it’s not a replacement for a proper vulnerability assessment tool…and should never be used as a replacement for either. Rather, its intended purpose is to keep developers and sysadmins honest at code promotion time.

      At best, it’s a proper complement (albeit an important one) to the other tools.

      1. JCitizen

        Well I’m an amateur in security at best, and even I know if you have worm like activity in you network, your UTM appliance should detect it! If it doesn’t, you need to upgrade to a new solution or better yet, fire the company providing it!

        I’ve been probed inside my network before, and been attacked in every way you can imagine outside and inside my LAN, and so far I’m pretty confident I’m at least suppressing malware activity. I NEVER assume I’m safe! I think this is the problem in the industry.

        Of course, I can’t attest to the in’s and out’s of POS maintenance or middle ware, or even the whole infrastructure of a big retail enterprise, but I’m still skeptical at this whole scenario, until they provide me with something I can grasp.

        I really appreciate this article from Brian, though; as it seems we are finally trying to clarify just what the hey happened here, and I suspect all over the world retail structure. CIO’s everywhere should be sitting up and watching KOS diligently for every scrap of information!!!

  8. Sharona

    While I am not in the business as the guests above, thereby I can’t opine on the hows and whys, as a victim I can state this….these people who this, do this because they can. Meaning that there is, even now, no real time traffic monitoring of movement of massive personal or financial data on the outbound side. There is no “scare tactic” or existing iron fist law or enforcement to adequately make one think twice before the hack and thus, the adrenaline accelerates. Most everyone dislikes the idea of “policing” the internet. Yet, it seems to me that more damage will have to again take place before people realize that some “control” is needed. Got to reinvent the firewall and make it MIT smart, together with some red flag kill switches and the institution of the tunnel monitor. Lastly, those nefarious RAS and VPNs seem to be just masks for the strangers who wear them.

  9. Sharona

    Oh, forgot to add that word “security,” which fill some people with false illusion. Perhaps a better EVN chip & pin system might work. Alternatively, maybe we can get some of those NSA folks to share their wares with the rest of the world. Hell, our government could use the money.

    1. JCitizen

      Or if in fact those gubbamints aren’t involved in your criminal scenario in the 1st place? All you have to do is witness the robbing of socialist forces of every financial entity world wide to realize all of the world is in debt, and we are all living under a house of cards, waiting for the collapse!

      Am I worried? NO. I have a bug-out bag and a simple plan! HA! Now permit me to remove my tin foil hat for a moment! 🙂

  10. Ryan Ferris

    I think comments here and at the Senate hearing that depict this attack as sophisticated and complex are excuses. *This is* the current level of cyber crime sophistication. The AF features are the level of sophistication to which cyber crime has evolved. The “mistake” the BlackPOS rings committed was taking so many card numbers at once; probably thus triggering discovery and publicity and outrage.

    Corporations and the financial industry and our government are not doing enough to stop cyber crime from gutting our financial infrastructure. Perhaps they are too busy worrying about the next big terrorist attack to stop the one going daily underneath their noses….

  11. Gavin Phillips

    Awesome article guys – two things stand out to me whenever I read about the Target breach…

    1) Why was Point-to-Point encryption not deployed here? If the sensitive data was encrypted on the way out of the PED then it wouldn’t really matter what was running on the POS. It’s significantly harder to install malware on a proprietary hardened device.

    P2PE manages to get one of the riskiest environments out of scope from a PCI DSS perspective, and limits your focus to the more easily secured centralised payments environment. I’m not saying P2PE is easy to deploy, but it beats trying to figure this mess out.

    2) EMV does not solve PCI DSS problems. There seems to be a popular misconception in North America that EMV is some sort of security panacea. Sadly not – there’s still track 2 equivalent data and PANs floating around. Even if Target had deployed chip (as they are now saying will be accelerated) the same malware would still have hunted-down the card numbers.

    1. Kurzleg

      @ Gavin Phillips: “P2PE manages to get one of the riskiest environments out of scope from a PCI DSS perspective, and limits your focus to the more easily secured centralised payments environment. I’m not saying P2PE is easy to deploy, but it beats trying to figure this mess out.”

      Is the reason P2PE isn’t employed because it would sacrifice processing speed? Or is it simply a matter of the cost of encryption?

      1. Gavin Phillips

        Generally processing speed isn’t a problem. Encrypting a PAN or track 2 is pretty speedy – especially on the decent Verifone hardware target are running.

        There was a general tardiness a few years ago by the PCI Council to provide guidance on P2PE, and hence a lot of retailers held-off on the investment. Obviously they’d need to invest in HSMs for their payment host which aren’t cheap, but again in the context of the costs associated with the breach…

        The other reason Target might not have implemented it is because they were actually doing some local processing on the POS which needs the card number or track 2 in full. Typically you see this at retailers who have business logic on the POS which needs the full PAN (maybe gift or loyalty or stored value activations) and isn’t handled by their central payments environment.

  12. BenT

    Excellent information–thanks Brian.

    Reading through, I keep hearing Eric Cole and his recent mantra: Watch your top 10 IPs that you connect to at the egress points; the top 10 IPs by traffic amount; the top 10 that are running encryption; the top 10 for length of time connected. You can tell a lot from that. Seeing 12 GB moving out to a foreign country’s IP over a few days probably means something.

    The point is that IDS, whitelisting, logging, AV etc doesn’t always work to stop things. So watch your firewall connections.

    1. SJ

      Do you happen to know if there are any commodity (free) tools for *nix to send a report (with some glue from Bash) on re: this to the sysadmin and a howto? I never thought about it that way. It’s a very solid point. If all these cards were going to 1 IP you might notice it if you ran the report:

      1. Daily
      2. Weekly
      3. Monthly.

      The problem is you might not notice it until #3. 1 month is too late. They were probably encrypting and compressing it. I would. And they could proxy it to 1000 different SOCKS proxies or tunnel it over DNS. No shortage of those on the web.

  13. Joseph Espinosa

    This was such a great read. Reminds me of the days when people were all bad ass about firewall this and firewall that and then you installed BlackOriface/system32 from an InstallShield SouthPark game and the fun started.

    No one has mentioned drivers. I remember making wrappers around the file system and/or the network driver you could pull information.

    I too agree, no one would have caught this for a very long time until someone got greedy. A good thief never bring any attention to weaknesses in security.

  14. Mike Schumann

    My wife and I own 3 high end furniture stores. We use simple Verifone Credit Card Terminals which are supplied by our acquiring bank.

    A couple of years ago, we were forced by our bank to go thru an annual PCI compliance process. What a nightmare. The SAQ forms that you have to fill out are idiotic for a typical small business environment. What mom & pop Chinese takeout restaurant is going to have security plan or any of the other gibberish that is in these specs. Just figuring out how to fill out the forms (and which form is required) is way beyond the capabilities of the vast majority of small merchants.

    We used to have our terminals connected to our store LANs so that we didn’t have to dedicate a phone line for our credit card machines. The PCI Compliance specs have made that impossible for a small business (you’re not allowed to plug your terminal into a LAN that has a WI-FI router or other PCs connected to it). So we are back to using dial up.

    As a small business, I should be able to get a credit card terminal that is properly designed, with the necessary end to end encryption, so I can plug it into my LAN and ignore it. If my computers get infected, that’s my problem, but the Verifone terminal supplied by my bank should be immune.

    This isn’t rocket science. Unfortunately, Visa, MC, and the banks don’t seem to be interested in stepping up to the plate and taking responsibility for the end to end security of their own equipment. It’s cheaper to hire some attorneys to draft some absurd specs so they have legal grounds to sue the merchant if some hackers take advantage of the vulnerabilities that exist in the current credit card environment.

    1. voksalna

      Mike, I do not know you but I appreciate your comments and situation. I wonder:

      “(you’re not allowed to plug your terminal into a LAN that has a WI-FI router or other PCs connected to it). So we are back to using dial up.”

      Have you considered a small data cap 3G USB solution (assuming the computer you use for POS is not connected to the internet, which it should not be any way)? This would satisfy both issues, I believe, though I would need to re-check on the PCI compliance aspect. It seems a far better solution than dialup, and the overhead is small enough that it may be the same price if not cheaper.

  15. voksalna

    Do you know what would stop a lot of this sh*t? To stop having ‘white hat companies’ explaining the intricate detailings of how this sh*t works. Speaking as somebody who does know the intimate detailings of how this sh*t works, before all of these companies decided to talk about it, there was little or no ‘Malware as a Service” for it available until talking about this sh*t became something popular for “white hats” to do.

    As far as I am concerned, articles like this (more so companies like this) create the problem so they can make money ‘solving’ the problem they create. Do you think most people have the technical ability or capability to get the information if it were not presented in an ‘accessible’ way with pretty pictures in nice pdf files and htmlised code reverse engineering and analysis? And for people who do not code it, you are telling them what to look for, what to ask for — you are providing the marketing for the things you are supposedly ‘against’ so that any person can just go find it. Dumb dumb dumb!

    I am not just talking about the POS issue. I am talking about all of these issues. It is the difference between a couple of dozen people being able to do something and a couple of hundreds or thousands or more (not to mention all of the people who will ‘want’ to do it because you make it more ‘cool’).

    This is not even a comment about Brian’s blog so much as it is a comment on the fact that people are either willfully ignorant of the fact they are causing this problem or their curiosity just makes it too much fun not to present the gift of allowing any person and their sister to re-create things. Everybody wants to have money and advance their careers. Yes, this is interesting. But is it helpful? It is counterproductive at best and ignorantly malignant somewhere in the middle.

    Fine if it is about the money but I do not know if it is so good to put yourself as so far above the people who make money who code these programs that get sold to people who steal the money when these programs do little more than (often poorly) using the methods companies like this and dozens others now compete to provide information on with more and more detail; it is just a form of extortion; a racket.

    Forgive the tone of this comment, please. I think this is probably my last comment for a little while, and I have been wanting to say some of these things for a long time. I am sure many people will be celebrating me not commenting for a week or two. 😉

    1. JimV

      Knowledge is a form of power, which can be abused. While “no knowledge” may equate to “less power” for those with zero curiosity or low intellect in the short-run, it simply won’t be a lasting effect as there are far too many with nonzero curiosity and much higher intellect who will pursue disruptive or criminal intentions — better to change their motivation or risk-level associated with the methods and opportunity, rather than suppression of knowledge associated with the means of their attacks. I think it’s even far better to provide general awareness and some measure of knowledge among the prospective targets about how such intrusions can occur in order to educate the public toward more perceptive and risk-averse behaviors to reduce the potential for any success in the many miscreants’ efforts, which Brian and most ‘whitehat’ individuals and firms take some care in presenting the information. Those with less integrity who simply wish to get a reward (monetized or just peer kudos) have far less integrity and in some respects could just be termed a ‘grayhat’ who rides the malware fence.

  16. Clark

    Target uses RealPOS terminals from NCR at least at their stores in Chicago.

    Has anyone ever heard of POS malware on Androids seems like every taxi driver uses Sqare to swipe your credit card on their phone as do many retails stores now. I would imagine it is just a matter of time before someone comes up with a way to capture all of those credit cards numbers with a similar method.

  17. Greg Scott

    Brian, great insights. But – why do Paul and Tom assume the malware came into those POS systems as rogue updates? If they’re really Windows PCs with card processing apps and peripheral devices attached, there are any number of ways to introduce rogue files onto those systems. If a bad guy compromised a server the POS systems trust, that server could send out a bunch of probes, find the POS terminals, and then worm its payload onto them. Especially if the POS terminals use well-known, factory default passwords.

    For Mike Schumann – there is an easy way around your restriction of not putting your POS system in your LAN. Just set up another LAN for your POS systems. Isolate them from your “people ” LAN at the firewall and use firewall rules to regulate what’s allowed in and out. That topology will pass your PCI audit and won’t cost a fortune. Topology counts and I’ll bet when the dust settles and we finally get some real information from the Target breach – instead of educated speculation – we’ll find that Target did not properly isolate its POS systems.

    For Voxsalna – I feel your frustration. But respectfully, hiding security information won’t be effective. You’re right, fewer people will know how to breach these systems, but fewer people will also know how to battle against it. The best way to raise the odds of safety is just the opposite. Use an open source development and maintenance model, with everything peer reviewed and tested by the appropriate smart people.

    And one more thought – If you isolate those POS terminals at the firewall, then you can easily monitor them. They should not have any interaction with the public Internet. So flag such traffic at the firewall. Similarly for the server(s) those POS terminals trust – those servers should only interact with a few well-known systems. Monitor it at the firewall and flag the unusual traffic.

    1. Mike Schumann

      Most small business people are barely computer literate. Getting a DSL line with a WI-FI router supplied by their phone company is the typical configuration.

      The credit card companies and banks need to be realistic and come up with 100% secure terminals that can be plugged into this type of LAN. It’s totally unrealistic to expect a small merchant to be able to secure this type of network, particularly given the track record of large companies with huge IT support staffs.

      1. Greg Scott

        Mike, you and I agree on the state of literacy among small business owners. A few years ago a friend called me and asked if I would be willing to visit a cigar store to look into a virus infected PC. I looked it over and sure enough, it was a mess. I also found they provided free Wi-Fi for their customers to relax in the store, and the Wi-Fi they provided was on the same LAN as their office PC. Of course they ignored all my warnings when I tried to explain how exposing their office PC to everything else that came into the store was dangerous. Just more money down a computer rat hole in their eyes.

        The point of my quick story is, that store owner’s attitude is typical of the small business customers I meet. IT is an expense and not an asset. And what a shame. If you don’t want to invest the time to educate yourself, you really really really should engage with an IT pro like me to evaluate what you have and how to make it better. Who knows – a good IT pro might even make you some money by coming up with some ideas you haven’t thought about.

        In the world of 2014, we are all interconnected and there is no such thing as 100% security. Expecting somebody to provide you with a 100 percent secure terminal will never happen. Just like you are responsible for locking the physical doors of your stores at night, you (not somebody else) is also responsible for securing your own business on the Internet.

        – Greg

    2. voksalna

      Greg,

      Respectfully, there is a difference between saying something is vulnerable and providing an exploit. There is a difference between telling people how to defend and telling everybody that will pay (or not pay now) how to exploit. And there is a difference between romantisising these sorts of things, like is done at BlackHat and DefCon and other conferences. There is a reason Barnaby Jack’s talk on ‘jackpotting ATMs’ was one of the most popular conference talks ever (both during the conference and now).

      Should the information be available to the people who make things? Of course. Should these companies provide the blueprints to duplicate somebody exploiting POS systems? There is a big difference. There is a “leap” necessary between reading a thing and understanding a thing and exploiting a thing, with each level in between.

      90% of browser exploit packs are made with wrapped metasploit payloads. This is the same sort of problem. Weaponised local linux exploits? Do you really think 99% of “hackers” out there would know how to, for instance, manipulate offsets to exploit a kernel-level vulnerability (or even manage to exploit a race condition) without ready-made exploits available to download? There is a reason “malware as a service” is so successful: It is because not many people have the necessary skills to make this malware “well”.

      Putting POS exploitation information online and telling people the names of it and showing it in action is sort of like demonstrating for all of the world how to better enrich something like plutonium. Only it’s cheap, if not free, and far easier to “create” because you need nothing to make it but a falling-apart computer and somebody with a brain.

    3. voksalna

      That was not meant to be an attack on you. In general I think a lot of your comments are sane; I am just not sure if you truly are paying attention to the cause and effect. Not all things are the same. There is a connecting piece here that is what I have a problem with, especially with things that may seem complicated but really are not as complicated as they may seem if you do not see things demonstrated (for instance raw packet sniffing). Ultimately people will not stop sharing this sort of information and my comments about this are pointless; I know this. My issue is with the fact that nobody else really seems to be willing to confront the hypocrisy (at least publicly) or take a stand against it — because that is not good press; if anything, it is no press at all.

      So as I said, this is not directed to you (and I thank you for addressing me).

      To comment on something else you said, that does tie in with this: Frankly the POS system does not need to be as “secure” if the network is designed correctly in the first place. The rules for firewalls to maintain this are so simple that I find there to be no excuse for people wanting to confront the needle in this haystack because it is more ‘interesting’ when nobody will deal with the haystack itself (this is probably a bad metaphor): proper network design and separation.

    4. voksalna

      Lastly (sorry, one more thing):

      ” Use an open source development and maintenance model, with everything peer reviewed and tested by the appropriate smart people. ”

      Except the problem really is not with the POS systems: It is with the people who architected, designed, managed, and ran the systems and networks, and that sort of thing is by very definition variable from place to place; this model does not at all apply. Which is one reason I am all for open source software, but do not believe it is a solution (and one reason why I am anti-disclosure of EXPLOITS even though I see value in disclosing a vulnerability).

  18. Greg Scott

    Voksalna, healthy debate is a good thing and from where I sit, you and I are OK. Nothing you’ve said is an attack and we’re sticking to the issue at hand. Vigorous public discourse at its best – Brian, thanks for the forum.

    And we probably agree on most of this. And as you know, the general debate around open vs. closed in security is probably as old as the IT security field.

    I need to answer this:

    > Do you really think 99% of “hackers” out there would know
    > how to, for instance, manipulate offsets to exploit a kernel-level
    > vulnerability (or even manage to exploit a race condition)
    > without ready-made exploits available to download?

    Of course not. And non-tech business owners such as Mike are probably overwhelmed just trying to read the sentence. I can see the eye-rolls already and Mike poking his wife – look, more tech gobbledygook.

    But here’s the thing. And Mike, this is for you too – the smart hacker who came up with that exploit probably didn’t come up with it on his own. He (probably not she) mostly likely built on the work from somebody else, who built on still somebody else’s work. The bad guys have their own forums and their own websites where they publish this stuff and buy and sell exploits.

    The bad guys share information because it’s profitable for them to do so. And that means bad guys who are not technologically smart can exploit a treasure trove of information produced by smart bad guys.

    So if the bad guys are sharing information, it only stands to reason that the good guys should also share information. Especially since it’s already out there anyway.

    – Greg

    1. voksalna

      Ok, but the problem here is you are not separating the VALUE of the information being shared. In general it is precisely things that ‘white hats’ exposed that IS shared. It is not all that common for a ‘black hat’ or a money-oriented hacker (I do not consider these same thing) to freely share this sort of data because it would impact their ability to take advantage of it in a positive way; there is too much competition.

      As a result it is the ‘white hat’ research that is grossly spread publicly and extrapolated on heavily by these private forums (and the public ones — and every other one).

      Exploits cost money. If you don’t easily agree with this read back on just Brian’s blog alone. “Zero Day” has value to a ‘black hat’ or a money-oriented hacker (and now, unfortunately, to many so-called “white hat” firms and governments who buy these exploits for even higher prices but this is another topic).

      DETECTING a “Zero Day” should be what matters to the typical IT worker. PREVENTING it should be enforced at a development and architectural level and made virtually transparent to IT (stuff like hardened PHP, grsec, PAX, even RBAC and such: these prevent the bulk of issues so the implementation has more “room for the wiggling”; yes more technical jibber). FIXING it should be what matters most to the vendor (or open source community, or internal computing department if it is proprietary or an internally developed system or a private network issue — such as poor architecture), even if the problem has been solved largely at the preventative level (because things can always be re-introduced).

      Systems, software, and networking, especially when it comes to ‘point of sale’ should be done with the lowest possible intelligence-level of user in mind, which is not meant as an insult to any technical worker (or any retail worker). But it is possible to make things much much harder, just like how a safe is rated not to never be breakable but for how long it can stand an attack. This buys time. Yet more than this, because unlike with a safe, systems do not generally respond to blunt force or applied physical methods in many ways: An internal threat is one possible way. So you design to prevent internal threats to be much harder.

      1. Greg Scott

        Well….yeah. Let’s say Bob discovers an exploit in Alice’s software. Bob is a responsible good guy, so he quickly notifies Alice privately and Alice puts together an update and sends it her customers, along with a public acknowledgement to Bob for his work. She might even send Bob a nice polo shirt with her company logo on the collar. That’s how things **should** work.

        But maybe Bob wants publicity. Or maybe Bob is mad at Alice. So instead of contacting Alice privately, maybe he publishes what he finds and now the world learns about it before Alice has a chance to respond. This is bad – in this case, Bob is a bad guy and should be treated as such.

        Or – maybe Bob contacts Alice privately but Alice does nothing. several days go by and no patch, no acknowledgement from Alice, just silence. What does Bob do? He wrestles with his conscious. He knows Alice’s software is popular – maybe millions of people use it. He knows if he found the exploit, the odds are high that somebody else also found it.

        What is the right thing for Bob to do? You make the call.

        – Greg

  19. Greg Scott

    I almost forgot about this story. In this case, I was one of Alice’s customers. Sort of. Except that “Alice” was the Norm Coleman for Senate campaign, not a software company. “Bob” was another IT consultant who uncovered the problem. And Wikileaks published it. This was before Wikileaks went too far and published US Government secrets. The whole story in the link below.

    http://www.infrasupport.com/how-a-gross-it-security-lapse-hurt-a-us-senate-campaign/

    The story is a great test case for proper and improper disclosure and the devastating results of bad security practices. Voksalna, you and I are probably on the same page for most of this.

    – Greg

    1. voksalna

      Apparently my long reply did not post/was moderated. O well.

      1. voksalna

        And it apparently keeps getting ‘moderated’ if I retry to post it. Why?

    2. voksalna

      My original reply:

      voksalna
      February 6, 2014 at 1:57 pm …awaiting moderation

      Publishing a vulnerability, publishing an exploit, and publishing the results of the vulnerability and publishing the results of the exploit (you see a difference between results of a vulnerability and results of an exploit I hope?) are different things.

      There is a term ‘Schadenfreude’ in German. And people are by nature restless and relentlessly sadistic whether they will acknowledge this or not.

      Do you acknowledge there is a difference between publishing the vulnerability, publishing an exploit, publishing the results of an exploit, and sharing the results of an exploit? Right now I am thinking of the Brown/Anonymous case and the sharing and use of the credit card data (some would say encouragement to use it but I wish to avoid to cast a stone in any direction there).

      Let me ask you is Adria Richards more responsible because she has an audience? Or does she have the same responsibility as some random person who might have said something and nobody would have noticed? And what is the culpability level of Wikileaks compared to Adria Richards?

      You have made the comparison which I think is not really an appropriate one (in this specific case it is a proprietary issue specific to the web site, it sounds like, and you had the ability to email every person on that list if you could also get the list which it seems any person could get — is this morally responsible? If they broadcast it would that have been morally responsible?) so I will try to use your own comparison.

      I will not state my own beliefs on this until I have understood what your argument is. 🙂

      1. Greg Scott

        > Publishing a vulnerability, publishing an exploit, and
        > publishing the results of the vulnerability and publishing the
        > results of the exploit (you see a difference between results of a
        > vulnerability and results of an exploit I hope?) are different
        > things.

        If a bad guy exploits a vulnerability at expense of a legitimate organization and then publishes the result, he should be punished. This is wrong.

        If a somebody finds a vulnerability and publishes it day zero without first informing the vendor and giving the vendor a chance to respond, this is also wrong.

        I don’t know anything about the Brown/Anonymous case so I don’t have anything to offer on that one.

        Adria Richards is a grey area and from what I’ve seen subsequently in the news, she does more harm than good. She created a lot of trouble in San Francisco, I think last year. It was all over Google. In the Norm Coleman case I wrote about, she should have informed the Coleman campaign quietly before publishing the screen shot of what she found. She went after the publicity instead of fixing the problem.

        Wikileaks did a service by telling me I was a victim. I would have never found out otherwise. A few years later, Wikileaks did a major disservice when it published Bradley Manning’s secrets.

        Here is the link to what Voksalna and I are talking about:
        http://www.infrasupport.com/how-a-gross-it-security-lapse-hurt-a-us-senate-campaign/

        – Greg

        1. voksalna

          Do you consider a configuration f*-up a vulnerability the same way you consider source code vulnerabilities vulnerabilities?

          Adria Richards wants too much attention. A lot of these people do. And it is the culture, especially in ‘white hat circles’ to reward that desire for attention with the attention people desire, as well. In this day and time attention gets page hits gets money and career advancement, and if it is not the way you want it is certainly one way.

          The way to stop this behaviour is to discourage it from being rewarded. That means ignoring it and not providing it press, not ‘drumming things up’ in media or glamourising things.

          That especially includes resisting urges to make idiotic stuff that stupid bloggers say memes, or providing people like LulzSec with media coverage in every possible place (which guess what the government also knows and uses to undermine legitimate protestations not just spoiled kids with too much of the free time on their hands and no idea how to make voices heard because they do not really know what they want to be saying).

          These things, just like the publishing of ‘0-day’ and the publishing of names and locations and methods of obtaining exploitation code (for free or for sale) all encourages and legitimises other people doing it. It sets an example. And I am talking about one single standard — you can not say a “legitimate researcher” can give talks about how to write bootkits to anybody who cares to watch or a “legitimate researcher” can talk about decimalisation tables and PIN cracking or whatever whatever on the one hand, and these very people also glamourised LulzSec and anonymous despite portraying themselves as ‘white hats’ because they want to be part of ‘current events’ — or make tools like Metasploit which punish people just like this Chinese food vendor for doing nothing particularly wrong, just not doing everything particularly right — but on the other hand condemn people for selling Java exploits or browser exploit packs or anything else that can be used for either purpose. Pick a standard. And have some ethics instead of using stuff like big breaches to push some sort of business method of making money because it doesn’t agree with your ethics.

          Tools do not have ethics. The people who use them do. Equally, the people who should be having more ethics aren’t, because they’re all jumping all over themselfs to be the first to ‘break news’ of something or be the first to go ‘me me me’ to make money.

          What angers me about this current idea of ‘the Internet’ is that you have to be ‘me me me’ now because everybody shouts so loud that everybody else just shouts louder but the debate itself doesn’t happen.

          With a vulnerability few people can attack. With an exploit anybody can attack. I see no reason for anybody, ever, ever, to publish an exploit, and shaming does not work for ‘reporting vulnerabilities’ — companies will fix something or they will not. In open source, somebody else will provide a fix (Brad Spengler (‘spender’) is a good example of solid practice; his work on grsec is mostly thankless but it is some of the most important SECURITY WHITE-HAT work there is.

          I can not remember who said this but a famous man once said that any body can complain — complaining is pointless. Solutions are harder. And they do not come in form of massive tarballs of exploits. Massive tarballs of exploits with pointy clicky interfaces provide journalists with stories. And it was probably some public tool that was used inside of many of these windows networks that have been breached from a POS standpoint — and it was probably some wrapped or rewritten version of a clientside from metasploit that was used as the initial vector.

          If one person gets plague, you do not say, well this is bad — better to make sure to give the whole village the plague because that way we know after that nobody else might get plague… and let us make sure we give every person who is in village the means to also spread the plague to any person that might pass through so that they too can spread it to their villages so we can wipe out the plague-vulnerable peoples.

          I ramble but I have not slept.

          I won’t comment on WikiLeaks because you are adding some weird political thing to all of this that I find strange and disagreeable to me (even your examples). I am curious though because you brought it up — do you feel this same way about the NSA documents that were published showing massive surveillance on a global scale? Is your objection because you felt data was not relevant, or is your objection because you believe a secret is a secret no matter what unless it is a vulnerability or exploit in which case it should be published?

        2. voksalna

          Moderated again. Am wondering if it is a length thing. But will add this last thing (which probably will post): Internet of today discourages humbleness and dignity and quiet resolution. When everybody shouts, nobody will listen. I am anti-censorship but it presents a problem — and a problem that is known in game theory (which is something I know a good amount about): when you know there is a limited ‘pot’ of available resources (attention, money, whatever), and you see more and more people you never really acknowledged existed because you didn’t have to hear them or see them or fight to be heard above them — that creates a resource grab.

          People want to be first to break a story, first to report a vulnerability, first to code an exploit, first to expose anything without confirmation, first to humiliate without proof, first to throw a stone without evidence, first to point out a mote in somebody else’s eye but not acknowledging biblical plank that is own because they are blind to it.

          I think a lot of people do not have a problem with a lot of these very black but spread by ‘white hat’ things because they imagine some day they will be the receiver of this luck or attention — and they want that opportunity to continue to exist so they can benefit from it — financially or from a career stand point — or if not them, some person they know which will benefit them directly or indirectly later. Then when a company googles they will say “I want this company because they have this person working for them” to do their pentesting. To me it is hypocrisy.

        3. voksalna

          Lastly (for now — hopefully my first post will post; it seems to take several days to have moderated things show up on here):

          You said we agree on many things in earlier reply. I believe we agree on a lot of technical matters when it comes to securing systems, but we seem to have a number of philosophical differences that I do not think we will bridge. Despite this I am curious what your logical/rational argument is in favour of disclosure of vulnerabilities, and in favour of disclosure of exploits. I do not count ‘to pressure vendor to release a patch’ as an actual argument though I understand that it is a common industry practice.

          These things are opinions, though, on both of our parts, and whatever the right opinion is (or wrong one is) I am sure we will all eventually find out. Probably not even too long from now at the rate we are going in the world right now. The only truly ‘secure system’ exists in a truly draconian surveillance state, which is to me the biggest problem. The ‘white hats’ are the ones pushing this thing that is a lot like an ‘arms race’ — not because it IS like an ‘arms race’ but because the profit model is far better for them to *make it* an ‘arms race’.

          1. Greg Scott

            Hi Voksalna –

            We’ve talked about much of this already. It seems the disagreement you and I have is over proper disclosure of exploits. Your position is, there is no such thing as proper disclosure. A disclosure probably contributed to the Target breach.

            My position is, there is a time and circumstance for proper disclosure and there are times when you have a responsibility to disclose.

            Do I have the issue framed correctly?

            The example we’re talking about is the case with the Norm Coleman for Senate campaign, detailed here:

            http://www.infrasupport.com/how-a-gross-it-security-lapse-hurt-a-us-senate-campaign/

            Here are some responses to your questions.

            Yes, from a security point of view, a configuration vulnerability is on the same level as a coding vulnerability.

            We agree that Adria Richards seems to enjoy attention. She did good detective work finding a blatant configuration vulnerability in the original Coleman website. Publishing the screenshot and exposing the vulnerability on her own website without notifying the campaign and giving the campaign some time to respond was wrong.

            Here is where we disagree. You think Richards should have informed the Coleman campaign privately and then dropped it. I also think she should have informed the Coleman campaign first and given them a few days to respond. After a few days, she had a duty to report the problem publicly.

            The sensitive information in that website, including credit card numbers or many donors, was exposed to the public. If the people in the Coleman campaign had their way, nobody would have ever found out and the donors whose credit cards were compromised would never know they were at risk of identity theft.

            Credit card numbers for some donors were, in fact stolen. You can make a good case they were likely stolen because Richards published the vulnerability and bad guys exploited it. They may have also been stolen before Richards found the problem. We will never know for sure. It’s a shame she **improperly** disclosed the problem. She failed in her duty to do it right.

            So let’s say you come across a website and find a blatant security issue. It exposes thousands of people to credit card fraud. You notify the organization, the organization sits on the information and does nothing. Now what? Do you drop it and leave all those people at risk or do you do something about it?

            You say drop it because disclosing the vulnerability puts everyone at more risk. I disagree. I say you **must** disclose the vulnerability to level the playing field, but only after giving the organization a reasonable amount of time to address it first.

            The same reasoning holds for both coding vulnerabilities and configuration vulnerabilities.

            As for presenters at conferences giving detailed talks about how to build rootkits and other such malware, my instinct makes me think this is OK. To a point. If I were giving such a presentation, I would feel a duty to also show the audience how to defend against it. Others will make their own decisions. These are professional conferences for people who do security for a living. This is the place to talk about this kind of stuff.

            There was a case a few years ago where Sony inserted rootkits on music CDs they shipped. Buy a music CD from Sony, put it in your computer to play it, and now your computer is compromised. Mark Russinovich found and disclosed the problem and did a service to the world. Had he not disclosed the problem when he found it, by now Sony Corporation would have its hooks into millions of computers around the world.

            There are times when disclosure is absolutely the right thing to do. There are other times when disclosure is wrong.

            You asked about Wikileaks and the recent NSA disclosures. I’ve already said this: When Wikileaks published the details around the Coleman campaign, they did a service for me personally. The Coleman knuckleheads exposed my personal information and Wikileaks told me about it.

            Later, when Wikileaks disclosed US Government secrets, Wikileaks committed a crime. The same is true for Ed Snowden and the NSA disclosures. Snowden also committed a crime.

            But this is all far afield of Brian’s original story in this blog post and we should probably continue this side discussion elsewhere if we want to keep talking about it.

            Go to my website – see above – and go to the Contact Us form and we can trade emails if you want.

            – Greg

          2. BrianKrebs Post author

            Voksalna, if your comments take a while to appear sometimes, it has more to do with the frequency of your comments than anything else. For whatever reason, yours get flagged far more than most, and I think it’s because you are often commenting on multiple stories at once and in rapid succession. I just checked and you have made more than 650 comments on this blog. I don’t think anyone else comes close to that number, so that’s what I think is going on.

            1. voksalna

              That many in only a few years? That is horrible. And I feel violated by my posting habits being so publicly exposed ]:).

              This said this is possible. I only visit this site once or maybe twice a day to reply to all I have a need or desire to reply to at the same time — other times not at all for weeks at a time. Depends on the story. Thank you for letting me know this though. I had thought it was per story not whole site. Or maybe word-count, as sometimes short things will post immediately after a long thing says ‘moderating’.

  20. John

    We haven’t tuned many signatures off on our IDS and when they first went in they didn’t light up seeing credit card data. Unless you’ve got a DLP solution or a custom signature at your egress points the run of the mill IDS provides little value in catching this data moving in your network.

  21. Greg Scott

    You might not need anything as sophisticated as an IDS watching traffic. You could do it with some Linux iptables rules and logging and dropping. Or maybe rejecting if you want to be polite. All the stuff in the POS infrastructure should only be interacting with a few systems outside the company. You know what those are, so you reject everything else on the Internet NIC that comes from the POS network and log it. And of course, you have more liberal rules for traffic coming from your “people” network.

    And then internally, if you properly isolate your POS network, those POS terminals and server(s) should only interact with each other and maybe a few other internal systems and only in certain ways. This should be easy to model using some iptables rules and then you log and reject the anomalies.

    I’ll bet a doughnut that if the folks at Target and the other retailers had such a simple system in place, they would have nailed the bad guys and saved themselves potentially $billions in liabilities and $millions in cash out the door.

    Even small furniture stores could afford a system like this. FWIW, I build these systems using Fedora Linux and I have dozens installed. As Jeffrey, Ralph, and others pointed out, excellent security does not need to cost a fortune. It only requires somebody who wants it bad enough to do something about it.

    http://www.infrasupport.com

    – Greg

    1. Mike Schumann

      Greg:

      Let’s get realistic. A small furniture store or Chinese takeout place typically has an owner who barely knows enough to get a DSL line installed so that he can get a PC and/or an iPAD connected to the internet to get his e-mail and be able to surf the net.

      These merchants need a turn-key credit card terminal solution that is supplied by their banks that they can plug in and forget. Hiring IT support is not in the cards. A. These merchants don’t have a budget for that. B. Even if they have the money, many of these merchants typically don’t have enough technical background to be able to intelligently determine who is a competent vendor vs who is a scam artist, or even a crook.

      1. Greg Scott

        Hi Mike –

        Well, OK, let’s really be realistic then. The state of IT security and the industry in general is what it is. Wishing it were different won’t make it different. There are smart bad guys out there and they want to steal your customers’ credit cards. Their tactics change over time and they’re doing constant R&D to find and exploit weaknesses. Set it and forget it on your side isn’t good enough because the world in which your technology lives is constantly changing.

        > Even if they have the money, many of these merchants typically
        > don’t have enough technical background to be able to
        > intelligently determine who is a competent vendor vs who is a
        > scam artist, or even a crook.

        You and I both know this is not true. As a business owner, you perform due diligence on any number of areas every day in which you’re not expert. So do I. So do all business owners. You use your gut instincts, you ask the same questions to multiple vendors, you shop around, you make good judgments, and you hold people and vendors accountable for what they promise.

        The real meaning of that sentence is, your IT system is not important enough to invest in it. And that’s your call. It’s your business. But if this stuff is not important to you, why should it be important to any of your suppliers?

        Those PCI self assessment questionnaires were not put together randomly. The PCI folks used the experiences of large and small businesses like yours and the best experts they could find to come up with those checklists. I’ve been through them a few times. They are thorough. As much of a pain as they are to go through, they’re in place for your protection.

        Instead of griping about them, use them as a learning experience because you have to deal with CC transactions every day if you want to do business with the public. Several hundred experts contributed to those SAQ forms to ***give*** you the information you need to keep your customers’ personal information secure. Why not take the time to go through them and ask the right questions and educate yourself?

        And think about this – if you have a breach and you didn’t follow the PCI best practices – and you didn’t even do your homework to find out what they are – who assumes the liability to make all those customers whole again? It won’t be the Russian or Chinese bad guys and it won’t be your bank. It will be you. Keep watching the news to see how much this whole thing will cost Target shareholders.

        – Greg

  22. Mike Schumann

    Greg:

    You are out of touch with the real world. Yes there are small business who value IT and can deal with the PCI Compliance complexities.

    Probably 50% of small merchants can’t. Just walk down Main Street in any small town in the US. For many of these merchants, it’s a stretch to learn how to run Quickbooks. Securing a LAN is a concept that doesn’t even register.

    Beyond that, how are these merchants even suppose to answer the PCI SAQ questionare: What mom & pop retailer has information security policies???

    Try taking a PCI SAQ B (the “simple” version that you get to use with a dial up terminal) to your average small town retailer and see if they can fill it out. Their eyes will glaze over…..

  23. Greg Scott

    > You are out of touch with the real world.

    This is funny. Or maybe it’s my mood right now after spending 2 hours on a support phone call on behalf of a small business customer who depends on me for sound IT practices. I live on the same Main Street as every other small business owner in America and I’m about to head to another small business site right now. I probably have a few dozen small business customers with similar circumstances as yours. Believe me when I tell you, I live in the same real world as you.

    For what it’s worth, I have helped customers fill out those PCI SAQ B forms. You’re right – asking a naïve customer to fill one out on her own is nuts. The terminology alone is daunting. It’s kind of like asking me to help somebody pick out furniture.

    Yup, it will cost you a few $hundred to engage somebody like me and do it right. And y0u’ll spend more than that to set up your network properly. I think it’s a good investment. I think you’ll get an ROI from it.

    As a business owner, you’re earned the right to make your own judgment calls. You can choose to hide behind illiteracy if you want. But it’s an illusion. What you don’t know can hurt you. That’s the real reality of the real world.

    – Greg

    1. voksalna

      Greg, I do not always agree with everything you have said, but I agree 100% with you here: It is the business owner’s responsibility to know what the business owner knows they cannot understand. A mild example is somebody doing plumbing when they do not know how to do plumbing; bad things happen; poor judgment. A better example is somebody doing surgery without medical experience; worse things probably WILL happen. It is not the business owner’s responsibility to do everything themselfs. It is the business owner’s responsibility to know they cannot (and most can not or will not and we both know this) and act maturely.

      They also have the option to not accept credit cards (and many small businesses do not) if it is too daunting (and a lot of that IS an ROI issue; if you are so small that you can not spend a few hundred USD (and hand your responsibility to them; yes you would then be responsible as the technician in my opinion at this point if they did not follow the guidelines you set) then you probably should not be accepting credit cards at all.

      1. SJ

        The problem is the Chinese takeout guy shouldn’t really need to understand this. From their perspective they should be able to plug it into the LAN and secure itself. Since it’s totally possible, it’s reasonable to expect that. It’s not like POTs was ever secure, and we totally dial our CCs every day via DTMF and I guess we hope that there’s not a DTMF scraper on a non-digital line from way back when. Fewer and fewer but I’m sure it’s around.

        Be realistic. If the box can secure it, it’s the best place to do it. The takeout guy shouldn’t have to think about it if it can be black boxed. Software 101 no? A CC processing box is a total commodity. Why make anyone think about securing his network for such a mundane task?

        1. Greg Scott

          That Chinese takeout guy needs to know lots of things. He needs to know about business licenses, sanitation, taxes, cash flow, depreciation, plenty of things unrelated to Chinese food to operate day to day. He doesn’t need to be expert in all these areas, but he does need to be aware of them and he needs to occasionally spend money and engage experts.

          The suggestion that IT is somehow on a different level and can remain a mystery is unrealistic. The Internet is fundamental to business in today’s world and that guy needs to be aware of the opportunities and threats across the Internet. And that means investing in a small, secure network and doing good due diligence around the best supplier. Just like with all other aspects of his business.

          How does a Chinese Food takeout guy evaluate, say, an accountant? He’s not an accounting expert so how does he know if an accountant is giving him BS or useful service? Why not apply the same due diligence to IT support instead of pretending it’s some kind of magic and wishing it was unnecessary?

          And of course the vendors who supply POS equipment should build them with the right safeguards. Maybe someday they will. But as we found out with Target, that day is not today.

          Somebody in this thread said the right mindset is “Trust but verify”. This is a good concept. If you trust that black box to handle credit card transactions, why not verify it at your boundary? And BTW, you don’t need periodic transaction reports – you need alerts when unusual traffic goes out and Linux based firewalls can easily do that with some iptables rules and logging. That’s how you find out right now if there’s a problem versus waiting for a summary report next day or next week or next month.

          If you’re that hypothetical Chinese food takeout guy, ignore IT security at your own peril.

          – Greg

          1. SJ

            See: “Since it’s totally possible, it’s reasonable to expect that.” I’m not saying anything different. You’re responding to a different question. I see no reason to make him understand intricacies of security when the device can be black boxed and made secure without understanding network security.

            1. Greg Scott

              > …when the device can be black boxed and made secure without
              > understanding network security.

              This is where we disagree. It is reckless and dangerous to assume any device can be made secure on its own. These things don’t live in a vacuum. You gotta protect it.

              Some how, some way, a Chinese food takeout guy somewhere will find himself in a world of hurt and on his own by blindly trusting one of these devices. And there will be no FBI, no Secret Service, no law enforcement help from anyone because he is not big enough for anyone from law enforcement to care.

              – Greg

  24. Mike Schumann

    Greg:

    The issue is not about whether a small business is making a smart decision or not. The reality is that many won’t, either because they are too cheap or don’t have the money to hire a professional or are ignorant about the threat.

    The bottom line, is that for a simple credit card terminal installation, it should not be the merchant’s responsibility to have to worry about the security of the equipment that is being supplied by the banks. The terminal should be bullet proof and unhackable on any LAN that is totally unprotected from outside attacks.

    That’s the only way this problem is going to be solved in the real world.

    1. voksalna

      See my reply to Greg as it also applies to you. And note that for the past two months in these articles I have been publicly saying this sort of thing exists but even WITH this sort of thing it is a business owner’s responsibility to implement it properly. Most security failures are things most normal people would not even think of and maybe even have almost nothing to do with the systems themselfs. Implementation will always be a big issue no matter how ‘secured’ a POS system is ‘out of the box’. The real world does not operate in this fashion.

  25. Greg Scott

    Hey Brian – thanks for letting this discussion go on in your blog. It’s great to get all these points of view and I hope readers enjoy the exchange.

    re: Mike –

    > The reality is that many won’t, either because they are too
    > cheap or don’t have the money to hire a professional or are
    > ignorant about the threat.

    It’s my job – the IT security practitioner – to educate you – the business owner – about the threat. It’s your job to listen and learn and act. If either of us fails to do our jobs, we all lose.

    > The bottom line, is that for a simple credit card terminal
    > installation, it should not be the merchant’s responsibility to
    > have to worry about the security of the equipment that is being
    > supplied by the banks. The terminal should be bullet proof and
    > unhackable on any LAN that is totally unprotected from
    > outside attacks.

    There is no such thing as an unhackable device. Such a device has never existed and will never exist. Wishing for it won’t make it so. Demanding it won’t make it so.

    Security is like an arms race. The bad guys come up with an exploit, the good guys come up with a defense. The good guys come up with the latest and greatest “impenetrable” system. The bad guys figure out a way to penetrate it. It’s been this way for 50+ years since the dawn of the computer age. It will be this way for the next 50+ years and beyond.

    Security is a game of probability and tradeoffs. You analyze and weigh the probability of an attack against the expense of defending against it. You harden your systems enough to make the potential risk for an attacker too much for the potential reward.

    Or you close your eyes and pretend security is somebody else’s problem. Kind of like when we were kids and somebody snuck a “kick me” sign on your back. Except the consequences in the grownup world are more serious.

    Think about this for a minute. In the real world. When you think about the smartest IT security team on this planet, who comes to mind? For most people, it’s the US National Security Agency. Security is in their name. It’s WHAT they do. And even they – allegedly the smartest security practitioners on the planet – can’t keep their own secrets inside their own walls. Note that I said allegedly. And if the NSA can’t keep its own secrets, you expect a bank POS terminal to keep your credit card secrets?

    Think about this. By definition, that “simple” bank-supplied POS terminal must deal with a group of credit card processors. How does your POS terminal “know” it’s really dealing with the real processors instead of somebody impersonating the processors? And how do the processors “know” they’re dealing with your real POS terminal and not somebody pretending to be you?

    You connect your POS terminal to the PSTN (public switched telephone network) and not the Internet, right? What happens if somebody intercepts one of those phone calls and launches a man in the middle attack against you and the credit card processor? Even if that POS terminal is perfectly engineered and bug free, it still depends on external factors over which it has no control.

    You claim illiteracy and therefore the luxury of throwing all those tough engineering challenges over the wall for somebody else to deal with. You say it’s not your problem, you just want to sell furniture and let the tech people play somewhere else.

    But like it or not, it is your problem. Bad guys want to steal from you and your customers and you have a sacred responsibility to protect yourself, your family, and your customers.

    > That’s the only way this problem is going to be solved in the
    > real world. [referring to the mythical, unhackable device.]

    Ya know, I want $1 million and a new car. I want an energy source that’s renewable, doesn’t cost anything, and doesn’t pollute. I want peace in the Middle East. So do you think if I close my eyes and wish hard enough, it will somehow happen? Your wish for an unhackable POS terminal is in that same category. I don’t know how to make the case any more clear.

  26. Greg Scott

    Throughout this discussion is an assumption that the computer systems business use are an expense to be minimized instead of an asset to be fully utilized, and that the people who build and support these systems are divorced from the real world on main street.

    And Voksalnan and I have spent enough time making the case that security is important and worth the investment. But security is the art of avoiding bad things. Instead of playing not to lose, what about playing to win?

    So here are a few suggestions for playing to win. Free advice is worth what you pay for it and I haven’t spent any time vetting any of these ideas. They’re just ideas. Useful for people who think the last good invention happened last week and now we’re done innovating.

    I’m just a dumb tech guy who plays with computers all day. I know nothing about furniture. If a furniture store were to put together an interesting video on how to care for my furniture, or maybe what to look for when shopping for quality furniture, I might be attracted to that furniture store. Especially if I’m tired of my plastic veneer particle board cabinets. Does making a good video cost a fortune these days? No. Millions of teenagers make videos with their cell phones every day. Some are pretty good.

    You have multiple stores. How are they connected? Are you using private and expensive telecom connections? Or (perish the thought) are your stores still connected by dialup? Have you considered cutting your costs significantly by using tunnels instead?

    What happens if Marcia wants to bring home a nice chair on display at one store, but it’s out of stock? Can the store sales rep look at a current inventory list to see if the item is available in another store? Or maybe on order from the manufacturer? And what’s the ETA? Do your sales reps have easy access to the real-time information they need to answer customer questions?

    Nancy is busy and doesn’t have time to drive across town to visit a store. But she wants a new dresser for her bedroom. What if she could visit your store over the Internet via live, interactive video? What if you could set up a video meeting with you, the manufacturer, and Nancy and come up with a custom dresser? Do you think there would be any profit margin in that kind of high-end business?

    What about tried and true stuff like taking orders over a website?

    Just a few ideas. I’ve burned enough disk space.

    http://www.infrasupport.com

    – Greg

    1. SJ

      Unfortunately, the vast majority of people don’t understand what techies do — who gets it, who doesn’t. What’s BS, what’s not. And it’s usually seen as a cost by (successful) pencil-pushers. The guys who are in competitive industries do it by bargaining with suppliers on commodities.

      It’s natural to extend that sort of treatment to service providers. It’s also unfortunate. So —

      1. I don’t think the Main St. guy just swiping cards manually should have to worry about this.

      2. But, for example, in the smaller eCommerce realm you will see a lot of businessmen who are ruthless with developers and get what they want until they get hacked. It’s not difficult to find smaller online businesses outsourcing their code to various 2nd and 3rd world countries, never doing an audit, giving SSH to Randomperson X. They don’t care about security.

      We’ve all been in that room with #2. PCI was supposed to solve it, but it doesn’t in my experience.

  27. Greg Scott

    Back to security.

    Voksalna and I may disagree on some of the details but we’re both giving you the same message. Security is important. Maybe vital to your business. Ignore it at your peril.

    Mike, you said half the merchants on Main Street are unable or unwilling to invest in the confidentiality, integrity, and availability of their own data. If so, and I tend to agree with you, then I want to buy goods and services from the other half. The ones who care about the portion of that data relating to me.

    And since I said there’s a security solution a small business can afford, what would that solution look like and how much would it cost?

    First, an assumption. No device is hackerproof. We’ve gone over that exhaustively already. You must assume bad guys want to get inside your devices and use your equipment and your network for their own evil purposes.

    That’s the nature of the threat. Now here are the pieces to deal with it.

    It starts at the firewall. You need a real firewall, not the junk bundled with DSL and cable modems, with provision for multiple LANs. A real firewall is just a router with multiple segments and some rules to regulate how each segment interacts with the other segments. Most credible DSL and cable modems can accommodate firewalls behind them if configured properly. I’ve built dozens for small businesses just like yours and I’ll teach you as much or as little as you want so you understand how it works.

    Your firewall will have at least one public, Internet facing segment. It might have more public segments if you want multiple Internet feeds from multiple providers so you always have a path out if one feed drops. Multiple Internet feeds is probably overkill for that Chinese takeout restaurant. Unless they depend on, say, a website to operate hour by hour.

    You may choose to have an HA firewall system at your boundary that can juggle multiple Internet feeds and do automated failover routing in case an Internet feed goes offline. This may also be overkill for a Chinese food takeout restaurant and a single furniture store. It may not be overkill for a multiple site furniture operation that depends on the HQ site always being available. Start small and scale as the business grows.

    It will have a “people” segment where you put your employee computers. This is where you put in the typical rules you see in most customer networks. You’ll want a credible antivirus solution on all your workstations in this segment. It can also become elaborate. You can put in web filtering appliances to regulate which websites your users visit, for example. If you choose to host your own email or web server(s), you can put in rules to accomodate those, and rules to accomodate spam filtering. Overkill for a Mom and Pop store, might make sense for larger businesses.

    If you’re a retailer, your firewall will also have a POS segment. If it’s a simple POS terminal, it will interact only with your credit card processors. Credit card processors all have IP Addresses, so you put in rules to allow anything in the POS network to interact only with those IP Addresses. You also have a rule that blocks anything between your “people” segment and POS segment.

    If your POS network is more sophisticated, maybe those POS systems need to interact with, say, a database server. That database server, in turn, needs to access servers in your “people” network. In this case, you carefully construct firewall rules to accomodate this traffic and log attempts at any other traffic. Overkill for that Chinese restaurant. But maybe necessary for a franchise of Chinese restaurants or a sophisticated retailer with, say, a loyalty program.

    Maybe you want to offer wifi as a convenience for your customers. It’s tricky to do this right because of the nature of wireless and because you don’t want your customer wifi to mingle with your employee wifi in your stores. Isolate the customer wifi from your employee wifi and all your other segments. The wifi segment is only a convenience for your customers to get to the Internet. Nothing crossses the border between the customer wifi into the “people” segment or the POS segment.

    And there you have it in a few short paragraphs. A topology that does a wonderful job of enabling your business, serving your customers, and keeping bad guys out. Total investment – a properly done firewall and either a few physical network switches or a smarter switch with VLAN capability. About $4k to start. Maybe a little less for a small operation, probably more for a larger operation. The antivirus subscriptions and other support subscriptions will also cost some op-EX each year.

    Mike, now that the solution is quantified, how do you feel about it?

    http://www.infrasupport.com

    – Greg

    1. SJ

      You can actually get a lot of that done on the cheap with DD-WRT =) There was that one REALLY embarrassing code injection vuln, though.

  28. Mike Schumann

    Greg:

    While you may think I’m a yahoo from the sticks, I was in the computer business since the mid ’60s before we started our furniture business 25+ years ago. Our IT infrastructure is actually pretty sophisticated, with a VPN network connecting our 3 locations. I am also pretty familiar with the whole credit card world, being one of the original class plaintiffs in the massive Visa / MC anti-trust litigation that has finally been settled after 8 years of litigation.

    Your attitude that no device can be designed to be unhackable is BS. The problem originates with a device’s hardware design. Any time data can be executed as code, you obviously have the potential for a bug in the firmware that can create a vulnerability.

    However, there is no reason that you can’t design devices that only execute code that is burned into ROMs where by the inherent design of the hardware it is impossible for a remote malicious user to change the function of the terminal. At that point the problem changes to a physical security issue; How do you prevent someone from impersonating a bank employee and fast talking a dumb clerk into replacing your terminal with a fake unit, etc……

    It may not be possible to prevent that type of attack, but there are ways to detect it very quickly before it does any serious damage. The key is to use two way encryption between the terminal and the bank’s systems with ID checks at both ends to make sure that no equipment is substituted or internet traffic is diverted to unauthorized destination.

    The final element is an automatic alarm that goes off at both ends if the connection between the terminal and the bank’s system is interrupted, so that both the bank and the merchant are alerted that something is amiss.

    All of this can be implemented by Visa / MC and the banks. A merchant has no possibility of implementing this kind of end-to-end security. All a merchant can do is put layers of bandaids over a a 20 year old system design that never anticipated the kind of sophisticated hacking attacks being launch by today’s breed of professional IT criminals.

    Fundamentally, this is NOT a technology problem, but a political / financial one. Who should be responsible for the cost of security for the credit card system: the banks or the merchants? To date, there are no clear rules with both parties pointing the finger at the other. Whenever you have two parties jointly responsible for something, neither party steps up to the plate.

    If the rules are changed, so that the banks are 100% responsible for security and fraud in the credit card system (at least for basic credit card terminals), a lot of these problems would get solved in pretty short order.

    That’s not to say that the retailer doesn’t have to worry about the security of his own computer systems. That’s obviously his problem and for many people may require professional assistance. That’s a completely separate issue.

    1. SJ

      I agree with you to the extent that this would secure a transaction for a mundane card present transaction that is keyed in manually. The second you add in another device that proxies the data in the clear, though you’re SOL.

      In eCommerce you’re basically SOL. Various companies like Stripe make it so that the card number never exists in memory of your server, but it’s a trivial exercise to work around that with a man-in-browser attack.

      POS I’m a little more optimistic. If PCI simply requires that the card reading device use a key pair to encrypt all data along with last-4 so it looks the same to the cashier, then the POS terminal can no longer be part of the attack vector. It’s proxying data that is opaque to anyone who can intercept it. The card reading device can work off burned ROMs and rarely needs an update (like you say). The POS terminal can be updated but simply proxies encrypted cards.

      If Target did that (as it did with PINs) I think it would have been much harder to get this done.

  29. Greg Scott

    Mike, why in the world would you get the impression I think you’re a yahoo from the sticks?

    > Your attitude that no device can be designed to be unhackable is BS.

    I’m stickin’ to my story on that one. And it’s not an attitude, it’s the truth. As long as that device interacts with the outside world – and by **definition** it must interact with the outside world, it’s subject to attack and penetration.

    Even if it’s a dedicated appliance with everything burned in to a ROM, armies of attackers will try to penetrate it and somebody somewhere will find a bug and exploit it.

    And what happens if you, the retailer, want to track customer purchases or run some other promotion the requires customization? With the ROM approach, you’re out of luck.

    It’s a fatal flaw to assume any device is safe. The proper solution is to isolate it as I described earlier and monitor the traffic in and out. Even that solution has risks, but they are much less than depending on the device to protect itself. Even a properly designed device.

    As for devices executing data instead of code – well duh! Of course that’s a bad design. Lazy developers who depend on Windows for security. A suggestion was made earlier in this thread that POS terminals should be based on Linux instead of Windows. That’s a great suggestion and I hope it catches on.

    But meantime, in the world in which we live today, I’ve offered a bunch of suggestions for what we can do right now with what we have available right now.

    – Greg

Comments are closed.