Ever since news broke that thieves stole more than 40 million debit and credit card accounts from Target using a strain of Point-Of-Sale malware known as BlackPOS, much speculation has swirled around unanswered questions, such as how this malware was introduced into the network, and what mechanisms were used to infect thousands of Target’s cash registers.
Recently, I spoke at length with Tom Arnold and Paul Guthrie, co-founders of PSC, a security firm that consults for businesses on payment security and compliance. In early 2013, these two experts worked directly on a retail data breach that involved a version of BlackPOS. They agreed to talk about their knowledge of this malware, and how the attackers worked to defeat the security of the retail client (not named in this story).
While some of this discussion may be geektacular at times (what I affectionately like to call “Geek Factor 5”), there’s something in here for everyone. Their observations about the methods and approaches used in this attack point to an adversary that is skilled, organized, patient and thorough.
So you first saw BlackPOS at a retailer in early January 2013?
Tom: Yes, it did seem like a game changer at that point because of the way it hooked into the POS system. By that I mean the fact that it hooks into the POS process, and it’s not just a general memory scraper like some people have stated.
Help me understand the distinction between a memory scraper and malware that runs completely in memory?
Tom: Well, it’s very specific. It’s what’s called an inter-process communications hook. If you look at a memory scraper — so if you were to dump the memory on your laptop right now — you would get this big old blob of information and you would have to filter through it to find what you’re looking for. But this [malware] is very specific: It’s very much designed for the POS system it’s running on, because it knows exactly where to hook and where the memory location is going to be when the data it’s looking for is flowing through it. But if you look at what it captures, it captures only the track data [information stored on the magnetic strip on the backs of cards]. So, it’s actually very, very sophisticated and that’s why I think this isn’t just a teenage hacker who put this together in his basement. I think this is a more sophisticated development effort. [HP last week published some interesting, uber-geeky details about the memory behavior of the version of BlackPOS found at Target].
Paul: It certainly hooks into a specific process, but we did not figure out if it was just good at scraping the memory of that process, or whether it actually altered the process to hook into the code somehow. That part of the code was obfuscated and not reviewed during the engagement.
What did you think of the iSight paper?
Tom: The iSight paper was good, and what it described was very similar to what we saw in the first variants of BlackPOS. But it didn’t talk about how it appeared on the network or where it came from or how a retailer might defend themselves against it.
In your prior experience with BlackPOS, has it been used against the same POS that Target uses? A source who seemed to have a clue told me that their setup was somewhat homegrown.
Tom: That I don’t know. I haven’t really looked at what Target uses. With a homegrown system, the problem is if you’re going to build a process hook, you have to have a test environment, you have to know what you’re looking at and what memory addresses to go after, and that’s not exactly something that gets published.
Paul: Target has a homegrown POS.
So you think it’s likely they were using some off-the-shelf equipment and software? Wouldn’t it be enough for the attackers in this case to have obtained a physical point of sale device that was once used at Target?
Tom: If they have one, sure, that would be different. I don’t know if they’re using a commercial product. A lot of the big retailers use commercial products and will customize those with their back end system. But at the front end and what happens at front of the house…a lot of those are just retail off-the-shelf applications. A lot of those retailers, when you have a hard disk that breaks on one of the [checkout] lanes, they’ve got an outsourced service provider of that POS that comes out there with a new hard disk and fixes it.
How do the bad guys break in, and how do they actually get the malware pushed out to the point-of-sale?
Tom: A lot of the POS systems use whitelisting software of some kind, such as Bit9 or SolidCore. Those two companies are the two you see most out there in the industry.
Paul: It could also come in through the point-of-sale application update process.
By whitelisting, you mean sort of the opposite approach of antivirus, right? As in, if the file or program isn’t approved by the whitelisting software, it simply won’t run on the system?
Tom: The software update processes at a point-of-sale that is running one of those [whitelisting applications] has to come through one of the software update channels and has to be reviewed for the update and approved. And when it’s approved, the whitelisting software says okay this patch is approved to come online.
It’s probably a good idea at this point to make sure we’re defining the terminology in a uniform way. When you think of point-of-sale device, are you talking about the cash register, or the card swipe terminal, or…
Tom: I’m using point-of-sale to mean the payment application that is running on the cash register. The vast majority of those devices, when you check out at the grocery store or large retailer, those are just PCs, and yes they’re mostly running Windows XP or WEPOS as their operating system. But above that, you have what I call the point-of-sale, or the point-of-sale application, and that’s the stuff that the cashier is interfacing with at the time you’re checking out. It’s a software application, running as multiple pieces of software inside the register itself. And whitelisting is put on to protect the register from any sort of unsolicited modification. A lot of the attacks before this [whitelisting became more broadly used] involved where you corrupt a sales clerk to put a USB stick in the cash register and infect the PC with some malware. But by using a whitelisting software, the USB stick will not work and the operations personnel back in the head office will get notified that something isn’t right with that register.
If they get past a whitelisting system, doesn’t that suggest that someone on the inside would have to be involved?
Tom: No, not really. You have to also consider the distribution server that distributes the point-of-sale software.
Paul: Right… three possible ways… it could come through a legitimate update channel, or the retailer was lax in their update procedures, or the attackers hacked the console of the whitelisting software and just whitelisted it themselves.
So when you look at a hacked system, what can you determine about the state of the POS software on the affected systems?
Tom: Well, there can be huge problems if the retail software vendor does not provide SHA-1 hashes or even MD5 hashes of their software. There would be no way to tell whether what you were receiving had not been tampered with..
How does the update process typically work?
Tom: Basically the retailer would receive notification of a patch, and go off to an outside server and pull it down. Very similar to you getting notified of a patch being available on a Windows machine. Except in these situations, when you’re dealing with critical retail software and installations, patching is very formal process, because uptime and reliability is very important. So they test it. They would install the patch on their internal lab servers, run and test it to verify that everything would run smoothly. Then they would then certify the patch when it was ready to run, and then distribute it to stores 1-5 tonight, and then go 6-10 then next day, or whatever their mechanism was to distribute them. But it would typically take them 2-3 months to finish the distribution to all the stores.
One of the questions still outstanding in the Target breach, which is where did this stuff come from and how did it distribute itself across the network? Do you have insights or best guesses on that? And wouldn’t the mechanism like the one you just described be the most likely way…some kind of POS update mechanism?
Tom: Well, if they were not running whitelisting software…maybe, yes. But you know the other thing was that the malicious software we have dealt with has sort of a worm-like characteristic to it. It would keep trying to worm its way around the organization. If it found itself already installed on a machine, it was bright enough to elect that machine and it would start seeking a way out of the organization. When we looked at this in the past, it was very difficult for us to determine how they tried to exfiltrate the information. I think what they did was…since they could find a host — remember, this is an early variant of what was seen at Target — they sort of elected a machine that became the exfiltration point and started doing the exfiltration. Worked exactly the same way, in that the data was encrypted, and moved off to an exfiltration point, and from there it was pulled off to Mother Russia.
Paul: Once they have a password that works on every system, they can use that to spread the malware. There would be nothing that that would stop a POS device in store 12 from talking to a register in store 23.
What type of encryption did they use?
Paul: RSA-512 bit.
But this information wasn’t encrypted by the point of sale or card swipe device at that point?
Tom: Well, remember this was taken off the memory, so it was encrypted from the card terminal, but when it got internally so the [cash register] could handle it, the system itself would decrypt it, and pass it to another process that used for communications, and then it would go off network to get the authorization. And the process hook would grab the data at the point that it gets decrypted out of memory. Then the malware used its own key to encrypt it.
So at that point are there two parallel streams of data here?
Tom: Yes. One good, one bad. And then the malware itself, would then encrypt that data, and pass it to where it was going to aggregate it from all these infected point-of-sale devices, and then the aggregation server would encrypt it again for exfiltration. So it’s pretty sophisticated stuff.
So the bad guys in the scenario that you describe were double-encrypting their stolen data?
Tom: Yes. And that’s why….a lot of people ask the question, hold on a second. Wouldn’t intrusion detection systems flag this; if they see credit card numbers floating around the internal network and moving outbound, they classically raise the alarm. Yet no alarms would go off, if it was encrypted. In the case of Target, it sounds like they were using those encrypted packets to hide from data loss prevention and intrusion detection systems.
Did the forensic report from iSight say the data stolen from them was encrypted in the Target breach? I don’t recall reading that.
Tom: Well, if they were using BlackPOS, I’m pretty certain it was encrypted. Because, if Target is following PCI best practices, there’s no outbound traffic moving at that point, and I would expect that they would have an intrusion detection system in there, because that would be required. And in that case, the traffic would have become visible as card data leaving the network. Now there’s nothing that says you have to do data loss prevention like that [in the PCI standard] but a lot of big organizations do.
We were never able to figure out how they egressed the data. The only protocols that were allowed outbound were DNS. So if they wrapped the stolen data up in outbound DNS packets, maybe. We never solved that problem, we were still looking when the engagement ended. We could see it was doing a lot of the work, but we couldn’t see it was actually leaving the organization.
How is that even possible? That you would not have been able to see how they offloaded the card data?
Tom: There were a lot of modules inside of BlackPOS that were anti-forensics modules, so it cleaned up a lot after itself. It did look for a specific time window for when it would actually move data. There was a whole timing vector as to when it would turn itself on, and then it would start seeking things. We put the software in a lab, but it never [sent data outbound] so we couldn’t quite figure out how. In my client’s case, the firewall rules outbound were very strict, and we couldn’t figure out how it was getting the data out of the organization.
You said earlier that this malware had something of a worm-like capability. Can you talk more about that?
Tom: The iSight report alluded to this when it said there are a whole bunch of hacking and forensics tools in there, and it really did have everything but the kitchen sink in there in terms of tools. The actual transfer mechanism seemed to be a VBSCript. When it got involved, it would very gently start mapping the network, and so that it became aware of what was in the network with it. And then it would start seeking out other point of sale devices. It was very careful, and it took its time doing this. It would attempt to communicate with those other POS devices. And if one of those devices was already infected, the two would form a handshake and a bond, and the two of them would look for the third, fourth and fifth, and so on.
If it would see like a manager’s laptop, it would try to push itself there, but then when it got to that laptop – if it successfully infected it — and it didn’t see the point of sale software, it would run its anti-forensics software and completely destroy itself, so that there was no trace of it at all. We actually took the software and forced an infection onto one of our test laptops, and it only lived for about 15 seconds before it completely destroyed itself. And then it went through and did a whole bunch of anti-forensics work before it destroyed the anti-forensics module. Very cute program. It cleaned up after itself pretty nicely.
So this is why I think as an old software engineer, I was looking at it saying someone did a lot of work to make this thing run properly, and they did a lot of testing. This is not just one guy. If it is the work of just one guy, he’s absolutely brilliant.
ANALYSIS
From talking with Paul and Tom — and in discussing the Target case with security experts from other retailers — a few things seem clear (these are my personal takeaways, not theirs). Firstly, many retailers only update their point-of-sale systems according to a well-planned schedule (a schedule, by the way, which typically happens well before Black Friday — the busiest and most profitable day of the year for most retailers). As a result, depending on the size of the retailer, that update process can take weeks, even months. Their experience suggests that whoever broke into Target was inside Target’s network for quite some time before the point-of-sale malware started stealing card data on Nov. 27, 2013.
Secondly, many reporters and readers have been asking what retailers like Target could have and should have been doing from a security perspective. I don’t have much information on what Target or other retailers were or were not doing, but assuming the attackers typically take months to orchestrate and execute these attacks, it stands to reason that more quickly detecting these intrusions would help quite a bit. According to Verizon’s 2013 Data Breach Investigations Report, 66 percent of breaches that Verizon responded to in 2012 took months or more to be detected. Too often, the breaches aren’t even first detected internally. It’s worth noting that Target’s case, the company’s CEO acknowledged being alerted to the breach on Dec. 15 by law enforcement. In the case of the breach at Neiman Marcus, which exposed some 1.1 million credit and debit cards, the company said it learned of the breach on Jan. 1, 2014, although the actual card thefts occurred between July 16 and Oct. 30, 2013.
Finally, I have to wonder how many times this scene played out in 2013: An individual forensics firm analyzes a sophisticated retail breach involving point-of-sale malware – collecting mountains of interesting and useful data about the threats, threat actors and their methods — but at the end of the day has no mechanism by which to share that information with others in the retail and security community. It’s telling that most of the details about the malware and methods used in the Target breach were first published here on this blog. That’s not a brag: That’s me being incredulous at how the industry as a whole still sucks at sharing important information.
I hope that this interview helps other digital first responders, and that it fosters a more public debate about the malware and miscreants responsible for what appear to be a large number of very similar data breaches that will no doubt continue to come to light this year via additional victim disclosures (willing or otherwise).
Totally agree. Getting more digital first responders to actually SHARE WHAT THEY LEARN is critical to fixing the crisis state we find ourselves in.
I totally agree, there should be some sort of way to securely share information with authorized individuals in the info sec community, i.e. some type of secure framework built with a private-public partnership needs to be created that can facilitate this.
Exactly Gary! At the Senate hearing yesterday the Secret Service refused to answer any questions about the details of the Target breach. The agent wouldn’t even say when it was first noticed at Target. His response was, “Sorry but it’s an ongoing investigation”. What are the protecting here? Without data all retailers are at risk to the advantage of the criminals!
@Edward “What are the protecting here?”
Insiders might be involved; they might be foreign nationals and/or residing in India. A foreign government might be suspected as having known about the actions in advance. Even simple negligence will result in many millions of dollars in legal fees and fines. There is a vitriolic finger-pointing game going on right now between Target and BMC Software. Target’s corporate officers are worried about their bonuses. And so on.
I simply can’t believe any of this. The POS interface into a complex corporate isn’t that simple. This needed inside info.
At least one thing Target could have done and clearly did not use was GeoFencing, there is really no reason any computer inside Target’s network should have been allowed to send a file via FTP to Russia. Not that geofencing would have blocked all exfiltration targets, but that or proxies using reputation scores for Internet destinations might have made things a lot more difficult for the crooks.
Another thing that might have helped Target and others affected by this malware would be to virtualized everything. The crooks are so paranoid about someone figuring out how their software works that a lot of malware simply won’t install or removes itself of it detects it is running in a virtual environment.
Totally agree, GIGA bytes of data leaving the network via FTP is just wrong. Any basic DLP/Firewall/SIEM tool should have at least sent an alert to an adminj.
Regarding virtualization: don’t agree, we call this “security by obscurity” and doesn’t work. As this was such a targeted attack by very sophisticated actors, they would have written the malware to work in a VM, but nonethelss ahev all the anti forensic features.
I’m not sure how much actual data was taken, but if it was just encrypted text files wrapped in DNS packets, it could have been eeked out in chunks over a long period of time to an exploited machine in the US, which then it could be moved it to where it needed to go. Doubt they ever saw any packets headed to Russia
I’m also wondering if the data was being leaked out of one datacenter or if it was sent out before it got to the actual datacenter. It was being stolen at the store level so why not move it out at the store level or at least district level to avoid detection. text file of 5000 shoppers data would be fairly small, couple of mb’s at the most.
A good IPS should be able to detect anything other than valid DNS requests and block anything that didn’t look legitimate even if the data is encrypted or otherwise concealed in DNS packets. Also why not restrict DNS requests to only trusted upstream DNS providers? This would prevent leakage via DNS.
Virtualization is not security-by-obscurity; it forces the virus-writers to tolerate environments where their attacks can be more easily dissected, and it simplifies implementation of some detection algorithms (for example, randomly snapshotting a VM for auditing and forensics).
I can imagine how they’d react, and a countermeasure would be to make utterly legitimate-looking sham purchases so that a terminal’s VM could even be snapshotted mid-transaction.
one of the articles said they uploaded the data first to a compromised websites then the hackers took it from said website to Russia. So said website might have resided in the U.S.. So geofencing/stopping all commutation to Russia not a bad thing would possibly not have stopped anything here.
The interviewees said that they couldn’t figure out BlackPOS method of ex-filtration of collected data. Their customer only had rules for DNS outbound. So.. it seems pretty unlikely to me that they really used DNS packets with an encrypted payload. I guess it’s not impossible, but sure doesn’t seem likely.
We don’t know what kind of perimeter security Target was running. There is some speculation that they gained access initially via a public facing web server, if that were true, 80 is open for business, at least on that machine. Brain’s mentioned quite a bit about SQL injection as a possible method of getting NCat aboard. That rings more true to me. (and i’m totally discounting the possibility of an insider/ex employee with a still valid login ect…) Do these guys know if BlackPOS had an NCat module? I guess that would be part of the “kitchen sink.”
And now this “wormlike” angle? One day we’ll know whether it fell from the sky (as part of patch distribution) or slithered up from the ground (Worm.) But, I Sooooooo want to know how they got the encrypted (d’oh!) data out. I guess there certainly is the possibility that a human with corrupt wet-ware was involved.
Anyway… Great interview! Now I have some new information to chew on…I’ll try to chew quietly 🙂
From Brian’s earlier article, Dell Secure Works CTU documented that the data was exfiltrated from a server on Target’s internal network through a firewall directly to a drop site running FTP in Russia. There’s just no excuse for allowing FTP or really any outbound protocol to anything other than specific trusted IP addresses or addresses filtered by a good proxy.
Seculert continues: “Further analysis of the attack has revealed the following: On December 2, the malware began transmitting payloads of stolen data to a FTP server of what appears to be a hijacked website. These transmissions occurred several times a day over a 2 week period. Also on December 2, the cyber criminals behind the attack used a virtual private server (VPS) located in Russia to download the stolen data from the FTP.
They sure did find a vulnerability and they exploited it. USB sticks? Unpatched software?
It will be interesting to see what information is made available following the forensic work on Target’s systems.
I used to work in the POS industry, one distribution mechanism that most pos applications have is a p2p aspect which is generally used to keep local db’s up to date i.e. update a single terminal with a revised pricelist then let the p2p client distribute the change between local terminals. However most systems also allow graphics, scripts (vbscript and bat) and executables to be transferred in the same manner to allow software updates to happen. the POS software i have worked with had some interesting idea’s about secure coding practices mainly stemming from a walled garden attitude and a belife that the system wouldnt be accessible from the net. i.e. a surprising number of text boxes allowed sql injection, files placed in certain folders would be executed without any checks on authenticity oh and they nearly always run as a administrator due to numerous reasons including putting logo’s on the termal printer attached to the serial port (9 pin, not usb) which again gives you an idea where the industry is in terms of pace of change….
That said some vendors of POS software do a much better job than others, but a lot are still shackled to the legacy of textmode applications which havn’t seen much change since the late 80’s early 90’s
nakchak,
Very interesting. Do you recall the name of the p2p client/process?
-b
On the systems i used it was a generic p2p implementation. Basically each terminal would broadcast over the subnet there presence on a given port (off the top of my head port 64646) and each receiving terminal would build a list of local machines, this was quite handy was we could unbox a replacement machine plug it in then wait 10 mins to know that it had all the customizations a given client wanted on there pos, by virtue of it syncing via p2p to each other terminal on that subnet. Its also used to allow for loss of connectivity to the back office server, as the terminals can still take transactions, by querying each other if a transaction id has already been used, so when connectivity is restored each transaction has a unique id and can be recorded as such. Some places still on dial up will operate in this manner as standard, so they will send the days transactions back to head office at night once trading has ceased for the day.
However it was and still is a huge security risk, especially as they tend to not use any form of encryption, i.e. if you set wireshark or a span port monitoring the subnet you will see everything in plain text, which made it trivial to do all sorts of things the vendor never intended their POS application to do, but thats a different tale
Did CNBC book you for this morning?
BT
There are a lot of ways these devices can become infected, and after reading this article, it becomes more important to separate the trusted, from untrusted networks. I mean totally separate. Don’t use firewalls and IPS/IDS systems. simply build 2 networks, one that contains the critical componets of the POS software and the other network is for all other internet facing devices.
It could have simply been a very specific and well crafted email to some employee that opened it up. Being that this evil software has/had not been discovered, it crawls into its intended target. from there as stated in the article, it crawls around the network, maybe supplies the evil outsiders with some sort of reports of what it has found, was successful at obtaining, and why it failed in other aspects. The evil side makes some minor modifications and pushes it to the victim computers (probably via port 80) in an encrypted format and waits to see if further progess can be made.
While the software is being updated and looking for victims, its actually acting like well, a learning mode software. Its probably modular specific software, where specific parts can be snapped in and out when the evil side pushes an update.
This is why I say totally separate your networks. There is absolutely no reason for the POS side of the house to use any common ports. the IPS/IDS… it SHOULD be IPS – and any packets leaving the building should be decrypted by the IPS inspected, logged and then encryted and then allowed to leave if the FQDN is on the list.
People often say, well the system has been up and reliable for quite some time…. That to me typically means that though it may be secure at that particular moment, it it probably screaming for updates (errrr like POS/XP) or the systems never truly get wiped and a known good copy of the latest software is put back on it.
Why cant the registers have a decent amount of memory and the operating system be run off a DVD? That way, at least each night or even a scheduled time/date, the systems can be rebooted at least weekly, if not nightly. Since there aren’t many updates to this procedure, it would be relatively safe. Deliver those DVD’s like they were gold. Put a paper verification code that the store has to call a human on the other end and say, yeah, I got the latest DVD, here is the one time passcode… If it doesn’t have to be complicated. It verifies that the DVD’s weren’t tampered with.
Theres a TON of things that can be changed. And should be changed. All it takes is one disgruntled employee that is willing to talk about the network, and devices and software thats been used, and the crooks build the software to defeat the security measures in place.
>This is why I say totally separate your networks. There is absolutely no reason for the POS side of the house to use any common ports.
There are no common industry standards or catalog solutions to implement totally separate networks. Having real time inventory control, accounting, and sales trends is extremely valuable. Not impossible, but in most cases would require a complete rework of networking and applications, company-wide.
>There are no common industry standards or catalog solutions to implement totally separate networks. Having real time inventory control, accounting, and sales trends is extremely valuable. Not impossible, but in most cases would require a complete rework of networking and applications, company-wide.<
MPLS VPNs? Sure it might well require changes in applications, but inventory and sales tracking systems shouldn't have any access to credit card data. I can't even think of good reason for your store level accounting system to have access to the stripe data and I could make an argument that you don't even need customer CC information of any type in that. All of this can be exported from the POS back end to another system the stores have access to.
And the ability to think out of the box is nil? If you think the current strategy of protecting business networks is working, its another epic fail. Just because it is different doesn’t mean it wont work. There are ways for all data to pass, it just requires an engineering team to make it work.
And with that, its WORK, not Facebook time, reading personal email time or Youtube time. The web proxy should catch, and management should warn about infractions. Its a message to thwart, but not completely remove most of the casual surfing which,, in my opinion is how people get infected.
Nothing standard is working. So, if your capability requires many to think out of the box and secure your customers valuable data, it should be paramount on each businesses’ mission – for most – it is not. Its sales, marketing and the god almighty dollar. Companies don’t deliberately go out and give up PII, but its damn close to it. If a CEO, CIO, CFO or any other “O” is blind to what has been happening around them, then they have no business in business. Become proactive and protect what is supposed to be protected at the best of your ability without any excuses. If designed correctly the network will be faster, safer, secure and pay for itself through customer satisfaction, quicker checkouts and the safety and security of that PII.
Brian,
Have you cross check the type of POS from those breaches, is there something they have in common ?
The major processors, banks, and retailers do share cyber threat information in forums like the National Cyber Forensics and Training Alliance (NCFTA), a consortium of FBI, Carnegie Mellon University and major retailers, banks, and financial services companies.
Jon,
I caught your post about the NCFTA organization. I’m wondering when the first post of information about BlackPOS or the forerunner BlackMoon was made in tis organization? Were there any hashes posted that could have helped in the investigations?
Since there is still a mystery around how Target’s data was exfiltrated, have folks looked into the possibility that it was commingled with a legitimate transmission to the backend processor and exfiltrated from there?
Just to put the ‘process’ out there, this ‘invasion’ isn’t a “quick” thing (like going out to your vehicle and it doesn’t start). It’s a process that can take months to establish itself within a system; mapping what lives where, how to get to it, what level of permission(s) is needed to do this or that, and which login’s have permission to do what.
As stated in almost all the articles I’ve read, the ‘invasion’ started (at least) mid-year last year with the “results” being what we’ve all been following.
Another item: IMO there would be multiple investigative teams working each breach; this team working the bug(s)/worm(s), that team working the network(s)/server(s), this other team working the computers connected to the network. That can get expensive for the company in a hurry, but may ultimately be the -right- answer to “How the HAIL did this happen?”.
Has anyone found a vetted tool or product that can be used to scan and detect any of the known variants of this malware. I know its designed to avoid things like AV and such, so surely there is something out there that can detect this stuff if ran against a POS. I know Darkreading had a tool from SecureState, but never heard of them and no one has even commented on that post or reviewed the tool seems sketchy.
I don’t mean to oversimplify, but in cobbling together the material in several of these articles, it’s seems to me that a Registry analysis would have detected the malware. Yes, I’m suggesting something that is a completely manual process, but apparently well worth the effort.
Sorry RZ, I missed your comment. I owno of no automated tool that would detect this Malware 100% of the time, but a manual scan would have (I’m pretty sure). I perform registry scans all the time to detect and remove malware.
@hawkeye53
manually scanning registry?
I dont think they are noobs enough to make it is easy to find the malware entries by using Ctrl+F and pasting the POS process.Manually checking thousands of loading points doesnt sound logical.
Registry Scanners:
Most of the registry cleaning tools do not identify malicious entries.
“but at the end of the day has no mechanism by which to share that information with others in the retail and security community. It’s telling that most of the details about the malware and methods used in the Target breach were first published here on this blog. That’s not a brag: That’s me being incredulous at how the industry as a whole still sucks at sharing important information.”
I can relate to your statement. I am a paramedic and share the same concerns in my field of employment. Bad things happen to good people every day and there is very little communication of best practices, opportunities and threats. The ambulance industry in my area is in an information vaccumm…
I read and enjoy your blog because you are a step ahead, relevant and your posts are easy to read and understand.
Carry on and keep us updated I love it!
@naren
I lost track of the forensics analysis, but it’s linked on Krebs. Actually, the signature was pretty obvious as was the loading point. Frighteningly simple! And again, I’m over simplifying, but I’m not talking Registry Cleaners, I’m talking about a scanning tool that enumerates the Registry. Although you can limit the scan to know Malware loading points, let’s pretend you do the whole Registry that you’ve analyzed and now have a solid/good scan. Going forward you do comparitive scans and then only have to analyze the changes.
I guess what I’m saying is, the Malware was in the Registry. A scan and analysis would have revealed it. On a go forward basis Registry Analysis should be part of the Security Enginers day.
Disclaimer: I work for Qualys.
We implemented a detection for blackpos on January 20 in our QualysGuard vulnerability catalog. It looks for the telltale signs of the malware in the registry. It is probably not practical to get the product for that explicit purpose now, but if you are already a customer you can use the detection to gain quick visibility.
Brian,
great and insightful article.
Absolutely fascinating article and comments. Thank you all for helping me understand more fully what is happening in the modern POS arena.
Years ago I worked with a network that, in retrospect, seems like it would have be invulnerable to an attack like this. We used old-style dumb terminals to interact with our in-house servers. There was no internet connectivity allowed at all on our network except at one point at night when our encrypted server data was uploaded in a batch to an off-site backup server. The data was whitelisted in such a way that it could only go to the one off-site server, which was in turn set up to only communicate with us. While that could have been a vulnerability, the offsite server was well tended.
In the case of a large organization like a big box store chain, I am wondering if a return to that kind of architecture might be safer. Data needing to go to banks or credit card companies could be handled in a daily batch from an off-site backup server and, in turn, that server could be whitelisted to only include specific destinations of banks and credit card processors and the whitelist consistently monitored and alarmed. In my case, updates were downloaded to an in-house server, tested, and only made live to dumb terminals when deemed safe at the in-house server, negating any worry about POS injections.
Sometimes simple is better. Anyone else ever wrestle with a Wyse 50? ha ha Just a thought.
Never messed with Wyse terminals, but had my share of ADDs ones. The problem with your daily batch scenario is that its in everyone’s best interest (except the consumer) to move those credit card transactions through as quickly as possible. Moreover, the industry is (has) moved away from dial-up over analog phone lines as the means to transmit credit card authorizations. They’d rather do it digitally over the Internet. Granted, all the data is encrypted over the wire, but I’ve for some time now been wondering whether it was more secure in its analog form.
VT100s… Those were the good ol’ days. Too funny. It was usually the application needing ADDS-VP that made me blanch, though.
All of your points are valid, VT. The current internet-dependent system is driven by the merchant and bank’s desire to quickly verify the validity of a given card at a POS (e.g., is the card maxed out? stolen?). A batch processing system could still work though, if banks and credit card processors would encrypt and download to corporate batching servers a table of known bad card numbers each day. Then, as the dumb terminal entered the number the next day it would be checked against the table and the transaction refused if the number was found to be bad. Analog lines aren’t a bad idea overall, either. I have seen a system work that was designed this way and employed that kind of checking, but not on the scale of a Target corporation. Just a thought…
Another real interesting article. I am starting to worry about using POS devices in retail stores now. Not good !
I am by no means a pos expert, but would randomizing the way the track data is stored make the hook useless? The data was encrypted at the pos device, then decrypted by system to verify the cards good then it was encrpyted again and stored. There has to be a way to randomize how the data is stored locally at every pos thereby making the inter process communication hook useless, they would then have to memory dump the entire contents of the memory making the track data capturing technique useless. Also that means they now have to dump more data offsite thereby increasing the chances of disovery..
Using a hash to verify the contents of the track data seems to be more secure then allowing the track data to be decrypted for verification.
I also agree with a earlier comment, a seperate secure network for the pos devices with no internet connectivity.
Of course there would be ways to get around these techniques. If a hacker wants in they will get in eventually. I was taught you have to make yourself a more difficult target then the next guy so they give up on your network and try elsewhere.
Kreb your the man, love your blog.
If *most* POS/cash register hardware is still running some form of XP, shouldn’t the vendors of such systems either have begun rolling out system upgrades or acknowledged in their public business documents (10-K, etc.) a program or schedule to do so by this point? After all, the end-of-life deadline for XP is now just a couple of months away, and there really hasn’t been much in-depth coverage of how serious a potential issue that presents for the retail industry or a wide swath of society that’s come to rely on debit and credit cards instead of using checks or cash — may be a “Y2K on steroids” scenario for real.
@JimV “After all, the end-of-life deadline for XP is now just a couple of months away”
Do not confuse XP with WEPOS; the latter will be supported for another year or two (using Bing, search for “WEPOS lifecycle support”).
Sorry, I don’t (and won’t) use either Bing or Google for Web searches; DDG serves me quite well — but thanks for the clarification.
Point well taken; I also use DuckDuckGo.
I should have said to search on microsoft.com for that phrase. I use Bing only for Microsoft searches; if that’s what they keep in their files about me, no big deal.
For detecting the running malware – memory forensics is normally used AFTER you have determined there is an issue. Tools like Mandiant Redline can do live memory analysis, check MD5, code signing and check for indicators of compromise. There are probably others as well.
Totally agree with IA Eng – air gap / enclave is they way to go. Completely separate network for POS – but the network may get tricky MPLS/VPN store corp corp to processor. Whitelist by IP, FQDN if you have to but tight control of DNS. RSA removed their airgap to speed up a business process, that was a big contributor to their breach.
Compromising patching systems is golden, since they have access and trust. Same goes for code repositories SVN, CVS, VSS, … Restrict access to these by tight ACLs and 2 factor.
Whitelisting should be solid but once again if it can be compromised and added that would be bad. Tight ACLs and 2 factor to access this system.
Egress filtering – I agree theoretically netflow may have shown some anomalies, but IPS should not be used as DLP = most use a simple regex against traffic which is very costly on CPU for an iffy match and have never evaluated with an SSL decrypt module so not sure options to handle encrypted data it cannot decrypt. DLP policy can be very granular – decrypt outbounds and inspect -> if cannot decrypt -> block and alert. If someone is egressing in ICMP or DNS then aside from packet shaping and enforcing strict RFC compliance, not sure what you would do.
“That’s me being incredulous at how the industry as a whole still sucks at sharing important information.”
Brian, surely you know the reason for this. It’s not because they suck at it, it’s because they don’t want to do it. This kind of information is a trade secret for a security firm that does reverse engineering. If they put all that hard work into figuring out how malware works, then just give it away, they are at a competitive disadvantage.
I would love to see sharing of this sort, but we all know this doesn’t happen. At least it doesn’t happen until a crisis of massive proportions hits. Short of (hopefully well-crafted) legislation, I don’t see how this will change.
That’s not really true – many antimalware companies regularly exchange information and samples all the time.
The real issue here has nothing to do with the forensics teams that retailers bring in to investigate – the issue is with the retailer itself.
tl;dr version: it’s their lawyers.
They’re so terrified of litigation in cases of a breach, they immediately go radio silent other than PR-filled invectives about how truly sorry they are and here’s a year of free credit monitoring. Oh, and did we mention how sorry we are?
But that’s as far as they go. And likely that’s as far as they will ever go until Congress *legislates* something providing some sort of limited immunity if a company goes public with their post-mortem analysis instead of the woefully ineffective self-policing PCI DSS standards.
procrast – track data is read by the swipe device, which in Target’s case all of the terminals I have seen are Verifone, and the only part that would be encrypted at this point is any PIN entered as part of a debit card transaction which is encrypted in hardware in the terminal and is not decryptable by the merchant. Everything else is plaintext by default as it is going over a usb/serial cable back to the application on the XP register. It would be possible to crypt that link to prevent a physical sniffer from intercepting it, but hat would have to be decrypted on the POS register to forward via ethernet to the payment gateway. There IS an end to end encryption option where the card is encrypted at the swipe and passed encrypted through the entire merchant system (Target) then decrypted by the processing bank or third party. This would have prevented this attack (it would have forced the attack to occur elsewhere or the attack to take place against the card reader directly – which would likely involve physical tampering which is problematic as it is slow and required physical access). EMV will be deployed to the US by October 2015 and would also have thwarted this attack.
By the way, to all you internet security fans out there, Firefox 27 was released today. Not much of of a update other then the Transport Layer Security (TLS) version 1.2 has been added to both desktop and mobile versions of the new browser .
The above is posted in the wrong article Sorry !
One element that seems to be getting glossed over a bit is the role very sophisticated organized crime. There’s this quasi-romantic mythology of industrious teen hackers infiltrating a network via clever firewall tricks. But I find it a lot more likely that the software was assembled under the aegis of Russian organized crime, and that overseas contractors with access to Target’s internal network were a vector for delivery. Contractors who applied to be hired by Target specifically for the purpose of industrial espionage. And since the Venn diagram of Russian organized crime and the Russian government pretty nearly overlap, this was actually a state-sponsored attack on a US retailer.
We’re not up against clever kids, we’re up against dead-eyed killers with vast resources, out for big cash.
It is a shame that they weren’t able to complete the forensic analysis.
The problem with trying to grab memory dumps of infected processes (in general) is that the malware could have a rootkit component, and it could detect the attempt to open the process handle and then hide its tracks. One might need to use a kernel debugger with an infected machine to get the dump, but the malware might also detect this. It can be a very tricky game of cat and mouse to get anything good.
POS used to stand for POINT OF SALE but now stands for PIECE OF SHIT & users need to decide if they will now use CASH for sales & refunds now you cant trust your credit/debit cards to the network.
Seems windows is still the problem here with so many vulnerabilities that simple VBSscrips can compromise whole networks! < thats the problem M$ shitOS's!!!
settle down, Beavis 🙂
This just as easily could have happened if they were running Macintrash or Sinux. The issue is more of a lapse of administration and security than their choice of OS.
A point that I’ve not heard mentioned yet here in the comments refers to this statement:
“t’s telling that most of the details about the malware and methods used in the Target breach were first published here on this blog. That’s not a brag: That’s me being incredulous at how the industry as a whole still sucks at sharing important information.”
There were people who had information, but due to ongoing investigations and TLP RED/AMBER considerations could not share that information. While the publication of the details opened the floodgates and provided for much discourse, the chances of a law enforcement investigation being impacted were present. I am not working the breach case personally so I am only speculating, however I know that as soon as that malware became well known, many researchers were surely attempting to connect and see what was there. Can anyone who may be from LE make a comment about this here?
The second point I want to make is that while DNS was only allowed outbound (and may have been the vehicle for exfil, as previously discussed), clearly internal SMB/CIFS type traffic was allowed based on connectivity indicators inside the malware (already pointed out multiple times). Any internal system that can talk to both the internet, or to a system that can get to the internet, that can also talk to anything that can talk to the PoS machines is vulnerable. This is stating the obvious, but without a proper network architecture from the get-go, it’s hard to put something like this into place. There is a reason why these lateral-movement-inside-the-org attacks are so popular and damaging.
I do hope that more details emerge after the LE investigation is wrapped up, and that other orgs can use those details to help them increase their defense.
gee whiz….we are only 30 years behind the rest of the planet with credit cards security ..so why surprised…and sharing when you can ruin your competition…until it becomes expensive for the credit card companies or we go back to cash….we are stuck…solution? Get real credit cards…