The breach at Target Corp. that exposed credit card and personal data on more than 110 million consumers appears to have begun with a malware-laced email phishing attack sent to employees at an HVAC firm that did business with the nationwide retailer, according to sources close to the investigation.
Last week, KrebsOnSecurity reported that investigators believe the source of the Target intrusion traces back to network credentials that Target had issued to Fazio Mechanical, a heating, air conditioning and refrigeration firm in Sharpsburg, Pa. Multiple sources close to the investigation now tell this reporter that those credentials were stolen in an email malware attack at Fazio that began at least two months before thieves started stealing card data from thousands of Target cash registers.
Two of those sources said the malware in question was Citadel — a password-stealing bot program that is a derivative of the ZeuS banking trojan — but that information could not be confirmed. Through a PR firm, Fazio declined to answer direct questions for this story, and Target has declined to comment, citing an active investigation.
In a statement (PDF) issued last week, Fazio said it was “the victim of a sophisticated cyber attack operation,” and further that “our IT system and security measures are in full compliance with industry practices.”
There is no question that, like Target, Fazio Mechanical was the victim of cybercrime. But investigators close to the case took issue with Fazio’s claim that it was in full compliance with industry practices, and offered another explanation of why it took the Fazio so long to detect the email malware infection: The company’s primary method of detecting malicious software on its internal systems was the free version of Malwarebytes Anti-Malware.
To be clear, Malwarebytes Anti-Malware (MBAM) free is quite good at what it’s designed to do – scan for and eliminate threats from host machines. However, there are two problems with an organization relying solely on the free version of MBAM for anti-malware protection: Firstly, the free version is an on-demand scanner that does not offer real-time protection against threats (the Pro version of MBAM does include a real-time protection component). Secondly, the free version is made explicitly for individual users and its license prohibits corporate use.
Fazio’s statement also clarified that its data connection to Target was exclusively for electronic billing, contract submission and project management. The company did not specify which component(s) of Target’s online operations that Fazio accessed externally, but a former employee at Target said nearly all Target contractors access an external billing system called Ariba, as well as a Target project management and contract submissions portal called Partners Online. The source said Fazio also would have had access to Target’s Property Development Zone portal.
According to a former member of Target’s security team who asked not to be identified, when a work order for an external vendor is created, the payment is collected through the Ariba system: Vendors log into Ariba, complete the necessary steps to close out the work order and they are later paid. But how would the attackers have moved from Target’s external billing system into an internal portion of the network occupied by point-of-sale devices? The former Target network expert has a theory:
“I know that the Ariba system has a back end that Target administrators use to maintain the system and provide vendors with login credentials, [and] I would have to speculate that once a vendor logs into the portal they have active access to the server that runs the application,” the source said. “Most, if not almost all, internal applications at Target used Active Directory (AD) credentials and I’m sure the Ariba system was no exception. I wouldn’t say the vendor had AD credentials but that the internal administrators would use their AD login to access the system from inside. This would mean the sever had access to the rest of the corporate network in some form or another.”
Last week’s story about Fazio’s role in the attack on Target mentioned that Target could be facing steep fines if it was discovered that the company was not in compliance with payment card industry (PCI) security standards. Among those is a requirement that merchants incorporate two-factor authentication for remote network access originating from outside the network by personnel and all third parties.
Another source who managed Target vendors for a number of years until quite recently said that only “in rare cases” would Target have required a vendor to use a one-time token or other two-factor authentication approach.
“Only the vendors in the highest security group — those required to directly access confidential information — would be given a token, and instructions on how to access that portion of the network,” the source said, speaking on condition of anonymity. “Target would have paid very little attention to vendors like Fazio, and I would be surprised if there was ever even a basic security assessment done of those types of vendors by Target.”
But according to Avivah Litan, a fraud analyst at Gartner, Target wouldn’t have needed to require vendors to use two-factor logins if the company believed it had taken steps to isolate the vendor portals from its payment network.
“In fairness to Target, if they thought their network was properly segmented, they wouldn’t have needed to have two-factor access for everyone,” Litan said. “But if someone got in there and somehow escalated their Active Directory privileges like you described, that might have [bridged] that segmentation.”
OPEN-SOURCE INTEL
Many readers have questioned why the attackers would have picked on an HVAC firm as a conduit for hacking Target. The answer is that they probably didn’t, at least at first. Many of these email malware attacks start with shotgun attacks that blast out email far and wide; only after the attackers have had time to comb through the victim list for interesting targets do they begin to separate the wheat from the chaff.
But Target may have inadvertently made it easier for the attackers in this case, in part by leaving massive amounts of internal documentation for vendors on its various public-facing Web properties that do not require a login. Indeed, many of these documents would be a potential gold mine of information for an attacker.
Here’s an example that just happens to be somewhat specific to HVAC vendors: A simple Google search turns up Target’s Supplier Portal, which includes a wealth of information for new and existing vendors and suppliers about how to interact with the company, submit invoices, etc. That page leads to a separate page of information on Target Facilities Management, which includes a slew of instructions on submitting work orders. That page also includes a link to another set of resources: A Supplier Downloads page that, oddly enough, is little more than a long list of resources for HVAC & refrigeration companies.
What could an attacker learn from this information? For starters, download any of the Microsoft Excel files listed at that page. Then scan the file with a free online service (like this one) that extracts metadata from submitted files. Scanning the file “FM_HVAC_Oct_2011_Summary.xlsx” from the Supplier Downloads Page, for example, tells us that the file was created in June 2011 with a copy of Microsoft Office 2007 licensed to Target Corp. That metadata also indicates the file was created or last edited by a person with the Windows username “Daleso.Yadetta,” and that it was most recently printed to a system on Target’s network in the following Windows domain: “\\TCMPSPRINT04P\”
Getting the layout of the various Windows domains within Target’s internal network would certainly help the attackers focus their attention. For example, consider what we know about a key piece of the malware known to have been used in the Target intrusion, first referenced in a story on Jan. 15, 2014. Investigators who examined the malware quickly noticed that it was designed to move data stolen from Target’s (then malware-infected) cash registers to a central collection point on Target’s network, a Windows domain called “\\TTCOPSCLI3ACS\”.
Investigators believe “ttcopscli3acs” wass the name of the Windows domain used by the POS malware planted at Target stores.
A little Googling shows that Target operates two data centers, both in Minnesota: Target Technology Center — or “TTC” for short — is located in Brooklyn Park, Minn. on one of the company’s corporate campuses. The company also operates a redundant data center — Target Technology Center Elk River (TTCE) in Elk River, Minn. as a backup. It’s a good bet that the server referenced inside the malware used in the Target breach resided within the company’s Brooklyn Park facility.
Free version of Malwarebytes Anti-Malware. This supposed to be “advanced” ? Seems they are invesdting is something better for sure 🙂
The “FREE” version of any solution is woefully inadequate for an enterprise. Malware Bytes Anti-Malware has an enterprise edition. Not saying it would have caught this – especially of it were an advanced persistent threat; but you know that no one was scanning with it manually enough during daily operations to amount to a pimple on a nat’s behind.
I’m pretty sure you can’t schedule auto-scans on the free version of MBAM. REAL TIME PROTECTION is the only hope.
If it was a targeted attack, at least one that was customized for the initial shotgun spread attack, then any real-time or scan-only system may not have caught it until days or weeks later.
Definitions are based on signatures of executables, and when you have unique signature because it’s a new piece of malware then the protection company needs to create new definitions to match the malware. This is a problem with any day 0 malware, and when attackers target a small group it can take a long time indeed for definitions to get created since the chance of that group capturing and submitting the malware is lower (and if they’re running their software in a noncompliant manner, the probability approaches zero quickly).
This is why whitelisting systems are becoming all the rage, they prevent these day 0 executables from running because they don’t match the whitelist of allowed executables. Definitions are basically a blacklist of disallowed executables, and you can’t blacklist an executable you haven’t seen yet.
This a gross oversimplification of the issue since many pieces of malware hijack onto day 0 exploits can end up being whitelisted by virtue of piggybacking into an existing whitelisted executable, but usually the first step or two after accessing one of those exploits is to install new processes that can perform a far greater range of tasks (which would then be caught by a whitelisting, or eventually by a blacklisting, system).
I know Windows Parental Control set to whitelist applications, has worked pretty well on my honeypot test lab. But since we seem to be talking about XP here in the environment described; that probably isn’t available.
There’s nothing wrong with MalwareBytes. It is good software. It is good at finding infections that other programs miss. But it doesn’t have real-time protection except in the Pro version. Also, the registers should not have been directly connected to the rest of their network.
I wonder if the Target users had admin access to their systems? I’ll bet the HVAC supplier’s users did since they don’t sound like they were on the ball. One of the easiest ways for a company to protect itself is to not allow users administrative access to their systems. Even the sysadmins should be forced to login separately to an administrative account to make changes.
FTA found here -> http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000009407
” We’ve had situations when we needed to deploy important security patches to more than 300,000 endpoints in our stores within a 24-hour period, and Microsoft lets us do that.”
I was curious as to how Target scheduled updates to the POS. It’s worth reading if you’re interested in how Targets network is structured. Brian may have already linked to this (sorry for redundancy if so.) Of course it’s an MS sales pitch for Hyper-V, but I found it enlightening. YMMV.
Interesting read, in keeping with Brian’s osint point the info about SCCM use might have been a big help to the attackers. Only, this time the “important security patches” were malware.
The problem as I see it is that their is huge fundamental problem with sending emails. Time and time again a unsuspected person opens a spear phishing email which leads to malware infection and then to internal network intrusion
Once the attackers is on the network, they clearly are able to perform reconnaissance for an extensive period of time to find systems that would enable the distribution of their malware. That suggests that in the case of Target had inadequate intrusion detection deployed that could identify extensive probing of their internal network, especially critical network segments where the POS systems reside. This all started because a person opened a email at a third party vendor. Makes your wonder if the problem doesn’t start with the process of utilizing email and it’s major flaws.
The main point of failure here is email and how it’s received and sent over the internet. A company like Fazio Mechanical appears to me like they didn’t properly train their employee’s on proper internet safety and or setting up a company wide policy on how to deal with incoming emails. One of the biggest problem in internet security as I see it, is that the way email is being sent hasn’t changed since the early days of the Internets inception. What is needed in my opinion is to fix many issues that relate to how email is sent and received . This would include the process on how the fundamentals of how STMP works by adding security measures . This would include , but not limited to adding DNS server authentication in the STMP process , using high grade security tokens to authenticate outgoing emails , the move to using PGP for worldwide use by making it more user friendly or even using some type of process by which the sending and receiving of emails utilized some form of 2FA.
Making changes to how email is sent and received would stop a lot problems like spam , phishing scams and other abuse like opening of malware attachments that leads to computers being infected . If you can’t change people’s behavior on how they use or interact with email due to the tactic of social engineering, then the old outdated technology that’s behind SMTP needs to be drastically changed for the current times.
I don’t personally know ANY companies that give training on how to use email. Maybe government offices, the military, government contractors, banks, etc. do, but I have not encountered any company that does this.
All these discussions around which AV product was used, how the attackers moved from Fazio to Target, why the networks should have been segmented, how and why an employee clicked on a malicious link…etc all strike me as being pretty irrelevant.
The facts are that a determined enough adversary is going to penetrate your network regardless of all the protections you put in place. The way that businesses leaders are demanding remote 24/7 connectivity for employees, suppliers & clients with the security implications having to take a back seat just makes it all the more inevitable. That does not mean that we just give up and open the doors. We just have to do the best we can with the tools & budget that we have but prepare for the worst.
Where Target fell down as far as I can see is the lack of monitoring. They should have been able to detect the malicious activity way sooner than they did and react faster & better to it.
Don’t take my word for it. Read the excellent series from the folks at Securosis
https://securosis.com/blog/why.-continuous.-security.-monitoring-new-series
https://securosis.com/blog/continuous-security-monitoring-defining-csm
Exactly. We harp on things like lack of security training for employees and ignore the fact that they have a job to do that isn’t security. We need to accept insecurity, plan for it, and make sure it doesn’t hurt us.
Unfortunately, a lot of the “make sure it doesn’t hurt us” will be shifting the pain to someone else. Fazio and Target have their insecurities, but it is the grassroots consumers who feel the pain.
Great graphic on the story! Who designed it? Devilishly clever.
As irresponsible as this reports paints Fazio, Target is more so at fault for their lack of security protocols….. laziness again, Fazio it was not a “sophisticated attack” more so Fazio was a much less than sophisticated Target!
I’m an IT guy in a retail org. I have a CS degree and decades of experience. My perspective:
First, PCI is impossible. It is nearly rocket science. It doesn’t help that there is a lot of non-formal hand waving in the “spec” if you could call it that. The authors probably don’t even realize where they are imprecise, and that they have no formal proof of correctness that PCI compliance will secure a network. Short of that, it’s just an elaborate set of opinions and general advice and conventional wisdom.
Secondly, note that PCI exempts encrypted info passing over the untrusted public network. The credit card processors have standalone equipment that encrypts so that the merchant’s network can be untrusted. Now ask yourself, why in the world would they define the processing protocol so that protection ends at the outside of the merchant’s network, when providing protection all the way is provable.
As an aside, it may be from what I read that the information taken was encrypted. It may be that Target did not have a CDE subject to PCI, and that its network was just part of the public untrusted network from a PCI point of view.
There should be NTSB style investigations and full public reports, with recommendations. That has made air travel the safest thing on the planet.
Note that at the center of all of this is the monopoly known as Microsoft, which has chosen for years to make things automatically connect in every way and to require people to understand as little as possible. Windows is infamous for its vulnerability, and my professional opinion is that it goes way beyond just being the attractive target. The MS ecosystem is so incredibly and hopelessly complex that no one small group of experts can understand it and no mere mortal developer can write secure software.
The business transaction is a three-party deal. Customer wants CC Lender to pay Merchant. In the early days, the merchant acted as the custodian of the paperwork. Anytime in the last decade, it should have been restructured — there is no need for merchant to ever have or handle any of the info between customer and CC issuer. It only needs a binding agreement to be paid and a voucher reference number.
Hopefully the electronic payment systems will get this right, and the CC giants will die the slow and agonizingly painful death it deserves, along with the publisher of the most hacked OS by far.
I understand why merchants use Windows — it’s what the POS software runs on. But I cringe to think that the only thing between my customers and the hackers is Windows security. It doesn’t matter how many ports you close if the system helpfully passes unrelated info along at every chance.
Liked the NTSB-style investigation, would love to believe it’s practical, but am pessimistic that the entrenched special interests would stand for it.
One thought, don’t know if it would work but perhaps have two styles of investigation: open public process for third-party reported breaches, and a more confidential process for self-reported incidents. That would provide a motivation for better monitoring and proactive measures. Without some such alternative I’m afraid that the possibility of an open public-visible investigation would just lead to stonewalling.
**Total speculation**
As far as I can tell, Neiman Marcus also uses Ariba. Could bad actors be using Ariba as an attack vector?
http://www.wikinvest.com/stock/Ariba_%28ARBA%29/Customer_Acquisition_Transactions
Everyone seems to be focused on the Ariba portal, but my sources told me there were likely at least two other portals that Fazio had access to:
“Fazio’s statement also clarified that its data connection to Target was exclusively for electronic billing, contract submission and project management. The company did not specify which component(s) of Target’s online operations that Fazio accessed externally, but a former employee at Target said nearly all Target contractors access an external billing system called Ariba, as well as a Target project management and contract submissions portal called Partners Online. The source said Fazio also would have had access to Target’s Property Development Zone portal.”
***yet more intuition***
Ariba. Who do you associate with that word? Mr. “Speedy” Gonzalez? Maybe it’s only me…. But, if you were a smart-***ed dump seller, and you knew the provenance of your wares, might you name them something “anti” Ariba (slow.) Like… a Tortuga?
Maybe I should get back to work.
Would it be too difficult for companies such as Target to simply ask their vendors to identify which security systems they are using, and to prove it by providing licensing information? If the system isn’t on Target’s “approved list” and/or cannot be verified, then slam the doors on the vendor. This simple system has been used for decades by general contractors, who require solid proof of insurance, licensing and workman’s comp from sub-contractors before any work begins. It’s senseless to do any less.
Chuck – Companies conduct risk reviews against vendors constantly. Tons of resources are sunk into identifying, profiling, and risk assessing third party service providers. This doesn’t necessarily make a company secure. Requests for information regarding security controls are often made. Unfortunately, many contracts do not require a vendor provide this information (legal/procurement/contracts is still behind the ball on this – mainly because if the business needs a vendor to generate revenue they are going to get the vendor. Remember this is a business. their goal is to generate revenue not secure peoples data – agree/disagree as you may, that’s reality). Even if a vendor is obligated, just because a vendor may have a control deployed doesn’t mean it is being used effectively. From my experience there aren’t many, if any, security controls that you just plug in and they work perfectly…without proper architecture, configuration, baselining, fine tuning, support, analysis, etc. Vendor risk management is a very tricky subject. I think the focus needs to be on Target not Fazio. Fazio is an HVAC shop…you really think they know a damn thing or care a damn bit about IT security?!
not true at all! they should be doing all those things with contractors, but they absolutely do not! lip service at best.
In addition to this, having Target do that would be a combination of fallacious Security-by-Checklist with Security Theater.
Plus, that puts a verification burden on Target. They can implement any sort of requirement they please, but it’s nothing more than lip service if they can’t actually verify compliance.
As I said before, external entities are simply beyond Target’s control. The best thing – really, the only effective thing – would be to design around the notion that the external entrant cannot be trusted. That is one hell of a large task of course, far easier to say than to do, but doing otherwise denies them the ability to control the risk without the benefit of actually reducing it.
JLind: Thanks for that info. Now I know.
Having worked for a bank, we were expected by industry regulators to understand the security posture of our suppliers; this was typically done by requiring the supplier to answer hundreds of questions across all security domains and less frequently, going onsite for a deeper and closer look. Our highest priority suppliers had persistent network connections to our network; these were in the hundreds. There were far more suppliers that had login creds to portals similar to those accessed by Fazio and we didn’t have time or resources to assess such parties. One must take a risk based approach and I agree there is an element of security theater involved due to the overwhelming number of third parties in the picture.
The operational considerations aside, I imagine that their legal advisories would put the kibosh on it for liability reasons. If Target creates a whitelist of AV/anti-malware software that vendors must adhere to and something happens, from security to “my mouse is acting funny” or “I can’t find my document”, Target is going to get a support call, and maybe a letter from someone’s lawyer. The rationale, or irrationale, being, Target said told me that I had to use this.
Steve; Thanks for the reply, and I see the problem. But wouldn’t a simple requirement for security verification, without a white-list, have prevented Fazio from using – illegally as it turns out – a bare minimum freeware system?
the white listing idea for a minimal requirement, as Chuck mentioned, at least would have gave a starting point, though even if it wasn’t perfect, at least having something in place is better than absolutely nothing at all in place. Steve did have very valid comments, but there can be more explicit/definitive and written out rules as a basis to have something to go by. i disagree that if it was implemented that Target’s IT support would then be legally liable from everyone else as a result. There should have been at least some form of “Trust, but verify” type idea when the company was first brought onboard as a possible subcontractor. From that point, that made them negligible in not following up with what they were using. Without equal levels of security, and verification, one is left with exactly what happened: one company having ok enough type cyber-security in compliance with law,.. roughly, & the other with using ( & illegally at that) a free-ware version of an on-demand virus scanner ? Far more could have been done on Target’s behalf to keep their PCI data segregated & safe from any entry point that someone from the HVAC account could have done on their network. However alas the ‘bean counters” (as a speculation as to why it was not done, as a motive) likely gave the money costs of doing the extra work and equipment, maintenance costs, work hours & etc that it may have been “cost-prohibitive” to do so previously. Lastly, one big problem with american society, is that we all just want to do what we want to do & most people don’t want to know anything beyond what they already know about computers beyond the bare minimum in all of our hectic & fast paced lives of rush, rush, rush: hence, for my point that a great percentage of society has only basic idea that malware & viruses = bad, no malware & no viruses = good but that is about all & they don’t care beyond that about computers, networks, & the like. How does one go about fixing that… ?
Finally we get an idea how it got on their network. This is what I’ve been waiting for every since Brian broke the story. As usually it is stupid is as stupid does. No amount of security can fix that!
At the same time, I gotta wonder if the cheap retail sector will ever be ready to pay for more effective standards? Maybe after paying billions in a class action? Nobody in industry seems to want the price for a competent IT staff, or especially for the same thing in a contractor.
The FBI seems to be refuting the Ariba theory in this release.
http://blogs.wsj.com/law/2014/02/10/cyberattacks-on-retailers-not-coordinated-says-fbi/
Hi Jeff,
How exactly does the FBI report refute the Ariba theory? Have you seen the FBI report?
It is NCIJTF not FBI report. Does not specifically address Ariba but does indicate different TTPs w/b required for each network. That makes sense, there are probably different network topologies, perimeter defenses, etc. so assuming one common system is enough to explain the whole thing is overly simplistic. Basically the bad guys are professionals who are working hard at this.
Keep that in mind and it changes the whole perspective. We do ourselves a disservice blaming the victims and looking for simple failures on their part. Try the assumption that the folks running Target were just like you and your colleagues. It’s uncomfortable because it makes you realize you could be next, but it’s probably more realistic than thinking they were dumb while you are smarter. That leads to complacency, and you put yourself in line to be the next victim.
+1
+1
…../ )
…..’ /
—‘ (_____
……… ((__)
….. _ ((___)
……. -‘((__)
–.___((_)
That article doesn’t(refute Ariba theory) .
But…..
It possibly says something; that the FBI considered the recent Big Breaches had the potential for a hefty whack at the nation’s financial base, and being from one point of attack or from multiple sources doesn’t diminish that issue.
The NCIJTF report does say that although there is no evidence of a coordinated campaign targeting the US economy, the economic impact “resulting from the successful use of POS malware…cannot be overstated.”
My first reaction is that Fazio is correct in stating that they are in line with accepted industry practices by depending on unlicensed free software, although that might be a tad cynical. It would be interesting to see an honest survey of what tools are really deployed and used (two categories, those are not the same thing) in the SMB sector.
As someone who does IT for an HVAC contractor, I definitely agree that a free on-demand malware scanner (that’s not even legally licensed) certainly isn’t appropriate protection.
However, it’s also important to remember it’s not reasonable to expect anything like the same kind of security posture from smaller companies with vastly smaller resources than one has every right to expeect of large Fortune X corporations that process large volumes of payment information.
Well, on top of that too, it’s a distraction. Really, Malwarebytes real problem is that in the free version it’s not a proactive scanner and that’s it. Would anyone really have been feeling different had Fazio been using some enterprise system and employing a really astute IT staff? You’re only as secure as your most uninformed user, and if someone fell prey to a phish, then it wouldn’t have mattered if they were using a full-on, centrally managed Shavlik or MS SCCM solution handling a fully-up-to-date EndPoint protection product. The staff member still fell prey to a phishing attempt.
Again, the real subject of discussion should be Target, not Fazio. Focusing on what Fazio should’ve done is concentrating on the minor aspect at the expense of looking at the major one. Target should’ve had things configured to where an Ariba compromise doesn’t allow access to the PCI data. Yes, that’s easier said than done plus it’s slightly idealistic a statement to make, but the point is that even making Fazio the best, most IT-secure company around doesn’t address the core issue.
From what I can tell at this point, Fazio’s role is minimal. The real failure is on Target’s end. Fazio’s screwup shouldn’t have had this large an effect. And all Fazio being secure does is fail to illuminate the very real problem on Target’s end.
@E.M.H.
Sorry if I wasn’t clear. My point was exactly what you said – it’s really about Target and Fazio is mostly a distraction. Fazio could be expected to have done more, but not a whole lot more. And they’re simply not the responsible party. Target was the one responsible for securing the payment data. Putting extensive security requirements onto contractors would just ensure there are fewer contractors out there.
” it wouldn’t have mattered… The staff member still fell prey to a phishing attempt.” said E.M.H.
Actually, it might have mattered a lot, if they were also using Secunia to keep all known exploits blocked by keeping vulnerable apps patched. Also, if the user was not logged in as an admin or protected by DropMyRights on XP, it’s very likely the phish would have failed.
When I was running XP, Thunderbird never had Admin privileges, as it was always invoked by DMR. Any click on a link or attachment in a message was also opened without Admin privileges.
This is still not an excuse for Target’s malfeasance.
If you are mad as hell like many of my clients about this; you might be motivated to go here:
http://cu.convio.net/site/Advocacy?cmd=display&page=UserAction&id=2869
While Fazio has a certain responsibility here, and will likely lose Target as their customer, focusing attention on them as the problem seem highly misplaced. Target has a responsibility to protect its own assets and that begins by distrusting *everyone* equally, including their partners and vendors. Target’s gross incompetence was the real thing exploited in this attack. Sadly, until these attacks seriously impact the bottom line, security theater and lip service will continue to be more important than actual information security.
Good line on thinking on how to use simple available tools to map things out.. Addition to the scan of public facing files… The background information for the owner mention is easily found in LinkedIn 😉
If you find yourself in a position trying to decide what to do about a vulnerability, here are some thoughts.
http://www.infrasupport.com/what-is-a-good-practice-around-dealing-with-security-vunlerabilities/
– Greg Scott
– Brian you rock. For InfoSec pros you are the go-to source for understanding attacks and enabling us to learn. Invaluable service – thank you!!
– Any security pro who takes the attitude “oh Target staff were sloppy amateurs” doesn’t understand the problem space. You try managing 300k endpoints w/ 100s of nodes in your supply chain in a growing business with tight margins. Read, learn, table top, stay humble.
– Earlier reporting was mentioning an RDP angle. is this refuted completely? Could Fazio have had an RDP connection to an internal host for some project-specific reason? This would explain exposure of AD credentials.
Would a HIPS (Host-based Intrusion Prevention System) have protected Target?
See http://www.privacyware.com/intrusion_prevention.html or http://www.comodo.com/business-enterprise/cesm3/index_v2.php
Target’s vulnerability came from two things:
1. They violated the principal of least privilege in managing credentials and privileges in their network
2. They made too much information publicly available (on their vendor sites). For example, stripping personal information including server names from an Excel spreadsheet is easy and a capability build into Excel.
I’ve seen how security is managed on Unix (Linux) shops: the admin password is passed around and even published on intranet sites. Windows is not the problem here; the reason Target was using Windows was because Microsoft was doing what they do best, which is making tools for business, with the best-available security tools – that can easily be misused. Microsoft publishes guidelines for best security practices though, that it looks like Target did not follow: http://technet.microsoft.com/en-us/library/dd277328.aspx
Why is the concept: “if you build the system so that an idiot can use it, then idiots will use it” so difficult for Windows defenders to understand?
Microsoft made their server product so easy to use that an idiot can use it – thus a lot of company CEOs realized they could hire idiots for administrators and save some money.
Microsoft eventually acknowledged this – and created UAC in Vista – but by then the idiot Windows system admins outnumbered the intelligent ones and forced them to disable it.
If your going to claim Microsoft has no blame here then your definitely putting yourself in one of those system admin camps – and not the flattering one.
As a former IT exec responsible for implementing PCIDSS security standards it always seemed to me like a futile exercise. Magnetic swipe credit card data are in clear text. It is always possible to intercept them before encryption (on the front end) or after decryption (in the back end). Once that’s done, game over.
The only real solution is end-to-end encryption using a smartcard with hardened, dedicated POS terminals. Wait, don’t they have something like that in Europe? Or in every country besides the US?
Could it be that the real purpose of PCIDSS is to shift the costs of an insecure system away from the card issuers and onto the processors, and to shift the remaining risk to us, the customers?
Oh YES! Lets spend billions of dollars switching to a system that’s already be cracked! Yeah! That’s the ticket, and my wife is Morgan Fairchild!
February 2008
http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-711.pdf
February 2010
http://www.bbc.co.uk/blogs/newsnight/susanwatts/2010/02/new_flaws_in_chip_and_pin_syst.html
” ”
http://news.bbc.co.uk/2/hi/science/nature/8511710.stm
September 2012
http://www.cl.cam.ac.uk/~rja14/Papers/unattack.pdf
Video summary of above report
http://www.bbc.co.uk/news/technology-19559124
“Magnetic swipe credit card data are in clear text. It is always possible to intercept them before encryption (on the front end) or after decryption (in the back end). ”
I have also implemented PCI DSS at my company. Our card swiper encrypts the stripe data immediately. The software only decrypts it long enough to send it to the card processor (using SSL of course) and we get back a token. The only thing stored on our disk is the token — no card number. So your word “always” is overstating the case.
I’m going to respond to this because it’s the heart of the problem:
“…You try managing 300k endpoints w/ 100s of nodes in your supply chain in a growing business with tight margins…”
I will point out that the CEO of Target is about the 3rd worst offender in exec compensation:
http://money.cnn.com/magazines/fortune/fortune500/2012/ceo-pay-ratios/
The guy made something like 30 million last year. for one year.
So cry me a river about tight margins, that is rubbish and baloney. Target’s exec staff is skimming an obscene amount of money off their business. They could have easily diverted some more of it to IT for security.
Ted, I was thinking the same thing with Target and with Fazio Mechanical.
I would HATE being the IT guy at Fazio and hear “no we cannot spend money on antivirus” yet looking on the net Fazio Mechanical is owned by a Ross E. Fazio who “During the summer of 1998, Mr. Fazio began meeting with a new investment advisor named Joseph Scarpo. At that time, Mr. Fazio’s business was continuing to grow and his annual personal income had increased to over $1 million. ” http://www.leagle.com/decision/In%20PACO%2020121212600
Anyway no one listens to IT.
I would guess Mr. Fazio will be listening pretty hard once Target sues him and removes that extra million.
Generally I deplore using fear to sell anything but some of these company owners have heads of solid bone, and stories like these are the only thing that penetrates.
‘…Our IT system and security measures are in full compliance with industry practices…’.
What part of ‘duhhhhhhh’ am I not understanding here?
I had to laugh when I read this. Late last year, I was touring a facility to contract for our DP/BC partner. The GM was so proud of the fact that they had and could mitigate “up to a 20 GB DDoS attack. I asked him if he had seen the latest industry numbers (he hadn’t) that had served warning that DDoS attacks were ramped up to the 150 Gb level (and obviously more now) which exceeded his entire bandwidth capability by 150%.
He proceeded to start stammering and needless to say, we looked elsewhere for a partner.
As Bullwinkle use to say “OOOHHH. Looks like I need a bigger hat.” (For those of you young enough to be my child, please reference the Rocky and Bullwinkle Show on YouTube or your favorite streaming service)
Heh!Heh! Love that moose & squirrel!! Boris and Natasha too!
I’m not sure if Boris and Natasha would be PC in today’s world…. 🙂
Mehaa!(Boris laff) – Dat’s why I’m liking him! 🙂
There’s lots out there who would say they could mitigate a 200Gb attack but would fall on their face if it actually happened to them. You have to look deeper than what the GM is showing you, my experience is that most of them aren’t very bright and don’t know much about what they are selling (or they wouldn’t have been promoted – Peter Principle at work)
As a GM of one company and President of another, I resemble that remark…. 🙂
My concern was that his paradigm of what a good platform was for IT breach mitigation didn’t wash with the then current reality. He was behind the 8 ball and didn’t know it. In IT security, that’s a death wish. I’ve learned the hard way to not let my cayman mouth over speak my hummingbird derriere.
There were other red flags as well, such as reliance on outdated audits (SOC 2 in a PCI DSS 3.0/HIPAA/HITECH customer mix.) I know not to rely on the acronym buzz for competency, but there was no admission at that point that these areas were even being addressed, even with OMNIBUS deadlines breathing down their neck.
Interestingly, their systems did crash later last year with a two day outage, despite an impressive physical plant. I never heard the reason, only customers leaving in search of a more secure BC partner.
I agree with you, Ted. Mitigating a 200 GB DDoS would test the mettle of many a IT Directors. But it wasn’t that long ago when 100 GB attacks were unheard of, and now the alarm is set for 400 GB.
Time for us all to bring our A game….
How do you mitigate a 400 gb DDOS attack? How do you even bring in that much bandwidth? I cannot imagine that much traffic pouring in to any IP Address.
Or maybe you control or influence a bunch of upstream routers so you can put in ACLs once you find out the profile of the attack?
I imagine if a 400 gb DDOS attack came my way, my ISP would notice it. Or maybe it would swamp my ISP. Without upstream help, how do you stand up to such an attack?
thanks
– Greg
Especially with the growth of high-speed broad-band in many markets; a good source for botnet marshaling. My ISP is boosting my service to gigabyte speed some time in the next year or so. Since they own a lot of fiber backbone, I assume they have the throughput to pull it off.
How to mitigate a 400 gb attack? Would our house experts ring in on this one. My little world wouldn’t ever be subjected to such an attack, well, let’s hope not anyway.
From my uninformed layman’s perspective (feel free to make fun of me):
There’s a limit to how big of a pipe that can built and supported properly, so in theory, the size of an attack and what damage it can possibly do is limited from a “space at the moment” problem. But that’s not really the problem, since a variety of techniques are used to deploy from different directions.
I
f I were a hacker, I would vary the frequency of the attack, meaning modulate it, so you have a hard time figuring out just how big it really is. Start small and amplify. Build the elephant slowly.
If you use an eat the elephant technique (a bite at a time) then mitigation becomes a matter of how long it takes to return to normal. Current numbers reported suggest these size of attacks can take 7 to 14 days to mitigate or at least bring under control. 400GB? could take a month.
You can use a black hole technique of flushing the whole system and then virtually bring it all back up elsewhere, then mitigate “at leisure,” but that likely only works on small to mid-range systems.
You can try a compression filter that starts to shrink the size of the data being accepted, depending on the data. Of course, they may be cruel and be shooting compressed data at you so their pipe is more effective.
If they use your systems list against you, friends don’t let friends suffer DDoS attacks, so the registry list they are using against you if you cooperate with each other can be mitigate at the 100’s of other sites. But that takes time as well.
Prayer is likely a good tool… But keep the Seppuku sword handy…
I will answer that one since one of the ventures I’m involved in is a small email provider and we get hit all the time by spammers and DDoS attacks by small spammer weenies who don’t have very large botnets.
For starters you got to understand that the very large, powerful botnets are very expensive. The supercriminals that run them cannot afford to use them on small fry and generally don’t – they have to keep a revenue stream coming in. My take on it is that the majority of botnets today are being paid for by criminals using them for revenge. Sometimes on people like me – I don’t think that there is any coincidence when I shut down an account that a foolish user has responded to a phishing email and handed over their password, that a few hours later I’ll get slapped with a small scale DDoS attack that lasts for a couple hours. The spammers really hate it when they spend a couple hours uploading a million pieces of mail to one of my servers and before 1% of it has been spooled back out, I’m deleting it and shutting down access.
But the majority of victims are other criminals, and nowhere more than porno. My experience is that the large botnet admins bread and butter comes from illigitimate operators paying them to attack other illigitimate operators.
So what all this boils down to is that it’s very rare for a really super sized botnet to be directed at a tiny little target. Most of the time the small botnets attack small targets and the large botnets attack larger targets.
Where I think the bulk of the victims of large attacks are is square in the illigitimate market. That is, if your running porno – you better be big, you better be able to mitigate a huge botnet attack. And the more illegal (child, etc.) stuff you have in your image and video mix, the more intense the attacks are gonna be. This is all about grabbing eyeballs. The people paying for porno are paying good money, the people paying for child porno are paying absolutely top dollar – and if your selling that stuff, if you can disable your competitor for the night – your going to increase your sales.
Remember that the child porno and even the regular porno operators all thieve each others images and videos all the time. They all have the same stuff. It’s not like they can sue someone for copyright violation when the stuff they are suing over is illegal. The consumers buying that stuff don’t care who sells it to them – they want it now, and if your site isn’t up when they want it, they go elsewhere.
If your makiing 50 grand a day with your dirty picture server, and you can pay a botnet operator 3 grand a day to take your biggest competitor offline – you may see another 20 grand a day coming in when he’s down. Of course, he’s going to go to his botnet providers and do the same to someone else, for the same reason, and they are going to go to their providers, and so on and so on. My guess is that the criminals running botnets are laughing their heads off at how much money they are extracting out of the dirty picture operators. And I will say right away that I have no idea how much money these dirty picture server operators make – but I’m quite sure it’s far more than I’ll ever see in my lifetime. And before anyone says anything, I do not see any distinction between the actual owner of the dirty picture server and the network provider who thinks they are “clean” because they aren’t doing the spamming or the dirty picture serving, they are just selling access to those who do. Trust me, the one time years ago that we had a guy show up with a server under his arm “to buy some colocate space from us” (that was back when we were selling colo space) within 15 minutes of him plugging the thing in, it became completely obvious what it was.
Anyway, getting back to the mitigation. There’s 2 prongs to any DDoS mitigation. The first is working with upstream providers. You identify the source IP’s of the botnet – pretty easy to do if you have a fast enough router, and the dirty picture operators have so much money they have the best gear on the planet – and you start chasing through the routes to you, contacting operators of those routes to erect network blocks.
Then at the same time, you start notifying the end-user ISPs that they have users with infected PC’s.
Most botnet operators are very sensitive to their botnets being destroyed, and they watch the numbers of nodes they have in their botnets when they commence an attack. If they see lossess of those nodes start to shoot up, that tells them the victim is busy notifying the ISP’s hosting their hijacked machines, and they will turn off the attack very quickly. I have found this to be far more effective for me to stop the sorts of low-level attacks that I get. Generally after kiling a hijacked account a spammer is using I’ll get about a 5-10% increase coming from DDoS attacks. I usually give it 24 hours to go away by itself and usually it does. But if it doesen’t, I start chasing down source IP’s and notifying admins in charge of those blocks and it stops almost immediately.
Ted,
May I plagiarize this? This is really good info to give to my IT guy who is very worried about his IT mitigation strategy/policy/procedures that has to be incorporated into our HIPAA/HITECH documents.
We’ll leave out the porno stuff, but it is, um, uplifting to know that most of us poor working class service providers aren’t worth the effort. And mitigating a small attack is many times, as you point out, a wait it out strategy until they get bored.
Sure, no problem, plagarize away! 🙂 A few other things to add, don’t give them the slightest edge in – make sure that any server with a public IP on it has a full firewall access list running on it and graph the Fsck out of everything on your network and look at the graphs every day. There’s not an interface on my network that isn’t queried by mrtg or rrd and I look at those graphs multiple times a day. I know what the normal and customary utilization is on everything and if I see anything funny I start digging. I’ve caught a lot of naughty people that way.
And, train your users! I just had a user call me earlier today – she got about 5 bounced emails from bogus addresses that she knew she hadn’t sent mails to – I dug into it and wouldn’t you know, a spammer out there had obtained her password and had sent some probe emails through the server, probably was getting ready to pump another 10 thousand mails through that server later tonight. Turns out she used that password for all her online stuff, after I changed it I told her as soon as we hang up, you go login to your bank and change the password there!
It is very very important to do this I cannot overemphasize how important it is to establish good relations with your users. I always tell anyone who will listen whenever I’m working on a PC that if they notice anything funny or out of the ordinary, ASK – and email only takes a few seconds. That’s the secret of handing lots of customers at once. You get on these little things before they blow up into big things, and if the users like you and your approachable and don’t talk down to them and all of that, over time you can teach them what’s right. Then later when they notice something funny they will tell you about it early, because they like talking to you.
Ted, all I can say is…amazing. Thanks.
– Greg
The plot thickens. Here’s a link to an article showing more financial institutions joing the suit against Target, including more details on Target’s response to warnings about their lack of security. http://www.courthousenews.com/2014/02/14/65371.htm
Yes it does, Chuck. There are those of us in the “kick the can around” club that are pegging an out of pocket number for Target that they probably can’t absorb. We didn’t bother to peg the brand damage, since trying to estimate that relies on knowing the hearts of consumers. Most love Target and will simply use cash for a while and forget about it, as long as they are made whole.
Think about this working list:
1. It’s normal for the guilty party to pay for 3 years of monitoring services for consumers. Even discounted, you could be looking at close to half a billion dollars.
2. There will be a piling on of class action suits in every state that Target serves. Settling a series led by the attorney general’s office for consumers and for banks. Just reissuing 70 million credit cards could approach a billion dollars in card cost, imprinting, mag encoding and postage.
3. The losses are likely in the 3 to 5 billion dollar range, depending on how many layers deep the list got “passed around” before it was detected. We are really conservative here, considering our office averaged over $100/card in fraud.
4. Attorney’s fees over the next 2 to 4 years will approach a billion dollars.
5. I cannot see this not having punitive damages attached to it. Several billion.
6. I haven’t a clue how bad the government case will be, both civil and potentially criminal. But with the current administration, even a settlement to educate and pay penalties for willful negligence could be several billion dollars.
All told, we estimate damage to Target in the 15 billion dollar range, perhaps more. this doesn’t include a shareholder suit that is sure to follow all this.
Currently, Target is carrying about a billion in cash. They earn after tax profits of about 3 billion per year, though that may be a high end number for future years. They could borrow perhaps 5 to 10 billion without terrible damage, but it would crimp their growth prospects, dividends, and ability to weather an increasing interest rate environment.
I don’t think a stock issue to bail them out is in the cards. Retailers who file for bankruptcy protection usually don’t fare well post emergence, there are of course exceptions and maybe Target can be one of them. But you would be looking at erasing a huge amount of market value, currently about 35 billion, which also hurts investors who will be compelled to jump on the class action suit as well, and given that the stock is currently tanking, this appears more likely by the day.
The thought that one fairly simple breach could bring down the number two retailer in the country would have been a fairy tale just a year ago, but here we are.
We guesstimate
Thanks for detailing that! As a layperson watching with interest as if this is all a true crime novel, I wouldn’t have been able to imagine those areas of penalization, nor the amounts.
So – indeed, there actually is a hell of a lot more at stake than just one beloved American icon of consumerism’s existence.
I got an email a few days ago that Kickstarer got hit(link to news story: http://www.nytimes.com/2014/02/17/business/kickstarter-acknowledges-a-breach-of-user-data.html?_r=00). They got ” user names, email addresses, mailing addresses, phone numbers and encrypted passwords” according to the article(and email I received), but….They use Amazon as a payment processor. If the hack to Kickstarter was a similar technique as Target’s breach, maybe Amazon is the end goal. That would have the potential to make Target look like the variety store of the town I grew up in, I’d think.
Of course, what do I know – just thinking as I type.
Thanks for the tip on the Kickstarter attack. Their email probably hit my junk filter. I changed my password immediately!
Not Happie,
No, you are on to something. It’s called oneupmanship. The hacker world is about prestige. And topping the Target hack would take a big boy. Amazon is the biggest of big boys. I would also imagine sphincter muscles are very tight over at Wal-mart.
But the game has changed a bit since retailers are on high alert and scrambling like mad not to be sucked down the same black hole as Target. Much harder to splash that big again.
But if decades of technology have taught us anything, the more we rely on it and the more advanced it becomes, the bigger the catastrophe when the system finally does fail.
And what we in IT seem to forget sometimes (my roots are actually in lean manufacturing) is that everything we do are a hybrid mix of human and mechanical/electronic processes. In the vast majority of process “autopsies” performed, it’s human inflicted, and sometimes a real nightmare.
“You’re an interesting species. And interesting mix. You’re capable of such beautiful dreams and such horrible nightmares.” – from the movie Contact
As a rank neophyte, just here watching and learning, I want to add something on a slightly lighter note. It’s related to the subject at hand, sort of. And it has a weird moral to it, again, sort of.
During the time period of the Target breach, I was shopping at a local Target here in San Francisco. I’d seen a few digital picture frames on Target’s website, where it clearly stated they were available online and at the store I visited.
When I asked the “sales technician”, or whatever they call themselves, about the frames, without taking a breath he said “we don’t carry those. Never have.” When I mentioned that Target’s website said differently, he said “well, they’re wrong.”
I assumed he was having a bad decade, so I went customer service, where a manager said pretty much the same thing. By the way, this Target had just opened a few days before, and there were few customers, so being busy wasn’t an excuse.
Anyway, another Target employee, who was sweeping the floor, overheard me and took it upon himself to check their computer inventory. Of course he found that they not only carried them, but that they were located about ten feet from where the first guy I’d talked to was stationed. I took a look at what they had, while Guy One glared at me and the guy helping me, apparently upset at being wrong.
The upshot was that by that time I was kinda creeped-out by the attitude, except for the helpful guy. So I left and went next door to a Best Buy, and got my frames.
The odd moral? If not for bad service by a couple of Target clerks, My credit card info might now be being traded in Slovakia or wherever.
If that had happened back the the Sam Walton days, that guy pushing the broom, if he were working for Walmart back then, would have been promoted to store manager!!
Ha ha! JC, you’re showing your age. I actually met the late Sam Walton in a business negotiation in the early 1980’s. There was never a more charming and ruthless negotiator. He understood leverage and he understood people. I am glad that I never met someone like him at poker tournaments.
Your experience reflects a change in dynamics over the years at Target. Target use to be the “cheap chic” store. Help was plentiful and pleasant. Unfortunately, they have succumbed to the pressures of Walmart and the internet.
But I agree, Sam Walton would have spot promoted the guy…
My brother’s boss and Sam were childhood friends; he used to stop by the helipad on the way to the local Walmart in San Antonio. He used to love flying helicopters to his stores to surprise the managers on site! I could tell he was a hell of a guy! 🙂
I loved your story, I have one related you might get a laugh out of. When the Target breach broke the news I reviewed our bank account and my wife had made a purchase during the time period, so I called our bank and they immediately canceled her card and reissued it. This was the day before we went on vacation. So for the next week she had no bankcard and every time we went anywhere on the vacation she had to ask me for my card to buy anything. Boy, was she pissed! However I have to thank the crackers as they probably saved me a good chunk of change on that vacation! 😉
Ted – your wife wasn’t fast enough. She should have convinced you to let HER hold the card and YOU ask for it when you needed it….. (Female Tip: Do NOT mention that fact to her – ever – even in jest. You guys may know IT but I knew how we women are wired.)