October 27, 2011

The IT director for an international hedge fund received the bad news in a phone call from a stranger: Chinese hackers were running amok on the fund’s network. Not seeing evidence of the claimed intrusion, and unsure about the credibility of the caller, the IT director fired off an email to a reporter.

“So do you think this is legit, or is the guy trying to scare us?” the IT director asked in an email to KrebsOnSecurity.com, agreeing to discuss the incident if he and his company were not named. “He has sent me the logs for the connections to the infected server. I checked the firewall and am not seeing any active connections.”

The call, from Hermes Bojaxhi of Columbia, Md. based threat intelligence firm Cyber Engineering Services Inc. (CyberESI), was indeed legit, and a follow-up investigation by the hedge fund revealed that at least 15 PCs within the financial services company were compromised and were sending proprietary information to the attackers.

CyberESI knew about the incident because it was monitoring several hacked, legitimate servers that the attackers were using to siphon data from multiple victims. Bojaxhi said the hedge fund notification was one of several he made that week to Fortune 500 companies that also had been hacked and were communicating with the same compromised servers.

And it wasn’t his first call to the hedge fund.

“On that particular victim, I tried to reach out to them a month prior, but I was handed off to an administrative assistant,” Bojaxhi said. “We had 25 [victim organizations] to call that day. But when they popped back up on the radar a month later, I tried again.”

The hedge fund incident illustrates the complexities of defending against and detecting targeted attacks, even when victims are alerted to the problem by an outside party.

Joe Drissel, founder and CEO for CyberESI, said too many companies think of cyberattacks as automated threats that can be blocked with the proper mix of hardware and software.

“So many firms are stuck in a paradigm of drive-bys, not targeted attacks,” Drissel said. “There seems to be a real disconnect with what’s really happening on a daily basis. We’re trying to fight an asymmetrical war in a symmetrical way, sort of like we’re British soldiers [in Revolutionary War], all walking in line and they’re picking us off one by one. By the time we turn around and aim, they’re already gone.”

None of the first three Trojans installed on the hedge fund’s computers were initially detected by any of the 42 anti-virus products bundled into the scanning tools at Virustotal.com.

Drissel said victims that his company notifies sometimes mistakenly think his firm is involved in the attack, or that they’re somehow joking.

“One guy laughed and said, ‘Thank you for watching out for our company,’ but he didn’t call us back,” Drissel said of a conversation with a victim earlier this year, declining to name the victim. “We watched [the attackers] exfiltrate weapons systems data for the Defense Department out of their systems, and ended up having to text the same guy a file stolen off their servers. Fifteen minutes later, we got a call back from him, and they unplugged their entire corporate network.”

Some say that the attacks CyberESI notifies companies about — often referred to as the advanced persistent threat (APT) —  are over-hyped, and that the malware and exploits used in these incursions usually aren’t that sophisticated. APT attacks also are frequently associated with targets in the U.S. government and companies in the defense industry.

But most APT attackers tend to be only as sophisticated as they need to be, which often isn’t too sophisticated, said Gavin Reid, senior manager of Cisco’s computer security incident response team. Speaking at a conference in Warsaw, Poland this week, Reid said successful APT attacks need not use zero-day software flaws.

“People will say, ‘Well, this attack wasn’t very advanced, so it can’t be APT’, but I will tell you the folks who are behind some of this stuff are not going to use cool zero-day stuff if they can go in the underground economy and say, ‘Hey, I need [access to] an infected machine in this organization,’ and pay $50 in Paypal in order to get that,” Reid said.

APT almost always involves social engineering, or tricking people into infecting their systems by disguising a malware-infected email attachment as something that is relevant to the recipient. Experts say this method usually works against targets if the attacker has enough resources, time, and solid information about his targets. In many ways, it is the “persistence” aspect of APT that makes it such a potent threat.

Drissel said any company that has valuable intellectual property can be a target.

“It’s not just the DoD and defense companies being targeted,” he said. “The truth is most companies have been compromised at one form or another.”


That was one of the key findings from an APT summit July 13 and 14, 2011 in Washington. The conference was put on by a large technology and security industry trade group called TechAmerica, and RSA, the security company that suffered a particularly high-profile APT intrusion earlier this year.

From the interim report published after that summit:

-Determined adversaries can always find exploits through people and in complex IT environments. It’s not realistic to keep adversaries out. Organizations should plan and act as though they have already been breached.

-Organizations should focus on closing the exposure window and limiting damage through efforts to compartmentalize systems, stop sensitive data egress and go back to the core principles of IT security such as ‘least privilege’ and ‘defense in depth.’

-The key is to know what digital assets are important to protect, where they reside, who has access to them and how to lock them down in the event of a breach.

The report also stressed the value of early detection of breaches, something that happens all too infrequently with APT intrusions. It stressed the importance of disrupting APT operations:

“The key is actively preserving, aggregating and reviewing data to detect a potential intrusion but also for post-event forensics. Don’t underestimate the power of disruption. Damage from APTs can be minimized or prevented by simply interrupting attackers’ work flow at multiple points. Organizations should strive for a disruptive approach to defense in order to match the rapidly evolving threat environment.”

Cisco’s Gavin Reid said organizations that don’t have a good record of internal network activity stretching back months or even years have little chance of understanding the breadth of an APT attack after it occurs.

“Without that information, there is very little victims can piece together to understand what came in, what went out, and who else was involved,” Reid said.

But Reid cautioned that logging is not enough, and the security industry has sold many companies on a lie: That automation and network logging solutions can take the place of skilled staff in detecting intrusions.

“One of the areas where we’ve failed as a security community is that we’ve got an over-reliance on automation,” Reid said. “We’ve sold this idea that we can automate it, in a way that will not only help your security staff identify threats, but that you can cut your staff down because these technologies are going to do the work of a lot of people. That has failed. We’re still stuck with [the reality that] you need smart people who understand computer, applications and networks, and a logging solution becomes a tool they can use to identify some of these things. Hopefully this has been a little bit of a wake-up call, and we can start looking at things a little differently and start putting people back into the equation.”


It is one thing for an APT victim organization to disrupt the flow of information from its own networks to the control networks run by the attackers. But is it anyone’s job to disrupt the infrastructure used to attack multiple corporations simultaneously? Does it even make sense for an organization with specific skill sets attuned to APT attacks to do this?

Drissel said CyberESI and other competitors who notify companies hit by APT attacks have lobbied the U.S. government for the authority to take more aggressive steps to target APT infrastructure, with little success.

“What [the U.S. government needs] to do is to allow us the latitude to go after the attackers,” said Drissel, former acting section chief of the intrusions section at the Defense Computer Forensics Lab, housed at the Department of Defense’s Cyber Crime Center in Linthicum, Md. “We all came out of the Department of Defense. All of us worked in some capacity for the federal government, and we do know where the line is that we can’t cross. We can stop them, but we don’t. We can cut them off, we just don’t.”

It’s not clear how far CyberESI or even the federal government would go to shut down command and control networks being used for these attacks, or whether that approach would be effective and desirable. I have interviewed several experts who told me that although the FBI regularly alerts companies infiltrated by APT attacks, it usually does nothing to disturb the attacker’s infrastructure for fear that disrupting it would eliminate visibility into future victims.

CyberESI requested that I not publicize the domain names, Internet addresses or other data included in the report that they sent to the hedge fund; the company said that publishing the location data would likely cause the attackers to alter their attack infrastructure, and potentially diminish the firm’s ability to identity and alert new victims.

Updated, 1:24 p.m.. ET: Fixed misspelling of Drissel’s name.

30 thoughts on “Chasing APT: Persistence Pays Off

  1. Robert Scroggins

    All blame aside, why in the world don’t these employees tell IT/someone when they click on an email attachment that doesn’t do what it is purported to do? Even if they shouldn’t have clicked on it, this would still be a good point to investigate things, rather than hiding/forgetting about it until it can’t be ignored!

    1. Greg Sergienko

      Hi, Robert,
      That would be nice, but I can see why it doesn’t happen.
      First, some of the things do do what they’re supposed to do, it’s just that they’ve got malware added. So, even after the click, it’s not obvious to the user that something bad has happened.
      Second, it’s really tough for people to admit they’ve done something they shouldn’t.

      And, even if it does happen–that they call IT–IT may blow it off. There are plenty of organizations where a message to IT help goes off to bit heaven and is never seen again.

      1. Uzzi

        Maybe it’s time for some folks to start reading the basics reaching back to 1992:

        Information security policy

        Information security means protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, perusal, inspection, recording or destruction.[…]

        Basic principles – Key concepts

        […] In 1992 and revised in 2002 the OECD’s Guidelines for the Security of Information Systems and Networks proposed the nine generally accepted principles: Awareness, Responsibility, Response, Ethics, Democracy, Risk Assessment, Security Design and Implementation, Security Management, and Reassessment. Building upon those, in 2004 the NIST’s Engineering Principles for Information Technology Security proposed 33 principles. From each of these derived guidelines and practices.

        In 2002, Donn Parker proposed an alternative model for the classic CIA triad that he called the six atomic elements of information. The elements are confidentiality, possession, integrity, authenticity, availability, and utility. […]

        Cyber security standards

        […] Cyber security is important in order to guard against identity theft. […] One of the most widely used security standards today is ISO/IEC 27002 which started in 1995. […] The National Institute of Standards and Technology (NIST) has released several special publications addressing cyber security. […] The International Society of Automation (ISA) developed cyber security standards for industrial automation control systems (IACS) that are broadly applicable across manufacturing industries. The series of ISA industrial cyber security standards are known as ISA-99 and are being expanded to address new areas of concern.

        From Wikipedia, the free encyclopedia



        Special Publication 800-12: The NIST Handbook to Computer Security:

        Generally Accepted Principles and Practices for Securing Information Technology Systems:

        Information Technology (IT) Security Essential Body of Knowledge (EBK):
        A Competency and Functional Framework for IT Security Workforce Development (Office of Cybersecurity and Communications, National Cyber Security Division, United States Department of Homeland Security):

        Cybersecurity – The growing number of attacks on our cyber networks has become, […] “one of the most serious economic and national security threats our nation faces.” The Department of Homeland Security plays an important role in countering these threats. We’re […] partnering with the private sector, and empowering the general public to create a safe, secure, and resilient cyber environment, and promote cybersecurity knowledge and innovation.
        U.S. Department of Homeland Security > Cybersecurity

        Governing for Enterprise Security Implementation Guide:

      2. prairie_sailor

        One thing you missed. The user who opened the e-mail may not recognize that something is amiss. A huge number of people using computers are not comfortable with them and when things go wrong they don’t recognize it – or they may just chalk it up to the way things are.

    2. LonerVamp

      To add to what Greg said, when you call in an investigation or IT, quite often that will lead to changed things, downtime, maybe even a system rebuild which gets in the way of employee productivity. Sadly, that happens a lot where employees don’t involve IT/security until they’ve hit a brick wall.

      Other times, the file may work and display data, only it’s not useful or looks like spam. Not every malicious file or web page will throw an error repeatedly and not load or crash the involved app.

      Trying to actually *prevent* issues like this is an eventual losing game. You should try your best, but you have to assume something someday will get through.

    3. Conrad Longmore

      “..why in the world don’t these employees tell IT/someone when they click on an email attachment that doesn’t do what it is purported to do?”

      This is a critical issue for any large organisation. Employees should report things that they think are suspicious without fear of being made to feel stupid or if they are wasting IT’s time.. and they should be thanked for their trouble in reporting any issue regardless to encourage them to do it again.

      I would much sooner deal with false alarms (which are easily and quickly dealt with) than to end up with a real problem that could have been avoided..

    4. John

      The reason this happens is because IT does not train the staff to tell them about this or what that could mean. It’s not the users fault if they have not been told about it. To many times in IT we think because we know everyone knows.

  2. bt

    The article ends saying “we can stop them.” I take it that is a reference private firms in cooperation with the govt being able to do whatever they wish with infrastructure or some variant?


  3. Mark Giles

    I believe the phrase “We can stop them” means that they have the technical ability to stop them, but that they do not take that action due to legal restrictions. I make that interpretation based on the context of the whole article.

  4. JCitizen

    This article confirms what I’m seeing in my clients. One of them is a target for sure. We’ve been battling the infiltration for months and they just keep coming back. Unfortunately my client can’t start totally from scratch. There are economic limitations to radial changes like that.

    If the government isn’t going to allow us to go on offense – when are they going to go to bat for us? This is part of the problem with the world economy, US innovation cannot survive without cooperation from our top level law enforcement and/or military mechanisms!

  5. Richard E.

    Joe Drissel is spot on when he says:
    “.. most APT attackers tend to be only as sophisticated as they need to be, which often isn’t too sophisticated, ..”

    Which is why I think that APA or “Adaptive Persistent Adversaries” is a more appropriate term.

    And for Robert Scroggins: I recently got a copy of an email with an attachment called “Report to Leaders G20 “.
    When you click on it, it actually is the report to the G20 leaders and absolutely nothing out of the ordinary happens when you open or close the file. But in the background your PC starts to cumminicate with a server in Indonesia.

  6. Christian

    giving ex DOD members more control over the net seems like a neat idea!
    what could possibly go wrong? 🙂

  7. george

    The solution against APT is simple:

    1. Cut any access to outside networks to anyone in Financial Department (because trojans like Zeus or SpyEye can be used to steal from company accounts)
    2. Cut any access to outside networks to anyone in Technical and Engineering Department (because various trojans can be set to steal Intelectual Property or to sabotage existing installations. Also for those guys USB ports disabled because of Conficker and Stuxnet.
    3. Cut any access to outside networks for anyone in the organization with a management position (because again, trojans can be set to steal sensitive information for a little insider trading)
    4. Cut any access to outside for guys in HR. (since poking through personnel files can lead to identity theft or to targeted/personalized attacks.
    5. Cut any access to outside to everyone else in the organization (since their PCs could be used as proxies to access one of the systems from 1 to 4).
    And of course, no Email – it is just too dangerous.

    1. Darren Reid

      It is funny…but actually, it *is* the answer. Everyone gets two machines: their internet/email/communications PC, and a seperate work pc for their actual must-remain-secure work. The second PC is connected to an intranet with no internet access.

      1. Huh?

        We are actually demoing a product by Integrity Global Security (IGS) that can do just that. 1 box, 2 VMs protected at boot and they partner with Dell. The workstations are also on 2 separate networks with the protected one secured through an encrypted gateway to a secure network. We wanted to see if they can help us provide a secure solution for the various infrastructure operators (water, power, etc). The problem is these various infrastructure owners don’t want to pay for it.

        1. Nick P

          I’ve actually mentioned & recommended that product a few times. I think it’s still vulnerable to covert channels regardless of EAL6+ certification if it runs on a standard Intel Core processor. The reason is that the information flow models used in the certification probably didn’t model the effects of hyperthreading or a shared cache, which are relatively high bandwidth covert channels. Other possible problem spots on COTS hardware are defects in trusted boot process & malicious DMA.

          The main problem I have with Green Hill’s solution is that they provide very little specifics on how they deal with these kinds of attacks. I still have the original INTEGRITY Workstation brochure, which covered it a bit. There’s lots of marketing prose for the new products, but little independent evaluation. Let Invisible Things Lab (makers of competing QubesOS) have a few months to analyze & pentest it. If they say it’s good, I recommend it en masse. More likely, they’re going to say, “We gave them the list of vulnerabilities we found. We will be presenting them at the next Black Hat.”

          I simply don’t trust obscurity & “we tested it internally so you should trust it.” These historically haven’t produced secure solutions. Green Hill’s certainly makes high quality software with their PHASE process, but how does this new company build & test products? What is their quality & equivalent EAL? We don’t know because there’s no trustworthy independent evaluation happening. So, give us something to go on rather than [paraphrased] “we have a piece of closed source software that magically turns an inherently insecure x86 box into secure virtualization solution with total isolation & information flow control.”

          – Nick P
          normally found on schneier.com

    2. LonerVamp

      @george : Yes, those are solutions, but they aren’t realistic for how technology and use of the Internet is going. You’ll really only get away with that in small groups or in places that already have it, like SIPRNet. Or where there is a life and death reason to do so, and has executive level backing.

      Otherwise the world is moving in a more-connected direction.

      And even if you cut access, you’re just back to “old school” espionage techniques with turning assets or planting assets physically. That’s really all APT is: today’s espionage.

      Thirty years ago, espionage occurred, but that doesn’t mean private organizations had the go-ahead to “take offensive measures.”

    3. Jim Dandy

      People look at me funny when I say we should just give everyone iPads so they can play farmville. Than we could cut them off from the internet.
      It would be cheaper and more secure.

  8. Luiz Firmino

    APT = Target Phishing + Social engineering + Lack of information security awareness + IT assets misconfigured.

    The origin of word Attack is Attach, c.1600, from Fr. attaquer (16c.), from Florentine attaccare (battaglia) “join (battle),” thus the word is a doublet of attach, which was also used 15c.-17c. in the sense now reserved to attack. It is interesting because the APT (Advanced Persistent Threat) attacks are based on malicious attachments and are not highly advanced and sophisticated. Attackers take advantage of organization making simple mistakes. They call the attack as an APT because the organizations does not know what happened but send emails to targets with malicious attachments, monitor their treatment and escalate privileges is just an step of the Social Engineering Pentest using emails.

    The most common way for cyber attackers to gain access to an organization’s network is through spear phishing, in which the attacker sends an email that looks like it came from a trusted source, when opened, installs that will enable them to exploit the target’s network. The compromised system continues to work without any evidence that the network is compromised. Information is gathered for future (and persistence) attacks and to escalate privileges.

    The attackers use new designed and customized malware to circumvent most common defenses and focus their tools and techniques on a specific target or just evading techniques, breaking the trojan file into multiple pieces and zip them as single file, changing the content of the trojan using hex editor and also change the checksum and encrypt the file and change trojan´s syntax to convert an executable file to VB script or Office files.

    Create a dropper, which is a part in a trojanized packet that installs the malware on the target systems and create a wrapper using tools to install trojan on the victim´s computer with an innocent looking extension (.pdf, .doc, etc) is not necessarily advanced. When the victim runs the wrapped file, it first installs the trojan in the background and then runs the wrapping application in the foreground. Trojan server is installed on the victim´s machine, which opens a port for attacker to connect. The client is installed on the attacker´s machine, which is used to launch a command shell on the victim´s machine.

    Command shell trojans gives remote control of a command shell on a victim´s machine. The trojan looks for using the victim´s machine for illegal purposes, such as to scan, flood, infiltrate other machines, steal information such as passwords and security codes using key loggers, replace OS critical files, download other malwares, record screenshots, audio and video, disable the local anti-virus and the personal firewall and infect victim´s computer as a proxy server for relaying attacks and use that machine as a convert channel.

    Compromised machines become springboards to infect other machines and the entire network. As the network becomes infected, backdoors are installed to gain further access to the company’s infrastructure. With the proper credentials in hand, the attacker controls the compromised system. As the infiltration continues, the victim’s network passwords are grabbed, email and files are stolen, and even the network topology itself is uncovered. The attack continues to expand its reach in the network into more sensitive systems via the Botnet master’s Command and Control infrastructure placing more and more critical data; such as financial data, marketing plans, and research and development information, at risk. With one compromised system, an attacker can establish full control over much of the corporate, enterprise, or critical network infrastructure.

    Reconnaissance, scanning, gaining access, maintaining access and clearing track are basic steps for any attacker or pentester and hiding files, cracking passwords, escalating privileges, executing applications, covering tracks are not new either advance and APT is just a new scary thing to say.

    Luiz Firmino, CISSP, CISM, CRISC, C|CISO

    1. Nick P

      Darn, Luiz, you beat me to it. This whole “Advanced” Persistent Threat monkier has been aggravating me because it’s applied to what’s essentially basic hacking. Even back in the NT days, we used many techniques to compromise systems. Unlike the script kiddies, real hackers were very effective problem solvers & would use any series of steps they needed to achieve the goal(s). There’s nothing advanced about that.

      They are just using APT to drum up sales in the INFOSEC and news markets. Companies seem to be using the term to hide that their software, systems, procedures, etc. can’t stop the average high school grade hacker. This trend needs to stop. Companies need to re-evaluate how they handle security & do something that actually works. They can start by applying some of the recommendations that their IT security staff have been giving them over the years.

      And only truly advanced attacks should be labeled as such. In modern times, I think I’d have only given that label when the encrypted, P2P, DNS shifting C&C systems appeared. That was pretty advanced, at the time. Advanced system attacks include using covert channels, processor errata, BIOS/OS combined infection, firewire (at the time) & BootJacker. (That was an awesome piece of work, actually, so I linked it.)


  9. AlphaCentauri

    Surely someone has thought of crafting some really nasty malware and putting it on an attractive target network with a tempting file name like, “Strategies for dealing with Chinese competition.”

    Where are our vandals when we need them?

    1. F-3000

      What I wonder, is that where are those greyhats who would breach into peoples’ PCs only to install a “malware” which only laughs at the user with a popup whenever the PC is booted up, by notifying the user about what he did to get the infection. With all the nasty stuff as “open source”, it wouldn’t require too much efford.

      I certainly would do something like that, if I had the knowledge.

  10. KJ

    I can easily speculate that the list of compromised sites in the last blog post that caused skepticism because of lack of source and methodology originated with someone at CyberESI.

  11. Neil Roiter

    Re APT:

    “If anything in this life is certain, if history has taught us anything, it is that you can kill anyone. “

  12. Neil Roiter

    Re APT:

    “If anything in this life is certain, if history has taught us anything, it is that you can kill anyone. ”

    –Michael Corleone
    Godfather Part II

  13. John Thompson

    As far as why people don’t report things to IT, I have found one or two explanations. The first is they don’t want to admit that they do not know what they are doing. The second is that they are worried about getting in trouble for clicking or opening something that they should not have. I have had users try and lie and say they didn’t click on anything when to AV logs and the other logs clearly shows that they did.

    I think it is really getting to the point where companies will have to block access to any type of social networking or external mail provider (gmail, yahoo mail, ect). People need to realize that their work computer is NOT their home computer and quit trying to treat them as such.

    There are also things that you can do as long as management is willing to put their foot down with the users and tell them this IS how it is going to be. For us we don’t allow the users to have admin rights, we have blocks setup to prevent .exe files from running from certain directories, and we have blocks setup at the firewall to prevent anyone with access to finicial information from downloading any type of executible file.

  14. Al Macintyre

    Better methods of notifications are needed.

    The notification should include whatever government agencies regulate the industry the company is located in, if that can be determined.

    SEC, The FED, Dept of the Treasury, FBI, Secret Service.
    A simple notification.
    We are working on this security breach. We have found evidence that other companies have been hit in a similar way. We have attempted to notify those companies. Some of them are not taking us seriously.
    Here is a list of all companies we believe have recently been hit by these cyber security breaches. You may wish to inquire whether they are doing anything about this situation, whether they have correctly reported implications to regulators.
    We can provide additional info on the details we know by victim company, as desired.

  15. Al Macintyre

    While we want security companies to be allowed to be more pro-active in defending our nation, we also see what can happen when the good missions are taken over by cost cutting. The anti-piracy regime is now taking down many people who are not pirates, with no judicial oversight to provide meaningful proof of accusations, and right of accused to face one’s accuser evidence in court.

    This business of notifying 25 victim companies a day, and spending a lot of time with individual non-believers, that has an expense for which the security company ought to be reimbursed.

Comments are closed.