The IT director for an international hedge fund received the bad news in a phone call from a stranger: Chinese hackers were running amok on the fund’s network. Not seeing evidence of the claimed intrusion, and unsure about the credibility of the caller, the IT director fired off an email to a reporter.

“So do you think this is legit, or is the guy trying to scare us?” the IT director asked in an email to KrebsOnSecurity.com, agreeing to discuss the incident if he and his company were not named. “He has sent me the logs for the connections to the infected server. I checked the firewall and am not seeing any active connections.”

The call, from Hermes Bojaxhi of Columbia, Md. based threat intelligence firm Cyber Engineering Services Inc. (CyberESI), was indeed legit, and a follow-up investigation by the hedge fund revealed that at least 15 PCs within the financial services company were compromised and were sending proprietary information to the attackers.

CyberESI knew about the incident because it was monitoring several hacked, legitimate servers that the attackers were using to siphon data from multiple victims. Bojaxhi said the hedge fund notification was one of several he made that week to Fortune 500 companies that also had been hacked and were communicating with the same compromised servers.

And it wasn’t his first call to the hedge fund.

“On that particular victim, I tried to reach out to them a month prior, but I was handed off to an administrative assistant,” Bojaxhi said. “We had 25 [victim organizations] to call that day. But when they popped back up on the radar a month later, I tried again.”

The hedge fund incident illustrates the complexities of defending against and detecting targeted attacks, even when victims are alerted to the problem by an outside party.

Joe Drissel, founder and CEO for CyberESI, said too many companies think of cyberattacks as automated threats that can be blocked with the proper mix of hardware and software.

“So many firms are stuck in a paradigm of drive-bys, not targeted attacks,” Drissel said. “There seems to be a real disconnect with what’s really happening on a daily basis. We’re trying to fight an asymmetrical war in a symmetrical way, sort of like we’re British soldiers [in Revolutionary War], all walking in line and they’re picking us off one by one. By the time we turn around and aim, they’re already gone.”

None of the first three Trojans installed on the hedge fund’s computers were initially detected by any of the 42 anti-virus products bundled into the scanning tools at Virustotal.com.

Drissel said victims that his company notifies sometimes mistakenly think his firm is involved in the attack, or that they’re somehow joking.

“One guy laughed and said, ‘Thank you for watching out for our company,’ but he didn’t call us back,” Drissel said of a conversation with a victim earlier this year, declining to name the victim. “We watched [the attackers] exfiltrate weapons systems data for the Defense Department out of their systems, and ended up having to text the same guy a file stolen off their servers. Fifteen minutes later, we got a call back from him, and they unplugged their entire corporate network.”

Some say that the attacks CyberESI notifies companies about — often referred to as the advanced persistent threat (APT) –  are over-hyped, and that the malware and exploits used in these incursions usually aren’t that sophisticated. APT attacks also are frequently associated with targets in the U.S. government and companies in the defense industry.

But most APT attackers tend to be only as sophisticated as they need to be, which often isn’t too sophisticated, said Gavin Reid, senior manager of Cisco’s computer security incident response team. Speaking at a conference in Warsaw, Poland this week, Reid said successful APT attacks need not use zero-day software flaws.

“People will say, ‘Well, this attack wasn’t very advanced, so it can’t be APT’, but I will tell you the folks who are behind some of this stuff are not going to use cool zero-day stuff if they can go in the underground economy and say, ‘Hey, I need [access to] an infected machine in this organization,’ and pay $50 in Paypal in order to get that,” Reid said.

APT almost always involves social engineering, or tricking people into infecting their systems by disguising a malware-infected email attachment as something that is relevant to the recipient. Experts say this method usually works against targets if the attacker has enough resources, time, and solid information about his targets. In many ways, it is the “persistence” aspect of APT that makes it such a potent threat.

Drissel said any company that has valuable intellectual property can be a target.

“It’s not just the DoD and defense companies being targeted,” he said. “The truth is most companies have been compromised at one form or another.”

ASSUME YOU HAVE BEEN BREACHED

That was one of the key findings from an APT summit July 13 and 14, 2011 in Washington. The conference was put on by a large technology and security industry trade group called TechAmerica, and RSA, the security company that suffered a particularly high-profile APT intrusion earlier this year.

From the interim report published after that summit:

-Determined adversaries can always find exploits through people and in complex IT environments. It’s not realistic to keep adversaries out. Organizations should plan and act as though they have already been breached.

-Organizations should focus on closing the exposure window and limiting damage through efforts to compartmentalize systems, stop sensitive data egress and go back to the core principles of IT security such as ‘least privilege’ and ‘defense in depth.’

-The key is to know what digital assets are important to protect, where they reside, who has access to them and how to lock them down in the event of a breach.

The report also stressed the value of early detection of breaches, something that happens all too infrequently with APT intrusions. It stressed the importance of disrupting APT operations:

“The key is actively preserving, aggregating and reviewing data to detect a potential intrusion but also for post-event forensics. Don’t underestimate the power of disruption. Damage from APTs can be minimized or prevented by simply interrupting attackers’ work flow at multiple points. Organizations should strive for a disruptive approach to defense in order to match the rapidly evolving threat environment.”

Cisco’s Gavin Reid said organizations that don’t have a good record of internal network activity stretching back months or even years have little chance of understanding the breadth of an APT attack after it occurs.

“Without that information, there is very little victims can piece together to understand what came in, what went out, and who else was involved,” Reid said.

But Reid cautioned that logging is not enough, and the security industry has sold many companies on a lie: That automation and network logging solutions can take the place of skilled staff in detecting intrusions.

“One of the areas where we’ve failed as a security community is that we’ve got an over-reliance on automation,” Reid said. “We’ve sold this idea that we can automate it, in a way that will not only help your security staff identify threats, but that you can cut your staff down because these technologies are going to do the work of a lot of people. That has failed. We’re still stuck with [the reality that] you need smart people who understand computer, applications and networks, and a logging solution becomes a tool they can use to identify some of these things. Hopefully this has been a little bit of a wake-up call, and we can start looking at things a little differently and start putting people back into the equation.”

OFFENSE AS A GOOD DEFENSE?

It is one thing for an APT victim organization to disrupt the flow of information from its own networks to the control networks run by the attackers. But is it anyone’s job to disrupt the infrastructure used to attack multiple corporations simultaneously? Does it even make sense for an organization with specific skill sets attuned to APT attacks to do this?

Drissel said CyberESI and other competitors who notify companies hit by APT attacks have lobbied the U.S. government for the authority to take more aggressive steps to target APT infrastructure, with little success.

“What [the U.S. government needs] to do is to allow us the latitude to go after the attackers,” said Drissel, former acting section chief of the intrusions section at the Defense Computer Forensics Lab, housed at the Department of Defense’s Cyber Crime Center in Linthicum, Md. “We all came out of the Department of Defense. All of us worked in some capacity for the federal government, and we do know where the line is that we can’t cross. We can stop them, but we don’t. We can cut them off, we just don’t.”

It’s not clear how far CyberESI or even the federal government would go to shut down command and control networks being used for these attacks, or whether that approach would be effective and desirable. I have interviewed several experts who told me that although the FBI regularly alerts companies infiltrated by APT attacks, it usually does nothing to disturb the attacker’s infrastructure for fear that disrupting it would eliminate visibility into future victims.

CyberESI requested that I not publicize the domain names, Internet addresses or other data included in the report that they sent to the hedge fund; the company said that publishing the location data would likely cause the attackers to alter their attack infrastructure, and potentially diminish the firm’s ability to identity and alert new victims.

Updated, 1:24 p.m.. ET: Fixed misspelling of Drissel’s name.

A closely-watched court battle over how far commercial banks need to go to protect their customers from cyber theft is nearing an end. Experts said the decision recommended by a magistrate last week — if adopted by a U.S. district court in Maine — will make it more difficult for other victim businesses to challenge the effectiveness of security measures employed by their banks.

In May 2009, Sanford, Maine based Patco Construction Co. filed suit against Ocean Bank, a division of Bridgeport, Conn. based People’s United Bank. Pacto used online banking primarily to make weekly payroll payments. Patco said cyber thieves used the ZeuS trojan to steal its online banking credentials, and then heisted $588,000 in batches of fraudulent automated clearing house (ACH) transfers over a period of seven days.

In the weeks following the incident, Ocean Bank managed to block or claw back $243,406 of the fraudulent transfers, leaving Patco with a net loss of $345,445. Because the available funds in Patco’s account were less than the total fraudulent withdrawals, the bank drew $223,237 on Patco’s line of credit to cover the transfers. Patco ended up paying interest on that amount to avoid defaulting on its loans.

Patco sued to recover its losses, arguing in part that Ocean Bank failed to live up to the terms of its contract when it allowed customers to log in to accounts using little more than a user name and password. On May 27, a magistrate recommended that the court make Patco the loser by denying Pacto’s motion for summary judgment and granting the bank’s motion.

David Navetta, a founding partner of the Information Law Group, said that Patco has about another week to dispute the magistrate’s recommendations, but that it is unlikely that the judge overseeing the case will overturn the magistrate’s findings.

Navetta said the magistrate considered the legal issues and propounded an analysis of what constitutes “commercially reasonable” security.

“Many security law commentators, myself included, have long held that reasonable security does not mean bullet-proof security, and that companies need not be at the cutting edge of security to avoid liability,” Navetta said. “The court explicitly recognizes this concept, and I think that is a good thing.”

But Avivah Litan, a fraud and bank security analyst at Gartner, took strong exception to the way the magistrate arrived at the recommended decision, calling it “an outrage.”

“In my opinion, this is frankly an egregious injustice against small U.S. businesses,” Litan said. “It is also a complete failure of the bank regulatory system in the United States, which should come as no surprise, given the history of the regulators in the 21st century.”

The Technology

Ocean Bank relied on service provider Jack Henry to process bank-to-bank transfers, and it selected an authentication process that required customers to log in with a company ID, user ID and password. Customers also were asked to provide answers to three “challenge questions” that would be asked if the system scored a transaction as “high risk.”

The Jack Henry product came with a risk scoring system developed by RSA‘s Cyota, which rates the riskiness of transactions by using several factors, such as the location of a user’s Internet address, when and how often the user logs in, and how the customer navigates the site. Challenge questions were prompted when the risk score for a transaction exceeded 750 on a scale of zero to 1,000 (RSA considers transactions generating risk scores in excess of 750 to be high-risk). Ocean bank also kept track of customer “device IDs,” an amalgamation of attributes from the customer’s PC that could be used to create a unique fingerprint for that machine.

Until 2008, Ocean Bank set its dollar amount threshold — transfer amounts that would automatically require the answer to a challenge questions regardless of the Cyota fraud score — at $100,000. But in July 2008, the bank lowered that threshold to $1. The bank told the court that it did so to enhance security following ACH fraud at the bank that targeted low-dollar amount transactions. After the change, customers were forced to answer a challenge question whenever they used the bank’s system.

The Analysis

Patco’s security expert, Sari Green of Portland, Me. based Sage Data Security, told the court that by setting challenge questions to be asked on every transaction, the bank greatly increased the risk that a fraudster equipped with a banking Trojan would be able to compromise the answers to a customer’s challenge questions. Patco also argued that because the questions were triggered on every transaction regardless of the scoring of the transaction, that system did not provide any additional security.

Navetta said the magistrate considered the question of whether Ocean Bank’s security was sufficient. The magistrate analyzed whether the bank’s security satisfied “multi-factor authentication” guidelines by incorporating at least two of three checks: Something the user knows (such as a password), something the user has (such as the passcode generated by a one-time token); and something the user is, such as a biometric identifier. (Those guidelines were established in 2005 by banking regulators at the Federal Financial Institutions Examination Council (FFIEC).

Navetta said the magistrate accepted the bank’s argument that the password-based scheme used by the bank was multi-factor as described in the FFIEC. “To some degree the court acknowledged that the bank’s security could have been better,” Navetta said. “Even so, it was technically multi-factor as described in the FFIEC guidance in the court’s opinion, and ‘the best’ was not necessary.”

The magistrate wrote that while the guidelines say two out of three of those factors should be incorporated, it says nothing about how banks must respond when one of those factors detects an anomaly. More importantly, the magistrate accepted the bank’s assertion that a device ID satisfied the “something the user has” requirement.

The magistrate was unswayed by evidence presented by Patco’s lawyers that modern malware threats like ZeuS can modify content in the victim’s browser (and thus prompt users for the answers to all of their secret questions). ZeuS also allows attackers to tunnel their communications through a victim’s own PC and browser, an attack method that can negate the value of a device ID as a second factor. Navetta said Patco’s main theory concerning the weakness of the bank’s security was that the lower dollar threshold set by the bank made customers easier prey for predators like the ZeuS Trojan, but that the magistrate was unconvinced by that argument because Patco did not have actual forensic evidence that a keystroke logger was the culprit. The magistrate said Patco erred by “having irreparably altered the evidence on its hard drives by running scans on its computers and continuing to use them prior to making proper forensic copies.”

Avivah Litan said the methods used by Jack Henry to support Ocean Bank were not appropriate to the risks associated with online business banking in 2009.

“Zeus, browser-based Trojans and other modern-day threats are known by anyone following online banking security to circumvent all the methods that were being used at the time by the bank and its processor,” Litan said. “Unfortunately, the 2005 FFIEC guidance referred to examples of relatively crude online theft techniques that were commonplace in 2004 and 2005. The cybercriminal of 2011 has long ago bypassed and surpassed those old techniques.”

The FFIEC was on the verge of releasing updated guidance at the end of last year to clarify the new and stronger types of multi-layered defenses required in 2011.  Litan said those updates were expected to explain that the examples of strong online banking security measures which they listed in 2005 have been rendered useless and obsolete by next-generation cybercrime techniques.

“It’s truly disappointing that the much-needed update was never issued, no doubt because of internal politics and disagreements among the regulatory agencies,” she said. “The regulators should not leave these matters in judges’ hands to decide and should protect U.S. businesses from bank shortcomings that compromise the safety and security of their accounts,  just as consumers are protected under Regulation E. In my opinion, this judge did not correctly interpret the 2005 FFIEC authentication guidance.”

Patco co-owner Mark Patterson said the company hasn’t yet decided whether to appeal.

“The one thing the judge mentioned in his decision is that there is basically zero case law on [question of what constitutes reasonable security] for the banks,” Patterson said. “Not anymore. That’s why we’re concerned this could have national implications. Tons of small businesses continue to be at a huge risk for this type of thing happening to them.”

The magistrate’s recommendations are by no means a done deal, even if the district court adopts them. The decision could be appealed, possibly all the way to the US Supreme Court. Interested parties could present further legal argument by filing amicus curiae (friend of the court) briefs at any time during the appeal process.

A copy of the recommended decision is available here (PDF).

KrebsOnSecurity will continue to follow this case and to bring you updates on new developments as they happen. Stay tuned.

This is the second installment of a multi-part series examining the tools and tactics used by attackers in the RSA breach and other recent network intrusions characterized as “ultra-sophisticated” and “advanced persistent threats.”  If you missed the first piece, please check out Advanced Persistent Tweets: Zero-Day in 140 Characters.

The recent data breach at security industry giant RSA was disconcerting news to the security community: RSA claims to be “the premier provider of security, risk, and compliance solutions for business acceleration” and the “chosen security partner of more than 90 percent of the Fortune 500.”

The hackers who broke into RSA appear to have leveraged some of the very same Web sites, tools and services used in that attack to infiltrate dozens of other companies during the past year, including some of the Fortune 500 companies protected by RSA, new information suggests. What’s more, the assailants moved their operations from those sites very recently, after their locations were revealed in a report published online by the U.S. Computer Emergency Readiness Team (US-CERT), a division of the U.S. Department of Homeland Security.

In RSA’s explanation of the attack, it pointed to three domains that it claimed were used to download malicious software and to siphon sensitive data taken from its internal networks: Good[DOT]mincesur[DOT]com, up82673[DOT]hopto[DOT]org and www[DOT]cz88[DOT]net. But according to interviews with several security experts who keep a close eye on these domains, the Web sites in question weren’t merely one-time attack staging grounds: They had earned a reputation as launch pads for the same kind of attacks over at least a 12 month period prior to the RSA breach disclosure.

What’s more, the same domains were sending and receiving Internet connections from dozens of Fortune 500 companies during that time, according to Atlanta-based Damballa, a company that mines data about malware attacks using a network of sensors deployed at Internet service providers and large enterprises around the world. Damballa monitors the domain name system (DNS) servers at those networks, looking for traffic between known good hosts and known or suspected hostile locations.

Gunter Ollmann, Damballa’s vice president of research, said that for more than a year his company has been monitoring the three malicious sites that RSA said were involved in the theft of its intellectual property, and that many other major companies have had extensive communications with those hostile domains during that time. He added that his company is not in a position to name the other companies impacted by the breach, and that Damballa is helping federal authorities with ongoing investigations.

The unceasing barrage of targeted email attacks that leverage zero-day software flaws to steal sensitive information from businesses and the U.S. government often are described as being ultra-sophisticated, almost ninja-like in stealth and anonymity. But according to expert analysis of several recent zero-day attacks – including the much publicized break-in at security giant RSA — the Chinese developers of those attack tools left clues aplenty about their identities and locations, with one apparent contender even Tweeting about having newly discovered a vulnerability days in advance of its use in the wild.

Zero-day threats are attacks which exploit security vulnerabilities that a software vendor learns about at the same time as the general public  does;   The vendor has “zero days” to fix the flaw before it gets exploited. RSA and others have labeled recent zero-day attacks as the epitome of the so-called “advanced persistent threat” (APT), a controversial term describing the daily onslaught of digital assaults launched by attackers who are considered highly-skilled, determined and possessed of a long-term perspective on their mission. Because these attacks often result in the theft of sensitive and proprietary information from the government and private industry, the details usually are shrouded in secrecy when law enforcement and national security investigators swoop in.

Open source information available about the tools used in recent attacks labeled APT indicates that some of the actors involved are doing little to cover their tracks: Not only are they potentially identifiable, they don’t seem particularly concerned about suffering any consequences from their actions.

Bragging rights may play a part in the attackers’  lack of duplicity. On Apr. 11, 2011, security experts began publishing information about a new zero-day attack that exploited a previously unknown vulnerability in Adobe‘s Flash Player software, a browser plug-in installed in 96 percent of the world’s Microsoft Windows PCs .  The exploit code was hidden inside a Microsoft Word document titled “Disentangling Industrial Policy and Competition Policy.doc,” and reportedly was emailed to an unknown number of U.S. government employees and contractors.

Four days earlier, on Apr. 7, an individual on Twitter calling himself “Yuange” and adopting the humble motto “No. 1 hacker in China top hacker in the world,” tweeted a small snippet of exploit code, apparently to signal that he had advance knowledge of the attack:

“call [0x1111110+0x08].”

It wasn’t long before malware researchers were extracting that exact string from the innards of a Flash exploit that was landing in email inboxes around the globe.

Tweeting a key snippet of code hidden in a zero-day exploit in advance of its public release may seem like the hacker equivalent of Babe Ruth pointing to the cheap seats right before nailing a home run. But investigators say the Chinese Internet address used to download the malicious files in the early hours of the April Flash zero-day attacks — 123.123.123.123 — was in some ways bolder than most because that address  would appear highly unusual and memorable to any reasonably vigilant network administrator.

This wasn’t the first time Yuange had bragged about advance knowledge of impending zero-day attacks. On Oct. 27, 2010, he boasted of authoring a zero-day exploit targeting a previously unknown vulnerability in Mozilla’s Firefox Web browser:

“Wrote the firefox 0day. You may see “for(inx=0′inx<0×8964;inx++). You should know why 0×8964 here.”

That same day, experts discovered that the Web site for the Nobel Peace Prize was serving up malicious software that exploited a new vulnerability in Firefox. An analysis of the attack code published by a member of Mozilla’s security team revealed the exact code snippet Yuange had tweeted.

On February 28, 2011, Yuange taunted on Twitter that new zero-day traps were being set:

“ready? new flash 0day is on the way.”

On Mar. 14, Adobe acknowledged that a new Flash flaw was being exploited via a booby-trapped Flash component tucked inside of Microsoft Excel files. Three days after that, EMC’s security division RSA dropped a bombshell: Secret files related to its widely used SecurID authentication tokens had been stolen in “an extremely sophisticated cyber attack.” A follow-up blog post from RSA’s Uri River two weeks later stated that the break-in was precipitated by the zero-day Adobe had warned about on Mar. 14, and that the lure used in the attack on RSA was an Excel file named “2011 Recruitment Plan.”

Source: FireEye

On Mar. 16, just one day before RSA disclosed the breach, researchers at Milpitas, Calif. based security firm FireEye released their analysis of an exploit that used the same zero-day Flash flaw. The specific attack FireEye analyzed included a different lure than the one used against RSA: An Excel file titled “Environmental Scan Matrix of Risk and Security Organizations.” When FireEye investigators dug deeper into the Excel file, they found metadata indicating the file had last been saved  by a user named “Linxder.”

“Who is this linxder?” FireEye’s Atif Mushtaq asked in a Mar. 16 posting to the company’s blog. “My colleague Darien pointed me to few links on google that tells us that a guy named ‘linxder’ is a known chinese threat actor. This guy is an old-school hacker that has a fairly expansive social network. If one searches linxder’s baidu profile, we can see that he talks a ton about weaponizing flash containers in other file formats, which is exactly what happens in this attack.”

The Linxder profile linked in FireEye’s write-up has since been wiped clean of more than two years worth of blog posts, but Google’s cache still contains some of his older blog entries from 2009, including one that indicates Linxder and Yuange were acquaintances.

WILL THE REAL YUANGE PLEASE COME FORWARD?

The Yuange1975 character on Twitter may be very well be a composite of several different individuals, said Andre M. DiMino, a cybersecurity expert and former director of Shadowserver.org, a group that tracks cybercrime activity.

“At first, there were a lot of people really intrigued by this guy,” DiMino said. “But it looks pretty likely that there are a group of folks who are tweeting to this account.”

Yuange’s Twitter profile lists a blog account on Chinese Internet provider Baidu.com by the same name, but the Yuange at that blog appears to be an old school hacker from Chinese Internet security firm NSFocus who claims to have had nothing to do with the RSA exploit. He  also complains that the “Yuange1975″ on Twitter is impersonating him.

Neither the Twitter Yuange nor the Baidu Yuange responded to requests for interviews. Frank Ip, vice president of North America operations for NSFocus, said the Baidu Yuange is a man named Yuan Renguang, one of 12 co-founders of NSFocus, and that Renguang left the company in 2005 to start his own data loss prevention firm. Ip said Renguang was being impersonated, and that he is quite widely respected in China.

“Not only is this [Twitter] impersonator using his name, but he stole [Renguang's] picture,” Ip said, adding that the real Yuange doesn’t speak English and has never published anything in English, wheres the Twitter Yuange tweets only in English.

Earlier this month, Reuters ran a story based on secret U.S. State Department diplomatic cables released by Wikileaks. The piece chronicled the theft of terabytes of data from U.S. firms and the government over the past several years, and attributed the attacks to specialized electronic espionage units within the Chinese People’s Liberation Army (PLA). But that piece didn’t address the legions of civilian hackers who conduct the same classes of attacks for patriotic reasons, for bragging rights, or simply to earn money.

Image: thedarkvisitor.com

Scott Henderson, a military analyst at the U.S. Army’s Foreign Military Studies Office in Ft. Leavenworth, Kans., wrote extensively about this phenomenon in his eBook titled “The Dark Visitor” (Henderson co-authors a blog on this subject). Henderson said it may be that the RSA attack was launched by members of what’s known as the Red Hacker Alliance, a Chinese nationalist hacker network made up of many independent Web sites directly linked to one another, in which individual sites educate their members on computer attack and intrusion techniques. Henderson said the Red Hacker Alliance is characterized by its members launching coordinated attacks against foreign governments and entities to protect actual and perceived injustices done to their nation, but that monetary motivations increasingly are becoming as important as patriotic passion.

“It’s interesting because so many of these guys are doing this stuff out in the open, and you have to ask why, and what’s the risk-to-reward ratio for these guys, and does [the Chinese government] use them as a political hammer or as a quasi-intelligence gathering network that is tacitly approved by Beijing, and I think you’d have to say ‘yes’ to all of those,” Henderson said in a phone interview. “I don’t think there has been enough pressure on Beijing to change that, because these guys are very much out in the open and talking about what they’re doing, and in some cases almost crowd-sourcing their work.”

Henderson said the most damaging common aspect of all the attacks is that the assailants never seem to quit. “We hear about these really sophisticated attacks these guys are doing, but really it always boils down to social engineering,” Henderson said. “They send out enough emails to enough recipients at Company X that someone eventually clicks on these things and suddenly the attacker gets  access to the target’s system. There are so many of these groups and this activity is going on so continuously that the challenge is trying get a handle on what exactly we should be looking at. I always wonder, if this is the stuff we’re seeing, where are the really good guys, the ones you don’t see? If the successful attacks are so blatant and open, and these guys probably aren’t the crème de la crème, where are the really good guys?”

This is the first in a series of planned stories on the RSA attack and the menace from advanced persistent threats.

Details about the recent cyber attacks against security firm RSA suggest the assailants may have been taunting the industry giant and the United States while they were stealing secrets from a company whose technology is used to secure many banks and government agencies.

Earlier this month, RSA disclosed that “an extremely sophisticated cyber attack” targeting its business unit “resulted in certain information being extracted from RSA’s systems that relates to RSA’s SecurID two-factor authentication products.” The company was careful to caution that while data gleaned did not enable a successful direct attack on any of its SecurID customers, the information “could potentially be used to reduce the effectiveness of a current two-factor authentication implementation as part of a broader attack.”

That disclosure seems to have only fanned the flames of speculation swirling around this story, and a number of bloggers and pundits have sketched out scenarios of what might have happened. Yet, until now, very little data about the attack itself has been made public.

Earlier today, I had a chance to review an unclassified document from the U.S. Computer Emergency Readiness Team (US-CERT), which includes a tiny bit of attack data: A list of domains that were used in the intrusion at RSA.

Some of the domain names on that list suggest that the attackers had (or wanted to appear to have) contempt for the United States. Among the domains used in the attack (extra spacing is intentional in the links below, which should be considered hostile):

A partial list of the domains used in the attack on RSA

www usgoodluck .com

obama .servehttp .com

prc .dynamiclink .ddns .us

Note that the last domain listed includes the abbreviation “PRC,” which could be a clever feint, or it could be Chinese attackers rubbing our noses in it, as if to say, “Yes, it was the People’s Republic of China that attacked you: What are you going to do about it?”

Most of the domains trace back to so-called dynamic DNS providers, usually free services that allow users to have Web sites hosted on servers that frequently change their Internet addresses. This type of service is useful for people who want to host a Web site on a home-based Internet address that may change from time to time, because dynamic DNS services can be used to easily map the domain name to the user’s new Internet address whenever it happens to change.

Unfortunately, these dynamic DNS providers are extremely popular in the attacker community, because they allow bad guys to keep their malware and scam sites up even when researchers mange to track the attacking IP address and convince the ISP responsible for that address to disconnect the malefactor. In such cases, dynamic DNS allows the owner of the attacking domain to simply re-route the attack site to another Internet address that he controls.

Sam Norris, founder of ChangeIP.com, the dynamic DNS provider responsible for many of the root domains on the US-CERT’s list, said he terminated all of the accounts on the list as soon as US-CERT published the list on March 18 (although that version of the list does not mention the RSA connection). Norris soon was contacted via email by the account holder who used the prc. dynamiclink. ddns. us domain. Norris said the account holder wanted to know the reason his domain was killed.

“This guy has been emailing me, asking me for the account back, saying things like ‘Hey, I had important stuff on that domain, and I need to get it back,’” Norris said. “The bad guys are definitely interested in getting it back, which means we probably cut off their communications or made it so that they couldn’t clean up their trail afterward.”

Much of the public speculation about the attack on RSA so far has invoked the term “advanced persistent threat” or APT, which is security industry shorthand for “We’re pretty sure it came from China.” At least as far as the domains that were routed through ChangeIP.com are concerned, that assessment appears to hold up (with the usual caveat that attackers can route their traffic through machines anywhere in the world in a bid to disguise their true location).

“Ninety nine percent of the time, when these guys logged in to one of their accounts to change the IP address for a domain, they were coming from a Chinese address,” Norris said.

An illustration of a targeted attack that used Poison Ivy. Image: Mandiant

A closer look at some of the domains also indicates the use of some familiar attack tools that have been associated with previous targeted attacks attributed to Chinese, state-sponsored hackers. For example, one of the few domains on the list not attached to a dynamic DNS service — mincesur .com — has been a well-known download source for “Poison Ivy,” a lightweight attack tool that attackers have  used quite a bit in previous pinprick attacks (PDF) to remotely administer hacked systems and to hoover up information from those machines.

Interesting as these tidbits of data may be, they don’t answer the questions that seem to be on everyone’s minds about the RSA attack: How much information did the attackers get, and can organizations still trust SecurID tokens as an authentication mechanism? A spokesman for RSA said the company wasn’t yet ready to publicly disclose more details about the attack. Several sources say RSA recently briefed a small group of industry leaders and customers, providing further information about the attack, but those folks had to sign a non-disclosure agreement barring them from discussing the details.

Since RSA’s initial disclosure, I’ve received many emails from readers asking for my take on the attack. I’ve avoided writing about it because I didn’t have much to add to the initial reporting, which remains very speculative in the absence of more details from RSA. And as I read back over what I’ve written above, I can see this that post seems speculative as well. As for RSA’s technology, I have noted in one story after another that one-time tokens such as those generated by RSA’s SecurID key fobs are better than mere passwords for authentication, but not by much. Today’s attack tools allow the bad guys to control not only the victim’s PC, but also what the victims see in their Web browser. I have written about a number of successful attacks in which the crooks got the information they needed to defeat tokens and empty bank accounts by injecting content into the victim’s browser. The latest attack on RSA serves to increase suspicion, even if unfounded, that its products may not provide sufficient protection to the user.

KrebsOnSecurity.com was honored at the annual Social Security Blogger Awards at the RSA security conference in San Francisco this week. Judges and voters picked this blog as the one they thought best represents the security industry today.

Among the four other finalists in this category were some fairly big names (in no particular order):

* Threat Post
* CSO Online Blog
* Threat Level (Wired)
* Schneier On Security

This is the second year in a row KrebsOnSecurity.com was recognized at the blogger awards gathering: Last year, it was named the “Best Non-Technical Security Blog“. Thanks to the judges, voters and to all you readers who make the discussion here so much more interesting, informative and worthwhile!

Sophos’s Naked Security blog won for “Most Educational”; Veracode’s Zero Day Labs won for “Best Corporate Security blog”; “Best Podcast” went to Pauldotcom; the Securosis blog earned the “Most Entertaining” award.

Below is a great video from Chris Eng who won the “The single best security blog post of the year” award, with the following text-to-movie clip on what it takes to be an authentic “thought leader” in the information security space:

In October 2010, I discovered that the authors of the SpyEye and ZeuS banking Trojans — once competitors in the market for botnet creation and management kits — were planning to kill further development of ZeuS and fuse the two malware families into one supertrojan. Initially, I heard some skepticism from folks in the security community about this. But three months later, security experts are starting to catch glimpses of this new hybrid Trojan in the wild, with the author(s) shipping a series of beta releases that include updated features on a nearly-daily basis.

It probably didn’t help that the first report of a blended version of SpyEye/ZeuS (referred to as SpyZeuS for the remainder of this post) — detailed in a McAfee blog post — turned out to be a scam. But a little more a week ago, Trend Micro spotted snapshots and details of SpyZeuS components, noting that the author appears to have received help from other criminals in polishing this latest release; in particular, an add-on that grabs credit card numbers from hacked PCs, and a plugin designed to attack the anti-Trojan tool Rapport from Trusteer. (Trusteer’s Amit Klein addresses this component in a blog post here).

Seculert, a new threat alert service started by former RSA fraud expert Aviv Raff, includes some screen shots of the administrative panel of SpyZeuS that show the author trying to appeal to users of both Trojans, by allowing customers to control and update their botnets using either the traditional ZeuS or SpyEye Web interface.

The hybrid SpyZeuS Trojan lets users interact with bots via the ZeuS control panel (left) or the SpyEye interface.

Raff said the author(s) has been adding new features to both the bot and the control panels nearly every day.

“This is under heavy development at the moment,” Raff said. “That’s why the version we wrote about was called 1.3.05 Beta, because it’s still not the [general availability] version. The author is still trying things out.”

The same day Raff’s post went up, a source forwarded me a link to a video posted to a popular hacker forum by a SpyZeuS customer who was using an even newer version, v. 1.3.09 Beta. The video (which the poster starts with a typo confusing ZeuS and SpyEye) shows how this user managed to hack the protection scheme built into SpyEye that is supposed to prevent buyers from making unauthorized copies of the crimeware package. Very shortly after posting that video, the user who recorded it had his forum account compromised and his personal and financial details posted online.

Update, 10:26 a.m.: Added response from Trusteer. Also, a previous version of this post incorrectly attributed a McAfee blog post to Trend Micro. The above text has been corrected.

ATM skimmers, or devices that thieves secretly attach to cash machines in order to capture and ultimately clone ATM cards, have captured the imagination of many readers. Past posts on this blog about ATM skimmers have focused on their prevalence and stealth in attacking cash machines in the United States, but these devices also are a major problem in Europe as well.

According to the European ATM Security Team (EAST), a not-for-profit payment security organization, ATM crimes in Europe jumped 149 percent form 2007 to 2008, and most of that increase has been linked to a dramatic increase in ATM skimming attacks. During 2008, a total of 10,302 skimming incidents were reported in Europe. Below is a short video authorities in Germany released recently showing two men caught on camera there installing a skimmer and a pinhole camera panel above to record PINs.

EAST estimates that European ATM fraud losses in 2008 were nearly 500 million Euros, although roughly 80 percent of those losses resulted from fraud committed outside Europe by criminals using stolen card details. EAST believes this is because some 90 percent of European ATMs now are compliant with the so-called “chip and pin” or EMV (an initialism for Europay, Mastercard and VISA) standard.

ATM cards store account data on magnetic strips on the backs of the cards, and thieves have focused their attention on lifting the data from customer cards — either through handheld skimmers — or via magnetic strip readers on ATM skimmers. The data can then be re-encoded onto blank ATM cards, and used at ATM along with the victim’s PIN to withdraw cash. The EMV approach uses a secret algorithm embedded in the chip planted into each ATM card. The chip encodes the card data, making it harder (but certainly not impossible) for fraudsters to read information from them or clone them. RSA‘s Idan Aharoni wrote an informative post about this technology earlier this year.

Needless to say, U.S. based financial institutions do not require chip-and-PIN, and that may be a contributor to the high fraud rates in the United States. The U.S. Secret Service estimates that annual losses from ATM fraud totaled about $1 billion in 2008, or about $350,000 each day.

While many of the images below are not new, they showcase some of the actual ATM skimmers deployed against European cash machines (click any of the images to view a slideshow).

Last week, security experts launched a sneak attack to disconnect Troyak, an Internet service provider in Eastern Europe that served as a global gateway to a nest of cyber crime activity. For the past seven days, unnamed members of the security community reportedly have been playing Whac-a-Mole with Troyak, which has bounced from one legitimate ISP to the next in a bid to reconnect to the wider Internet.

But experts say Troyak’s apparent hopscotching is expected behavior from what is in fact a carefully architected, round-robin network of backup and redundant carriers, all designed to keep a massive organized criminal operation online should a disaster like the Troyak disconnection strike.

Security firm RSA believes Troyak is but one of five upstream providers that encircle a nest of eight so-called “bulletproof networks” – Web hosting providers considered impervious to takedown by local law enforcement (pictured in red in the graphic below). RSA said this group of eight hosts some of the Internet’s largest concentrations of malicious software, including password stealing banking Trojans like ZeuS and Gozi, as well as huge repositories of personal and financial data stolen by these Trojans and a notorious Russian phishing operation known as RockPhish.

According to a report RSA issued today, these eight networks connect directly to Troyak and four other upstream providers that “surround the malicious core,” and help to “mask the true malware-hosting armada and provide solid uptime to the malware servers.” In addition, Troyak and the other four upstream providers (shown in orange in the diagram above) all share connections amongst themselves, and individually connect to one or more legitimate, regional ISPs (the green circles in the picture above) that can provide connections to the global Internet.

In fact, RSA said, when Troyak was initially knocked offline on March 9, it was because several regional ISPs (green networks on the left side of the graphic) simultaneously denied it service. Presumably, these ISPs cut the cord to Troyak due to pressure from security researchers who enumerated and explained to those ISPs the criminal networks they were supporting.

The trouble was, the four other providers in Troyak’s hub also had their own connections to regional ISPs, and so the entire network of bulletproof hosts that largely depended on Troyak to reach the larger Internet could suddenly shift gears and connect to the Web through these peers. The regional ISPs are depicted in the green circles in the map above, and RSA calls them legitimate ISPs, although anti-spam outfit Spamhaus on Tuesday listed one of Troyak’s main regional connection — Russia-based NLINE — on its spam blacklist for “repeatedly hosting cybercriminal spam gangs.”

“It is important to understand that although part of this infrastructure may lose connectivity, these bulletproof networks are still able to resume online activity through other upstream providers they have access to; most are back online having accessed alternate connections within that same cybercrime infrastructure,” RSA stated in its report on the Troyak takedown. “This redundancy mechanism is at the core of keeping malicious servers up and running over time, as observed through the past week’s events.”

RSA isn’t alone in trying to map badness and ISP reputation on the Web. In an excellently timed-paper, a trio of university researchers released a study this week at the IEEE Infocom conference in San Diego that used data from at least a dozen spam, malware, bot and phishing blacklists to identify malicious networks. The researchers, from the Oak Ridge National Laboratory and Indiana Unversity at Bloomington, identified several dense clusters of ISPs – particularly in Ukraine and Turkey – that appeared to be overly tolerant of activity emanating from their networks.

For example, the researchers also sought to identify ISPs and hosting providers that had a disproportionate number of network peers that were malicious. For this measurement, they focused on ISPs with at least three such partner networks. They found 22 networks that had 100 percent of their customers classified as malicious, while some 194 networks had at least 50 percent of their customers fall into that category.

A story I wrote on that study can be found in today’s online edition of MIT Technology Review, at this link here.

I’ll be writing more about other data-driven efforts to identify problem ISPs and hosting providers over the next few days. Stay tuned.

Organized cyber criminals stole more than $25 million from small to mid-sized businesses in brazen e-banking heists in the 3rd quarter of 2009 alone, federal regulators said last week. In contrast, traditional stick-up artists hauled less than $9.5 million out of U.S. banks over that same time period last year.

Speaking at the RSA Security Conference in San Francisco last week, David Nelson, an examination specialist with the Federal Deposit Insurance Corporation (FDIC), said online banking attacks against small businesses of the sort I have chronicled countless times over the past year netted thieves $25 million between July and September of 2009.

I wondered how that stacked up against real-life bank robbers here in the U.S., so I had a look at the FBI‘s published bank crime statistics for that same time period last year. Turns out, traditional bank robbers committed a total of 1,184 bank robberies during those three months, netting slightly more than $9.4 million (including $3,071 in travelers checks).

In fact, real-life bank robbers stole a total of just over $30 million in the first three quarters of 2009, just $5 million more than cyber crooks did in the third quarter of last year alone.

Small wonder that the haul from cyber bank robberies has overtaken that of physical heists:  Cyber thieves take far fewer risks to life, liberty and limb than do real-life bank robbers. In that same three month period last year, the FBI says bank robberies at bricks-and-mortar institutions caused five deaths — all them perpetrators of the crime.

What’s more, the perpetrators of these incessant attacks against small businesses banking online for the most part reside in countries that are traditionally beyond the reach and influence of U.S. law enforcement. Sure, bank robbers occasionally kill people (more often themselves) while they’re stealing your money, instead of silently lifting it out of your bank account from afar like cyber thieves. That alone makes them a more emotional high-value target for the feds. But let’s face it: Traditional stick up artists are a lot easier to collar. For one thing, by necessity they are all here in the United States.

In addition, while traditional bank robbers are limited to the amount of money they can physically carry from the scene of the crime, cyber thieves have a seemingly limitless supply of accomplices to help them haul the loot, by hiring so-called money mules to carry the cash for them.

I can’t help but notice one other important distinction between these two types of bank crimes: The federal government sure publishes a lot more information about physical bank robberies that it makes available about online stick-ups.

Indeed, the FBI’s bank crime stats are extraordinarily detailed. For example, they can tell you that in the 3rd quarter of last year, bank robbers were more likely to hold up their local branch between the hours of 9 a.m. and 11 a.m. on a Wednesday than at any other time or day of the week; they can tell you the number of tear gas and dye packs taken with the loot, the number of security cameras activated, the number of food stamps taken, even what percentage of suspected perpetrators had illegal drug habits at the time of the robberies. About the only thing the stats don’t tell you is what brand of jeans the perpetrators were wearing and whether the getaway car had cool vanity plates.

What do we get about e-crime statistics from the federal government? One guy from the FDIC giving a speech at the RSA conference. And as we heard from the FDIC last week, the federal regulators could start collecting (and hopefully publishing) these kinds of statistics from America’s banks, but that would require an okay from the White House.

One of the first posts that I published at krebsonsecurity.com was a story about how much time and effort I put into trying to get the government to acknowledge how much cyber crooks were stealing from small to mid-sized businesses last year in these online banking attacks. Given this latest disclosure, it’s not hard to see why the banks and feds would be reluctant to part with that information.

The FBI hasn’t yet published the 4th quarter 2009 bank crime statistics, but if the $25 million cyber heist figure is representative of a quarterly trend last year — and the first three quarters of stats from last year’s FBI stats don’t deviate much in the 4th quarter — cyber crooks will have stolen well more than twice as much as traditional bank robbers last year in the United States.

I’m quite certain that if the infamous Willie Sutton had his heyday in the present culture, Sutton’s fabled answer to the question of why he robbed online banks would have been, “Because that’s where the *easy* money is.”