March 1, 2010

I’ve grown fascinated over the years with various efforts by Internet service providers to crack down on the menace from botnets, large groupings of hacked PCs that computer criminals remotely control for a variety of purposes, from spamming to hosting malicious software and attacking others online. Indeed, the botnet problem has become such a global menace that entire countries are now developing anti-botnet programs in collaboration with domestic ISPs.

One of the more unique and long-running examples of this is Japan’s “Cyber Clean Center,” (referred to hereafter as CCC) a little-known effort by the Japanese Computer Emergency Response Team Coordination Center (JP-CERT) and a collection of 76 Japanese ISPs covering 90 percent of the nation’s Internet users.

Participating ISPs that have customers with botted PCs may send those users an e-mail — and in some cases a letter via postal mail — instructing them to visit the CCC’s Web site, and download and run a cleanup tool developed by the JP-CERT in coordination with Trend Micro, the dominant anti-virus and computer security firm in Japan.

Relatively few of the thousands of U.S.-based ISPs have such programs in place, or if they do then not many have been willing to discuss them publicly. Some notable exceptions are Cox, Comcast (which is rolling out a trial bot infection notification system), and Qwest (if I missed any other biggies, readers please set me straight).

It’s unfortunate that such programs aren’t more widely emulated, because a majority of the world’s bot problem begins and ends here in the United States.  According to a recent report (.pdf) by McAfee, the United States is home to the second largest pool of botted PCs — 2nd only to China — and is the world’s biggest exporter of junk e-mail.

Obviously, as with most customer notification schemes, the primary challenge is messaging the user in a way that is not easily spoofed by criminals to convince people to download “disinfection tools” that actually infect their systems.But at any rate, I thought the Japanese example was interesting as a regional approach, and so over the past several months have engaged the CCC folks in a Q&A via e-mail.

Below are some of the questions I asked and answers I received about the CCC program.

BK: About how many customers has the CCC reached over the years?

CCC: Between Nov. 2009 and its inception in Dec. 2006, Japan’s CCC  has helped more than a million customers remove bot infections from their PCs.

BK: Is this a manual or automated process?

CCC: The detection of bots, collection, analysis and alerting of affected users is mostly automated.

BK: Is there a direct cost to ISPs for participating? Is it compulsory in any way?

CCC: Involvement is optional for ISPs, but they get the benefit of a system they can provide free of charge to their customers to deal with bot infections.

BK: What is the policy among participating ISPs as to how they deal with customers whose systems are botted? How long do they have to be on the list of problem PCs to get a notice?

CCC: It’s generally up to the ISP on how to deal with infected users.  ISPs receive infection notices from the CCC project, but the ISP will follow their own procedures to contact the user.  So, the exact means they use to deal with infected customers and the timing is the ISP’s decision.

BK: Is that notice sent via e-mail, or snail mail? Both?

CCC: On the whole, notices are sent via email, but some ISPs are trying to improve on the response rate by using both email and snail mail. Here’s an example of one type of page that notified users might be asked to visit.

BK: Do you think there particular aspects about the Japanese culture that might be a factor in helping this program succeed, such as societal mores that may not be present in other cultures?

CCC: That’s an interesting question.  It’s hard to be sure whether or not there are particular cultural traits responsible for the CCC project’s success, though we expected that we might run into some user opposition at the beginning.  For example, users asking “Why are they trying to spy on my computer?”, or that infection notices would be interpreted as trying to interfere too much.  Even though we’d identified this as a risk, the replies we received expressed thanks, with very little negative feedback.  Perhaps this kind of acceptance when an outsider points something out is partly a trait of typical Japanese personalities.

One other difference might be that overseas, anti-bot activities involve law enforcement agencies trying to catch criminals or perhaps focusing on making life difficult for those infected.  However, the CCC doesn’t have significant police co-operation.  We’re just contacting people who get infected, as well as raising awareness of the problem.  Perhaps this kind of approach is also particularly Japanese.

BK: I notice that the rate of downloading disinfection tools by alerted users is 30 percent (cumulative). So that means that for every 10 people notified about a bot on their system, 3 people will respond and download available removal tools? Or could it be that removal tools just aren’t available for the bot that’s on their system?

CCC: From the users who receive the infection reports, around 40% access the CCC website and about 30% download the CCC Cleaner removal tool.  In other words, for every 10 people, 4 read the mail and visit the web site.  By reading the guide on the CCC Cleaner web site, users can get to grips with the download process.  So in answer to your first question, four out of ten people will visit the site, and three out of ten people will download CCC Cleaner.  Additionally, the notifications are for users who are already known to be infected with a bot which can be disinfected.  Therefore, there’s no chance that the removal tool will be completely useless for a particular user receiving an email.

BK: Does the number of alerts sent pretty closely track the number of bot infections detected? Or are there other factors that trigger when a customer whose machine shows signs of being botted gets notified?

CCC: When an infected user is detected, we send a notification.  If the CCC project detects another bot infection after the first notification, another one will be sent.


19 thoughts on “Talking Bots with Japan’s ‘Cyber Clean Center’

  1. Bernhard Schulte

    I would have asked “How do you make sure that your email is not mistaken for a scareware/phishing attempt?”

    Email senders can be spoofed easily.

  2. Pete

    Kudos for taking this on as a country!

    I’d have said 30% is a good return, it must be a pretty targeted communication for returns of that level.

  3. jbmoore61

    Alternatively, ISPs could route their customers to a cleaning service through browser redirection via DNS. Once their system is “clean”, the redirection ends. ISPs already hijack DNS queries for ads, so it wouldn’t be difficult to redirect infected users to a cleaning site since the ISP’s control the network through DHCP and DNS. The ISPs’ likely know who is infected from their IDS and firewall logs and they could enable egress filtering to prevent confidential user information or remote system control going overseas, but many ISP’s just don’t care about their customers. They only see them as revenue sources, not as people.

    1. JackRussell

      Comcast is already doing effectively this. When they suspect machines are compromised, they place that customer in a “walled garden”, which severely restricts their browsing to a set of sites related to the cleaning of viruses and other malware.

      I wouldn’t say that ISPs don’t care – viruses and botnets on customer computers to them are a nuisance, and if there were an easy way to eliminate them, they would certainly do it.

      1. Dan

        @JackRussel,
        Have you actually seen Comcast do this ““walled garden” against infected computers? I read about it, but never seen it done or what technology they use to do it.

    2. anonymous

      DNS is not a good measure to block users as it is trivial for a bot/trojan to reconfigure the DNS on systems and avoid such redirections.

  4. Andrew

    Rather than emailing an easily spoofed message, why not include large red text on the user’s statement that indicates their machine(s) is(are) compromised?

    DNS redirection could be an ok idea for most users, but what about those of us who don’t use their ISP’s DNS?

    1. BuddhaChu

      With automated billing, who actually opens the snail mail statements?

  5. Andrew

    @anonymous who posted right before me…

    Yep…good point on DNS reconfiguration.

  6. Terry Ritter

    It makes sense for ISP’s to detect botnet activities and notify owners. Illicit outgoing traffic is difficult to monitor from the infected machine since malware can avoid software firewalls.

    On the other hand, here we have a massive assumption that, since malware can be acquired, it naturally can be removed. Well, it used to be that way, and malware files can be removed if they are found. But a modern botnet often downloads a range of infections, some of which may be too stealthy or too targeted or too new to find. No removal tool can remove what it cannot find.

    Malware can make serious system changes beyond OS startup files. We do not and can not know about these changes, and therefore cannot reverse them without reloading everything.

    The era of removing malware is over. With modern malware, recovery means re-installing the OS, or loading an uninfected image. Like a pet which has just acquired a serious disease, computers with malware are suddenly much more trouble (and more expensive) than we thought they would be.

  7. Tim

    Interesting article. Wonder if ASP’s will or have pursued this model. Cloud services ripe for abuse.

    That said prevention would be a better alternative.

  8. KFritz

    Are there any stats on response percentage comparison.
    IE comparing response rates by notified Japanese customers to response rates to Americans who are notified by their ISP?

  9. prairie_sailor

    I used to work support for Qwest – at that time (about 6 years ago) if a customer was infected and appeared to be ignoring notices we would shut off their account utill they cleaned up. An alert in the customer trouble ticket database would tell support about it. I have no idea if their policy is the same now.

  10. Rick

    Great article. But one wants to cry for an Internet that’s never had a chance. All this time and expense – all because an individual in the northwest of the americas refuses to release a secure operating system. It’s criminal.

    1. BuddhaChu

      Microsoft bashing goes over like a fart in church.

      This “Rick” sounds a lot like Richard Stallman. RMS, is that you?

  11. Sniper

    good for the japanese at least its seems they got their act together not unlike the americans with wide-open holes such as those of NASA.gov all listed at pinoysecurity.

    i mean if we cant protect the most valuable govt servers, what chance have we got as ordinary end-users?

  12. jbmoore61

    The bots don’t usually modify DNS so much as add an entry into /etc/hosts on Windows systems. AV products do the same thing by adding 127.0.0.1 entires to redirect the browser through the AV proxy the customer installed. H Security has an article about Cox Communications using DNS redirection to clean bots: http://www.h-online.com/security/news/item/ISP-uses-DNS-to-redirect-from-IRC-to-bot-cleaner-733318.html (from 2007). It’s 2010 and I haven’t heard of any other ISP doing it, but I’ve seen Verizon and OpenDNS redirect me to ads.

  13. jbmoore61

    Illicit traffic is difficult to monitor? IDS will show you the source and destination of a session. If Latvia, Ukraine, or Russia is talking to one of your customers via IRC or web and it’s not a legitimate IRC or web server, what do you think is the obvious answer? Plus, the bot will have very brief sessions on the order of less than a minute whereas humans will chat or surf for minutes or hours on IRC or the web. Due to PCI signatures, I’m sure the IDS will fire when a bot sends out credit information overseas in the clear. Most customers credit card traffic would be encrypted via HTTPS. This is not rocket science here and detection is not difficult unless your IT Security staff is either understaffed or incompetent and they deployed their sensors incorrectly.

Comments are closed.