More than 900,000 customers of German ISP Deutsche Telekom (DT) were knocked offline this week after their Internet routers got infected by a new variant of a computer worm known as Mirai. The malware wriggled inside the routers via a newly discovered vulnerability in a feature that allows ISPs to remotely upgrade the firmware on the devices. But the new Mirai malware turns that feature off once it infests a device, complicating DT’s cleanup and restoration efforts.
Security experts say the multi-day outage is a sign of things to come as cyber criminals continue to aggressively scour the Internet of Things (IoT) for vulnerable and poorly-secured routers, Internet-connected cameras and digital video recorders (DVRs). Once enslaved, the IoT devices can be used and rented out for a variety of purposes — from conducting massive denial-of-service attacks capable of knocking large Web sites offline to helping cybercriminals stay anonymous online.
An internet-wide scan conducted by Shodan.io suggests there may be as many as five million devices vulnerable to the exploit that caused problems for so many DT customers this week. Image: Badcyber.com
This new variant of Mirai builds on malware source code released at the end of September. That leak came a little more a week after a botnet based on Mirai was used in a record-sized attack that caused KrebsOnSecurity to go offline for several days. Since then, dozens of new Mirai botnets have emerged, all competing for a finite pool of vulnerable IoT systems that can be infected.
Until this week, all Mirai botnets scanned for the same 60+ factory default usernames and passwords used by millions of IoT devices. But the criminals behind one of the larger Mirai botnets apparently decided to add a new weapon to their arsenal, incorporating exploit code published earlier this month for a security flaw in specific routers made by Zyxel and Speedport.
These companies act as original equipment manufacturers (OEMs) that specialize in building DSL modems that ISPs then ship to customers. The vulnerability exists in communications protocols supported by the devices that ISPs can use to remotely manage all of the customer-premises routers on their network.
According to BadCyber.com, which first blogged about the emergence of the new Mirai variant, part of the problem is that Deutsche Telekom does not appear to have followed the best practice of blocking the rest of the world from remotely managing these devices as well.
“The malware itself is really friendly as it closes the vulnerability once the router is infected,” BadCyber noted. “It performs [a] command which should make the device ‘secure,’ until next reboot. The first one closes port 7547 and the second one kills the telnet service, making it really hard for the ISP to update the device remotely.” [For the Geek Factor 5 readership out there, the flaw stems from the way these routers parse incoming traffic destined for Port 7547 using communications protocols known as TR-069].
DT has been urging customers who are having trouble to briefly disconnect and then reconnect the routers, a process which wipes the malware from the device’s memory. The devices should then be able to receive a new update from DT that plugs the vulnerability.
That is, unless the new Mirai strain gets to them first. Johannes Ullrich, dean of security research at The SANS Technology Institute, said this version of Mirai aggressively scans the Internet for new victims, and that SANS’s research has shown vulnerable devices are compromised by the new Mirai variant within five to ten minutes of being plugged into the Internet.
Ullrich said the scanning activity conducted by the new Mirai variant is so aggressive that it can create hangups and crashes even for routers that are are not vulnerable to this exploit.
“Some of these devices went down because of the sheer number of incoming connections” from the new Mirai variant, Ullrich said. “They were listening on Port 7547 but were not vulnerable to this exploit and were still overloaded with the number of connections to that port.”
A Deutsche Telekom Speedport DSL modem.
FEEDING THE CRIME MACHINE
Allison Nixon, director of security research at Flashpoint, said this latest Mirai variant appears to be an attempt to feed fresh victims into one of the larger and more established Mirai botnets out there today.
Nixon said she suspects this particular botnet is being rented out in discrete chunks to other cybercriminals. Her suspicions are based in part on the fact that the malware phones home to a range of some 256 Internet addresses that for months someone has purchased for the sole purpose of hosting nothing but servers used to control multiple Mirai botnets.
“The malware points to some [Internet addresses] that are in ranges which were purchased for the express purpose of running Mirai,” Nixon said. “That range does nothing but run Mirai control servers on it, and they’ve been doing it for a while now. I would say this is probably part of a commercial service because purchasing this much infrastructure is not cheap. And you generally don’t see people doing this for kicks, you see them doing it for money.”
Nixon said the criminals behind this new Mirai variant are busy subdividing their botnet — thought to be composed of several hundred thousand hacked IoT devices — among multiple, distinct control servers. This approach, she said, addresses two major concerns among cybercriminals who specialize in building botnets that are resold for use in huge distributed denial of service (DDoS) attacks.
The first is that extended DDoS attacks which leverage firepower from more bots than are necessary to take down a target host can cause the crime machine’s overall bot count to dwindle more quickly than the botnet can replenish itself with newly infected IoT devices — greatly diminishing the crime machine’s strength and earning power.
“I’ve been watching a lot of chatter in the DDoS community, and one of the topics that frequently comes up is that there are many botnets out there where the people running them don’t know each other, they’ve just purchased time on the botnet and have been assigned specific slots on it,” Nixon said. “Long attacks would end up causing the malware or infected machines to crash, and the attack and would end up killing the botnet if it was overused. Now it looks like someone has architected a response to that concern, knowing that you have to preserve bots as much as you can and not be excessive with the DDoS traffic you’re pushing.”
Nixon said dividing the Mirai botnet into smaller sections which each answer to multiple control servers also makes the overall crime machine more resistant to takedown efforts by security firms and researchers.
“This is an interesting development because a lot of the response to Mirai lately has been to find a Mirai controller and take it down,” Nixon said. “Right now, the amount of redundant infrastructure these Mirai actors have is pretty significant, and it suggests they’re trying to make their botnets more difficult to take down.”
Nixon said she worries that the aggressive Mirai takedown efforts by the security community may soon prompt the crooks to adopt far more sophisticated and resilient methods of keeping their crime machines online.
“We have to realize that the takedown option is not going to be there forever with these IoT botnets,” she said.
…and so the battle rages on.
I am willing to lend my gray hat services for Peppermint tea, steaks, and massages that end with happy endings for at least moderately attractive females.
The cyber-criminals will just start using TOR to connect to a command and control server via a proxy , which then take downs will be next to impossible.
It’s already happening with ransom-ware
Not that I’d put it past them, but you have to realize these are routers sold to OEMs for pennies or nickles. To say that they have extremely limited RAM and storage space is redundant.
From the last few IoT articles it sounds like most of the compromises use exploits that allow them to write and execute code from RAM. When you only have 8MB (or less) of RAM, it’s very hard to execute 32MB of code.
Not likely, as setting up a secure TOR circuit is not trivial in a small memory space.
‘Criminals will connect to command and control servers via tor…’ Isn’t that how botnet works already? You control a piece of software that doles out commands to multiple sources? Whether directly from the attackers computer or from a hosted VPS.
I’m sorry – but that’s exactly what is going on now and people are getting caught.
There has been no reports of use of the TOR (TOR Onion Router) anonymizing network as a channel for IoT attacks. Maybe Brian Kerbs knows otherwise? TOR on Ubuntu, as an example, takes 3490 kB, not counting necessary dependencies. To infect an IoT device with a TOR capable malware would require the minimum code to connect to the network and make a circuit, and the malware code itself.
TOR could be used on the server side of things using an exit point, absolutely. But not likely on the IoT side of things. So while the server side may be anonymized, the IoT side will be open to view, unless SSH or SSL is implemented, which doesn’t appear to be the case from reports available.
You DO realize it’s iTunes not ITUNE, right?
semantics ! who cares, stick to the story
I care – I have to look at what is basically error every time I want to peruse the krebsonsecurity.com comments section. It’s an eyesore.
Meanwhile, ISIS uses .mil vulnerabilities to launch attacks against various nations making everyone think it’s the US trying to start a war. (atleast the potential for such a thing)
*citation needed*
There is no need for .mil to have any vulnerabilities. There are many articles available online on IP address spoofing.
One wonders how successful these miscreants would be if they actually did something useful.
Crime is tax free, and no regulations. No need to bribe legislators, either. This underground economy exists for many reasons.
This one seems pretty easy to stop at the border routers, though: just block incoming connection attempts to port 7547 and outgoing ones to the 256 IP addresses hosting the control servers.
Or what am I missing?
Not missing much, Ron. There is very little reason for inbound solicitation requests on non standard services port numbers to be anything but denied. A port filtering bypass attempt would use 21,80,8080,etc for the initial inbound solicitation to make it appear as somewhat legitimate traffic.
Malware bypassing ruleset used to be more effective, initiating an outbound solicition (trusting), and then allowing inbound connection on non standard port numbers bypassing the firewall inbound ruleset.
Why have 7547 open at all?
http://www.dslreports.com/forum/r23497316-Port-7547-is-open-in-the-SE567-router-why
That thread is from 2009. Seems like asking Why I keep having these people hit me up on port 27374? Are they really trying to find a machine with Sub7 running on it in 2016? Lol, they are.
Ron, here is an easy fix. No need to block anything or not use a service.
Instead of using telnet – they could of used an IPSEC SSH tunnel with a cipher signature based login. Each router could have a string cipher that can be used to get a password and username combination.
So the initial SSH knock would say hello — I am tretR^$#rtfs54w^#$ please enter credentials now.
The credentials are derived from the knock string.
However, it will only allow limited SUDO-ing of a user so that input would cannot be put in one after the other and executed like ROOT can do. So, a user would be in the sudoers – allowed one command execution at a time. There is no sudo -s, or sudo -i, Only sudo . sudo and so on.
Which then allows a remote connection from the ISP to push updates securely.
The cipher key could be leaked, broken, and so on. But at this point its better than turning it off or blocking or dropping.
These routers also need some intelligence. It needs to have some advanced firewall features so that when it detects a scan that would put it off line – it can just turn the wan port off for 15 minutes – and notify the users via the gateway. Something like comcast was doing -“Brand firewall model xxx has detected an aggressive scan on – temporarily shut down service for 15 minutes.. 12 minutes left.. if you want to reactivate the wan port, please click here…”
Then there should be a scan log sent to a department at the ISP so they can make adjustments to block at the network level – and protect all its assets erm, clients monetary interest.
The problem is that would require hiring programers who have some competency.
Filtering was set up after the attack was discovered: https://www.telekom.com/en/media/media-information/archive/information-on-current-problems-444862
Why it is always the cyber criminals finding and exploiting the vulnerabilities ? Can’t the Corporate cyber security guys be proactive rather be reactive ? Can’t they fix the vulnerabilities BEFORE exploitation. In this case it seems the vulnerabilities is already known but not fixed. IF true, wonder what the Corporate cyber security is doing all day in office. Just talking cyber security in conferences and obtaining certifications so that they can move to next job with big, fat pay check ?
Ed,
It is cheaper to be reactive. The security appliances and competent IT staff needed to thwart such attacks are typically costlier than paying for the aftermath of one. Plus insurance covers the breaches, they do not buy appliances and devices. Win/Win for the company.
For instance database breaches are so common now – its hardly news worthy. WHen you read about you – you sigh and go — “again!?”
Go look up the price of a enterprise grade UTM device.
A WatchGuard XTM 535 Network Security Appliance is $4,000. That’s what you need. Now imagine the cost if you need several of those Firewall Appliances let a lone other server and networking appliances. The company buys the bare minimum security and spends money on devices that will make money.
There needs to start being penalties for companies who retain user identifiable information like address, SSN, credit card numbers when there is a breach. Each subsequent breach is costlier than the previous – until the company may no longer do that type of business online.
Like getting a DUI, the first two a misdemeanors and cost thousands and spent face, the third one is a felony and you lose it all.
Thanks for the reply Brian. I got it.
But regarding the corporate cyber security guys , I think – based on my insider experience – they lack the zeal & passion but see that just a way to make money. I should add not all of them are having this mindset, but majority do have.
Bunch of dummies talking about it but get caught and their text copied….,…..
I wish it was as easy to remediate vulnerabilities as you seem to think it is. In every organization I’ve done InfoSec at, the security professionals like myself don’t have access to patch and reconfigure server/network settings. We can find the vulnerabilities, but then it is a ticket to the proper support group to do that work. Next, every single server/network team is understaffed. So a zero-day vulnerability is quite often exposed for days, highs for weeks or months, lows likely never fixed. The business is extremely reluctant to reboot servers/network devices, which means that most patching gets done once a month during a designated patch window.
My team has been fighting for most of this year to move all new applications to use 443/TCP instead of 80/TCP so that data in transit is encrypted. For every project we manage to get half-way decent security built-in, two go through with inadequate. What this means is that security MUST be everyone’s responsibility.
Defense is always reactive to some extent. The bad guys just have to find one hole to wiggle through, we have to watch everything. Sure we can apply best practices, defense in depth, monitoring, etc., but we can’t do anything to stop vendors from writing code with bugs in it. We can’t do anything to stop hackers from spending lots of time looking for that one hole.
I understand Alan. I myself – as an one worked with giants in energy sector – observed how the procedures slow down the pace with which the problems are fixed.
But I also observed that the procedure itself is used as a cover not to act faster / not acting swiftly.
Seems ( what is called in India for too much procedures ) RED TAPE plays a vital part in slowing down the speed of cybersecurity
Brian, your information is incorrect, these routers haven’t been infected. The worm was looking for a TR-069 protocol vulnerability affecting Zyxel routers. The Speedport routers in question weren’t affected – in fact, with their custom operating system they couldn’t possibly be infected by an exploit targeting Linux systems. What rather happened was the constant hammering on TR-069 port overloading these routers and causing them to hang up. It seems that a DDoS on German Telekom customers was merely an unintended side-effect of an attack on Zyxel routers deployed by the Irish provider Eir. You can read up more on the technical details here:
https://comsecuris.com/blog/posts/were_900k_deutsche_telekom_routers_compromised_by_mirai/
The official communication from the ISP also agrees with your post. So the Krebs-article is probably wrong as you point out. But it fits well with his current narrative that “IoT is very bad!! And routers are IoT!”, so.. 😉
https://www.telekom.com/en/media/media-information/archive/information-on-current-problems-444862
I was just going to comment similarly. The Mirai sourcode was altered by some rather unexperienced hackers and the Telecom routers were just freezing up.
In fact turning them on&off resolved the problem until the attack was blocked entirely or stopped otherwise.
Having blasted that sourcecode into the wild was a rather idiotic move. We will see more of these attacks in the future.
Middle of second paragraph:
“One enslaved, the IoT devices can be used and rented out for a variety of purposes…”
Should it be “once enslaved”?
Thanks, Krabs, for getting on it (in advance)!
It’s not as simple and just being proactive. Corporate information security teams are only so large and only have so much time for everything they need to accomplish in a day. If I have to spend 4 hours of my day pouring over logs looking for any intrusion into my network, I only have 4 left to review the devices we have for any vulnerabilities or problems (and whatever I am willing to put into research at home). On the other hand, as Anonymous so succinctly put it, “[they] are legion.” For every one of me sitting at an office trying to secure a network, there are thousands of attackers out there looking to crack into networks. Sure, of each 1000, maybe 10 will have the technical ability to discover new zero-day vulnerabilities. That still puts me at a minimum 10-1 disadvantage. And those 10 are probably doing this 8 hours or more a day, putting me further behind. To further complicate my life, they only have to get lucky once to crack my network, I have to get lucky every single time.
This is why many networks now operate in an “assume compromised” posture and implement security in several layers. The goal is to slow the attacker down long enough that the initial compromise can be seen and responded to before the attacker can get far enough to do damage and/or exfiltrate data.
Certainly, security teams the world over are racing as fast as they can to find and report vulnerabilities. Unfortunately,a lot more criminals are racing even faster to find and exploit those vulnerabilities. They have the time and money incentive to win. The defenders rarely are given as much time. I’d love to just sit and do vulnerability research all day long. Are you willing to pay my bills while I do that? Probably not, most companies hire information security professionals to secure *their* network. So, that means research is a secondary job function, at best. Most companies are also not willing to pour hundreds of thousands of dollars into chasing vulnerabilities. They want an ROI on any work done. Demonstrating that 8 hours spent on reviewing Mozilla’s code has any positive ROI for a random company is very difficult, if not impossible.
If we (as a country or community) want to be serious about vulnerability research and patching, we’re going to need to fund people to do just that, and make sure that they have the tools to do so. It won’t be cheap, and the criminals will still win on occasion.
Thanks for the reply John.
Just look for a tweet by Mikko Hypponen about how the password was handled by an admin in Saudi GACA. This is what disturbs me.
This malware is resident in volatile memory? Why not just have code in non-volatile memory that every X? minutes restores the proper code in that volatile memory, writing into areas outside of the code being executed. If the malware doesn’t modify the code itself, but instead resides in empty memory spaces, just overwrite those areas.
“If the malware doesn’t modify the code itself, but instead resides in empty memory spaces, just overwrite those areas.”
Oops, can’t do that since the malware would put a jump within the proper system code to jump to that area. Back to the progressively overwrite everything idea.
Another idea: continuous checksums of volatile memory other than areas that are expected to change such as variable storage.
The advantage of these methods are that they are not malware code or infection mechanism specific. ANY unauthorized code will be overwritten.
I have to say that is just the beginning, more powerful attacks are coming. Something that can be as high as 10Gbps
I’m still not understanding just what the bad folks are getting out of this. Money changing hands is my guess but where are the reports of it? Ransomeware – that makes sense – but what’s going on here?
Meanwhile, has Deutsche Telekom blocked the ports?
Hello,
I have a Zyxel AMG1312-T10B router with firmware V2.00(AAFP.6)C0
TR-069 protocol is disabled on my router (like Upnp, WPS etc.) but after finding this article I ran a scan with Nmap on port 7547 and with great surprise it was found open even though the firewall was active !
How to identify if a router was compromised with this exploit? There is some particular check to make on the router file system accessible via ssh ?
– A comparison of the current config file with an old one revealed no changes in the configuration file itself.
– A firewall rule to block port 7547 access from the wan has no effect on my router.
– The only effective way to disable TR-069 was to set the port to 0 on my router admin.
… it’s very suspicious that even though TR-069 was disabled on my router admin and the firewall was active with a specific rule to block access to TR-069, the service was still available on the public wan …
It seems like Zyxel has added an hidden rule to always allow TR-069 traffic …
Time and time again, network security best practices are not properly implemented and down she goes! There’s a reason why some folks poke fun at the “NOTwork team”.