Imagine buying an internet-enabled surveillance camera, network attached storage device, or home automation gizmo, only to find that it secretly and constantly phones home to a vast peer-to-peer (P2P) network run by the Chinese manufacturer of the hardware. Now imagine that the geek gear you bought doesn’t actually let you block this P2P communication without some serious networking expertise or hardware surgery that few users would attempt.
The FI9286P, a Foscam camera that includes P2P communication by default.
This is the nightmare “Internet of Things” (IoT) scenario for any system administrator: The IP cameras that you bought to secure your physical space suddenly turn into a vast cloud network designed to share your pictures and videos far and wide. The best part? It’s all plug-and-play, no configuration necessary!
I first became aware of this bizarre experiment in how not to do IoT last week when a reader sent a link to a lengthy discussion thread on the support forum for Foscam, a Chinese firm that makes and sells security cameras. The thread was started by a Foscam user who noticed his IP camera was noisily and incessantly calling out to more than a dozen online hosts in almost as many countries.
Turns out, this Focscam camera was one of several newer models the company makes that comes with peer-to-peer networking capabilities baked in. This fact is not exactly spelled out for the user (although some of the models listed do say “P2P” in the product name, others do not).
But the bigger issue with these P2P -based cameras is that while the user interface for the camera has a setting to disable P2P traffic (it is enabled by default), Foscam admits that disabling the P2P option doesn’t actually do anything to stop the device from seeking out other P2P hosts online (see screenshot below).
This is a concern because the P2P function built into Foscam P2P cameras is designed to punch through firewalls and can’t be switched off without applying a firmware update plus an additional patch that the company only released after repeated pleas from users on its support forum.
Yeah, this setting doesn’t work. P2P is still enabled even after you uncheck the box.
One of the many hosts that Foscam users reported seeing in their firewall logs was iotcplatform.com, a domain registered to Chinese communications firm ThroughTek Co., Ltd. Turns out, this domain has shown up in firewall logs for a number of other curious tinkerers who cared to take a closer look at what their network attached storage and home automation toys were doing on their network.
In January 2015, a contributing writer for the threat-tracking SANS Internet Storm Center wrote in IoT: The Rise of the Machines that he found the same iotcplatform.com domain called out in network traffic generated by a Maginon SmartPlug he’d purchased (smart plugs are power receptacles into which you plug lights and other appliances you may wish to control remotely).
What is the IOTC Plaform? According to ThroughTek, it’s a service developed to establish P2P communications between devices.
“I read the documentation provided with the device as well as all the website pages and there is no mention of this service,” wrote Xavier Mertens, an incident handler and blogger for SANS. “Manufacturers should include some technical documentation about the network requirements (ex: to download firmware updates).”
In another instance from May 2015, this blogger noted similar communications traffic emanating from a digital video recorder (DVR) device that’s sold in tandem with Internet-enabled surveillance cameras made by a company called Swann.
Likewise, postings from Dec. 2014 on the QNAP network attached storage (NAS) user forum indicate that some QNAP customers discovered mysterious traffic to iotcplatform.com and other Internet address requests that also were found in the Swann and Smart Plug traffic.
What do all of these things have in common? A visit to ThroughTek’s Web lists several “case studies” for its products, including Swann, QNAP and a home automation company based in Taiwan called AboCom.
ThroughTek did not respond to requests for comment. A ThroughTek press release from October 2015 announced that the company’s P2P network — which it calls the Kalay Network — had grown to support more than seven million connected devices and 100 million “IoT connections.”
I contacted Foscam to better understand the company’s relationship to ThroughTek, and to learn just how many Foscam devices now ship with ThroughTek’s built-in, always-on P2P technology. Foscam declined to say how many different models bundled the P2P technology, but it’s at least a dozen by my count of the models mentioned in the Foscam user manual and discussion thread.
Foscam customer service representative David Qu wrote in reply to requests for comment that “ThroughTek provides P2P technical support service for us.” He also said the P2P cameras merely keep a “heartbeat” connection to Foscam’s P2P server to check the connection status with the servers, and that no camera data will be stored on the company’s servers.
“The details about how P2P feature works which will be helpful for you understand why the camera need communicate with P2P servers,” Qu explained. “Our company deploy many servers in some regions of global world.” Qu further explained:
1. When the camera is powered on and connected to the internet, the camera will log in our main P2P server with fastest response and get the IP address of other server with low load and log in it. Then the camera will not connect the main P2P server.
2. When log in the camera via P2P with Foscam App, the app will also log in our main P2P server with fastest response and get the IP address of the server the camera connect to.
3. The App will ask the server create an independent tunnel between the app and the camera. The data and video will transfers directly between them and will not pass through the server. If the server fail to create the tunnel, the data and video will be forwarded by the server and all of them are encrypted.
4. Finally the camera will keep hearbeat connection with our P2P server in order to check the connection status with the servers so that the app can visit the camera directly via the server. Only when the camera power off/on or change another network, it will replicate the steps above.”
As I noted in a recent column IoT Reality: Smart Devices, Dumb Defaults, the problem with so many IoT devices is not necessarily that they’re ill-conceived, it’s that their default settings often ignore security and/or privacy concerns. I’m baffled as to why such a well-known brand as Foscam would enable P2P communications on a product that is primarily used to monitor and secure homes and offices.
Apparently I’m not alone in my bafflement. Nicholas Weaver, a senior researcher in networking and security for the International Computer Science Institute (ICSI), called the embedded P2P feature “an insanely bad idea” all around.
“It opens up all Foscam users not only to attacks on their cameras themselves (which may be very sensitive), but an exploit of the camera also enables further intrusions into the home network,” Weaver said.
“Given the seemingly cavalier attitude and the almost certain lack of automatic updates, it is almost certain that these devices are remotely exploitable,” he added. “It is no wonder that Director of National Intelligence James Clapper is worried about the Internet of Things, how many government officials have or may unwittingly install potential spies like this in their home.”
If you’re curious about an IoT device you purchased and what it might do after you connect it to a network, the information is there if you know how and where to look. This Lifehacker post walks through some of the basic software tools and steps that even a novice can follow to learn more about what’s going on across a local network.
Now when people ask why I don’t want a ‘smart’ thermostat or fridge that connects to the internet, I’ll kindly point them to this article. Excellent write up Brian.
+1
Well said.
You’re not ok with this, however you’re probably unaware, apathetic, or ok with every app on your smartphone basically doing the same thing, re. sending as much info as they can gather on you?
Which is exactly why my apps are very few and far between.
Which is why I don’t have a smart phone.
S.M.A.R.T. = Surveillance Marketed As Revolutionary Technology.
I feel a little smug. I have a remote desktop connection available, but only via the local lan. To get to the local lan I use OpenVPN. It requires specific certificates on the router and local pc, a custom user id and a couple of passwords.
A variation of this is needed to be sure cameras and whatever else are private. I don’t have those yet, but will be sure to take this new information into consideration before going there.
Even a secure router, such as pfSense, combined with snort and pfBlockerNG, will fail without specific support blocking ip addresses since SPI will allow outbound connections without much interference. On;y unsolicited inbound communications are blocked unless the router knows of special signatures or IP addresses of note.
“Even a secure router, such as pfSense, combined with snort and pfBlockerNG, will fail without specific support blocking ip addresses since SPI will allow outbound connections without much interference. On;y unsolicited inbound communications are blocked unless the router knows of special signatures or IP addresses of note.”
Only without proper rules. With the proper configuration, the device will only have access to hosts that are absolutely necessary, blocking access to all others.
This Is Why People Read Krebs.
@Brian _ I think you meant foscam model # FI9826P, not 286.
I am a strong supporter of IT security. I am also a big fan of IoT and see a lot of benefits for mankind. The companies that band together with the security professionals to deliver secure products will find a mountain of cash waiting for them.
Thanks Mr. Krebs for staying on top of these things for us.
Thanks Brian! As always, I can use your blog to drive points home to my clients.
Unfortunately, the peeps wants…what the peeps wants. It’s amazing what people with ignore or put up with for plug and play!
When you say some “serious network expertise” what do you mean? I have some IP cameras in my home and blocking any tcp/udp traffic from their ip to the net, seems to sort it and I am comfortable that it’s stopped them spying on me.
Am I taking too much faith.
@CP You might want to ensure you block IPv6 communication from IoT devices also. That attack vector is usually missed by most home users.
So, I have since sniffed data between my router and modem and learned a couple of things about my IP cameras.
There is no traffic other than STUN and NTP traffic. Not sure what that could be exploiting but it is there because my firewall rules only block UDP and TCP.
In addition, I have a CCTV DVR system and I learned of an undocumented master password, which lets you reset the system and that too, has internet connectivity… Yikes!
Hundreds of thousands of connections in a few minutes is NOT just a heartbeat (as seen in the foscam forum link). Or if it is, it’s a completely incompetent heartbeat that will self DDOS many lower end home routers.
The scary world we live in with IOT
This is great info. It points up the fact that we must be ever vigilant to stop or prevent this sort of shenanigans.
The lifehacker link was excellent as well.
Thanks Brian
Whats even more cavalier than the IoT innovators’ security practices is that, according to IBM, 69% of CEO’s think disruptive tech investment is crucial to their future. Disruptive has more than one goal: a product may crack a new market but the ideology is rooted in those adhering to the Sharing Economy. That is, to bypass personal privacy for the good of all; we should be willing to give-up or share more of our private data if it serves the whole of society even if it means intrusion.
For Capitalists, they surely do sound an awful lot like Communists.
Thanks Krebs! 🙂
almost as bad as: http://www.theguardian.com/world/2016/jan/29/snowden-files-us-uk-spied-feeds-israeli-drones-jets
Nice writeup, but this type of issue has been going on well before IOT…
I have had to block ports on various firewalls because some major name brand DVRs, security panels, HVAC panels, and other enterprise class business devices that were uploading data that was much more that just a heart beat to Asia.
The DVR stuff I ran into 6 years ago and when I asked to disable it the feature, the multi-billion dollar company said they could not help.
Just putting this out there…
Thanks Brian
https://www.pentestpartners.com/blog/pwning-cctv-cameras/
This person found thousands of DVR’s phoning home to send pictures of DVR images and should be included in the article.
more specifically, read the part “Sends stills to a strange hardcoded email address”
“Looking into the dvr_app binary further, we can find some very odd functionality.
For whatever reason, snapshots of the first camera are sent to lawishere@yeah.net.”
iOT & “the cloud” Why use these vulnerability’s?
As this becomes a greater and greater problem, I see a big opportunity for security researchers to publish a list of “safe” network devices to purchase.
Has anyone started something like this?
Pretty sure there’s no such devices.
Just in the last week or two cisco had a vulnerability rating 10 on an exploit that allows people to take total control of some of their more popular firewall/router models.
Anyone and everyone can be hit with exploits/vulnerabilities, and that’s not counting all the crap that’s intentional like this IoT stuff.
This is why all of my IP Cameras sit in a DMZ with no internet access and all traffic dropped aside from the ftp session used to copy recorded images.
I had this same issue with a “HooToo” branded IP camera. I suspected it was phoning home when I was able to access it when not on my home network via the included mobile app. A quick check with OSMonitor for Android showed all sorts of traffic to China. Not cool. My solution was to block all outbound traffic to the Internet on my router for the MAC Address associated with the camera. That appears to have resolved the issue. It said nothing in the configuration software about this “feature” or anything in regards to P2P stuff going on. Good job for making the public aware of this sort of thing.
Hopefully someone will get rich selling a router that prevents this type of thing. It could be as simple as putting up a list of devices on the home subnet, and the user clicking the box next to those that can connect to the public Internet. If there’s no reason for your camera or smart TV to have access, uncheck the box and that device is securely blocked, regardless of its settings.
We need to establish a company that provides a security rating for all internet connected devices. The consumer has no clue. A certification agency needs to give out letter grades A..F or something the consumer can understand. Sort of like a UL listing but not as easy to counterfeit.
At least a couple of startup efforts in work now to rate devices. Unfortunately, rating something “secure” isn’t as easy as rating safety (cars) or customer satisfaction (consumer reports).
None of these products are going to be secure if updates are not provided, and at $99 or less, the business model isn’t going to support maintaining them for the life cycle of the hardware.
I think the secure router with check boxes to allow/disallow and perhaps have white lists that a consumer can update would help, but even that won’t be easy due to the constantly changing IP addresses.
I guess we’ve reached the point where we need to firewall the internet from the LAN, as well as the other way around.
Why would companies ship products with features like this turned on by default?
Money
They want to offer the ability for people to view their cameras over their cell phones, but don’t want to spend a lot of money to develop it.
Another company comes in says, we have this free software that does exactly want you want, your customers won’t have to configure anything.
Nobody asks if this is a good idea or not. They offer it as a feature to sell more devices.
The IOT is also the Leave No Data Security Consultant Unemployed technology.
‘This time next year, Rodney, we’ll be millionaires!’ ….if only!
Corporates would prefer to pay the ransom rather than actually address their blindingly obvious security problems with some simple training and SANS advice.
See http://www.theregister.co.uk/2016/02/18/la_hospital_bitcoins/
and all too many other examples.
(I’m a cyber security consultant)
Went to the Emergency Room a year ago. PC they used to prescribe morphine was an XP box… USB ports accessible on front. I watched the nurse enter the user ID and password. I was left in the room for several hours alone, no visitors, footsteps always easy to hear long before anyone entered the room.
Every PC in the ER treatment rooms was the same.
I could have owned that PC in a heartbeat (pun intended). Then what could I do? I could power off the monitor until my USB hack was done and rebooted…
The people that work in the ER have no clue what I could do or if what I might do would matter. How many other ER’s are there like that? From what I’ve read, most of them.
Won’t force the upgrade until the its cost is less than the cost of not upgrading.
Nice writeup, this is why old school techniques, like segmentation of the network, even at home, are still valuable. When it comes to IP cams (ONVIF and others), the best approach is to have a dedicated network and use a NVR. Only the NVR should be reachable or have access to the Internet. This limits your liabilities with the different cam manufacturers (some cameras still ship with a Telnet port that CANNOT be disable). Additionally this also prevent attackers to gain physical access to your network from outside your home (like the CANBus access on a car via side mirrors, outside IP cameras are a vulnerable extension of the internal house network).
I just checked my Western Digital MyBookLive NAS and it’s got a connection to OpenX, a digital marketing firm. My Epson wireless printer has 2 connections open to YuMe.com, another digital marketing firm. I blocked the IPs in my router but this is so exasperating!
First I should say that I have no interest in TP-Link; I use the inexpensive ($16 on Amazon) model 702N to segment my home network. Lots of them.
To isolate a device from the internet, add its MAC to a disable list and the device is dead to the world.
To isolate any device from or allow connection to certain sites on the internet, use domain filtering and either add the questionable domain to the list to deny or allow a device to reach only a few desirable domains.
I have not found a circumstance where the 702n would not isolate a device in either direction – be sure to update the firmware.
Perhaps a bit tedious but, I am the guy that two years back stopped allowing JavaScript across the board on any browser on any device. Allowing JavaScript only for regularly visited sites that require it. Network segmentation and home network management is required on today’s internet.
I can’t remember whether it was Brian or someone else recently who pointed out that EVERY router manufacturer routinely abandons every one of their designs soon after introduction in terms of firmware updates even if someone uncovers a “back door”.
They would prefer you just buy the latest model and spend more money.
This news just simply does not surprise me. I’m glad Brian is covering it, so the unsuspecting can deal with it – hopefully. My thanks to all the posters here that pointed to hacks to mitigate or at least study the problem.
One possible reason why we are seeing this: PPC click fraud.
Those that are interested in easy fraud money have a harder and harder time. Ad defensive systems do a better job at blocking them, they lose money. The best attacks are to infect a user’s computer so it becomes very hard to tell user clicks from bot clicks. But as computers get more secure, that becomes more difficult. The best option may be to have users BUY equipment that will appear to be valid user traffic and click on ads. In this case, the ISPs are our only hope for a defense.
since there are other paranoid people around, i can share my paranoia. a few years ago, i remember being baffled by the push for voice controlled devices, like smartvs. i thought, i microphone on my remote control? what a stupid idea–its quicker to just push buttons. never mind that the batteries die in a fraction of the time as ordinary remotes. i wondered whether those features were subsidized by the government in some way… sneaky… thats all i got. tin foil hat is never coming off…
Why would I not consider credit card readers to be IoT devices?
It all kinda makes me wonder about all those Norse sensors.
It would seem more dangerous to surround your home with IoT devices than it would using old non-updated computers.
The ‘A Register’ reported today that a number of Chinese DVR’s have a default hard coded password of 519070. May not be related, but more details at:
http://www.theregister.co.uk/2016/02/18/blank_519070_the_pin_to_enter_to_pwn_80k_online_security_cams/
Read the lifehacker post. One of the options given for monitoring your network is using Wireshark. Looking up URLs for all of those IP addresses is frustrating at best, and impossible for long captures.
A program called NetworkMiner (at netresec.com) makes short work of evaluating pcap files from captures. It provides URLs for all of the IP address by using the DNS queries captured in the pcap file. This way you see the original URL that was requested, not a reverse DNS URL that points to some content delivery server that bears no resemblance to the requesting URL. Just open or drag your pcap file into the open NetworkMonitor window.
Many people look to Wireshark. A server as a replacement for the router running on pfSense is a very good choice.
CDN’s shouldn’t be dismissed though. A compromised CDN can be a source of infection/malware. Krebs has had articles in the past relating to cloud related issues.