What do we do with a company that regularly pumps metric tons of virtual toxic sludge onto the Internet and yet refuses to clean up their act? If ever there were a technology giant that deserved to be named and shamed for polluting the Web, it is Xiongmai — a Chinese maker of electronic parts that power a huge percentage of cheap digital video recorders (DVRs) and Internet-connected security cameras.
In late 2016, the world witnessed the sheer disruptive power of Mirai, a powerful botnet strain fueled by Internet of Things (IoT) devices like DVRs and IP cameras that were put online with factory-default passwords and other poor security settings.
Security experts soon discovered that a majority of Mirai-infected devices were chiefly composed of components made by Xiongmai (a.k.a. Hangzhou Xiongmai Technology Co., Ltd.) and a handful of other Chinese tech firms that seemed to have a history of placing product market share and price above security.
Since then, two of those firms — Huawei and Dahua — have taken steps to increase the security of their IoT products out-of-the-box. But Xiongmai — despite repeated warnings from researchers about deep-seated vulnerabilities in its hardware — has continued to ignore such warnings and to ship massively insecure hardware and software for use in products that are white-labeled and sold by more than 100 third-party vendors.
On Tuesday, Austrian security firm SEC Consult released the results of extensive research into multiple, lingering and serious security holes in Xiongmai’s hardware.
SEC Consult said it began the process of working with Xiongmai on these problems back in March 2018, but that it finally published its research after it became clear that Xiongmai wasn’t going to address any of the problems.
“Although Xiongmai had seven months notice, they have not fixed any of the issues,” the researchers wrote in a blog post published today. “The conversation with them over the past months has shown that security is just not a priority to them at all.”
PROBLEM TO PROBLEM
A core part of the problem is the peer-to-peer (P2P) communications component called “XMEye” that ships with all Xiongmai devices and automatically connects them to a cloud network run by Xiongmai. The P2P feature is designed so that consumers can access their DVRs or security cameras remotely anywhere in the world and without having to configure anything.
To access a Xiongmai device via the P2P network, one must know the Unique ID (UID) assigned to each device. The UID is essentially derived in an easily reproducible way using the device’s built-in MAC address (a string of numbers and letters, such as 68ab8124db83c8db).
Electronics firms are assigned ranges of MAC address that they may use, but SEC Consult discovered that Xiongmai for some reason actually uses MAC address ranges assigned to a number of other companies, including tech giant Cisco Systems, German printing press maker Koenig & Bauer AG, and Swiss chemical analysis firm Metrohm AG.
SEC Consult learned that it was trivial to find Xiongmai devices simply by computing all possible ranges of UIDs for each range of MAC addresses, and then scanning Xiongmai’s public cloud for XMEye-enabled devices. Based on scanning just two percent of the available ranges, SEC Consult conservatively estimates there are around 9 million Xiongmai P2P devices online.
[For the record, KrebsOnSecurity has long advised buyers of IoT devices to avoid those advertise P2P capabilities for just this reason. The Xiongmai debacle is yet another example of why this remains solid advice].
BLANK TO BANK
While one still needs to provide a username and password to remotely access XMEye devices via this method, SEC Consult notes that the default password of the all-powerful administrative user (username “admin”) is blank (i.e, no password).
The admin account can be used to do anything to the device, such as changing its settings or uploading software — including malware like Mirai. And because users are not required to set a secure password in the initial setup phase, it is likely that a large number of devices are accessible via these default credentials.
Even if a customer has changed the default admin password, SEC Consult discovered there is an undocumented user with the name “default,” whose password is “tluafed” (default in reverse). While this user account can’t change system settings, it is still able to view any video streams.
Normally, hardware devices are secured against unauthorized software updates by requiring that any new software pushed to the devices be digitally signed with a secret cryptographic key that is held only by the hardware or software maker. However, XMEye-enabled devices have no such protections.
In fact, the researchers found it was trivial to set up a system that mimics the XMEye cloud and push malicious firmware updates to any device. Worse still, unlike with the Mirai malware — which gets permanently wiped from memory when an infected device powers off or is rebooted — the update method devised by SEC Consult makes it so that any software uploaded survives a reboot.
CAN XIONGMAI REALLY BE THAT BAD?
In the wake of the Mirai botnet’s emergence in 2016 and the subsequent record denial-of-service attacks that brought down chunks of the Internet at a time (including this Web site and my DDoS protection provider at times), multiple security firms said Xiongmai’s insecure products were a huge contributor to the problem.
Among the company’s strongest critics was New York City-based security firm Flashpoint, which pointed out that even basic security features built into Xiongmai’s hardware had completely failed at basic tasks.
For example, Flashpoint’s analysts discovered that the login page for a camera or DVR running Xiongmai hardware and software could be bypassed just by navigating to a page called “DVR.htm” prior to login.
Flashpoint’s researchers also found that any changes to passwords for various user accounts accessible via the Web administration page for Xiongmai products did nothing to change passwords for accounts that were hard-coded into these devices and accessible only via more obscure, command-line communications interfaces like Telnet and SSH.
Not long after Xiongmai was publicly shamed for failing to fix obvious security weaknesses that helped contribute to the spread of Mirai and related IoT botnets, Xiongmai lashed out at multiple security firms and journalists, promising to sue its critics for defamation (it never followed through on that threat, as far as I can tell).
At the same time, Xiongmai promised that it would be issuing a product recall on millions of devices to ensure they were not deployed with insecure settings and software. But according to Flashpoint’s Zach Wikholm, Xiongmai never followed through with the recall, either. Rather, it was all a way for the company to save face publicly and with its business partners.
“This company said they were going to do a product recall, but it looks like they never got around to it,” Wikholm said. “They were just trying to cover up and keep moving.”
Wikholm said Flashpoint discovered a number of additional glaring vulnerabilities in Xiongmai’s hardware and software that left them wide open to takeover by malicious hackers, and that several of those weaknesses still exist in the company’s core product line.
“We could have kept releasing our findings, but it just got really difficult to keep doing that because Xiongmai wouldn’t fix them and it would only make it easier for people to compromise these devices,” Wikholm said.
The Flashpoint analyst said he believes SEC Consult’s estimates of the number of vulnerable Xiongmai devices to be extremely conservative.
“Nine million devices sounds quite low because these guys hold 25 percent of the world’s DVR market,” to say nothing of the company’s share in the market for cheapo IP cameras, Wikholm said.
What’s more, he said, Xiongmai has turned a deaf ear to reports about dangerous security holes across its product lines principally because it doesn’t answer directly to customers who purchase the gear.
“The only reason they’ve maintained this level of [not caring] is they’ve been in this market for a long time and established very strong regional sales channels to dozens of third-party companies,” that ultimately rebrand Xiongmai’s products as their own, he said.
Also, the typical consumer of cheap electronics powered by Xiongmai’s kit don’t really care how easily these devices can be commandeered by cybercriminals, Wikholm observed.
“They just want a security system around their house or business that doesn’t cost an arm and leg, and Xiongmai is by far the biggest player in that space,” he said. “Most companies at least have some sort of incentive to make things better when faced with public pressure. But they don’t seem to have that drive.”
A PHANTOM MENACE
SEC Consult concluded its technical advisory about the security flaws by saying Xiongmai “does not provide any mitigations and hence it is recommended not to use any products associated with the XMeye P2P Cloud until all of the identified security issues have been fixed and a thorough security analysis has been performed by professionals.”
While this may sound easy enough, acting on that advice is difficult in practice because very few devices made with Xiongmai’s deeply flawed hardware and software advertise that fact on the label or product name. Rather, the components that Xiongmai makes are sold downstream to vendors who then use it in their own products and slap on a label with their own brand name.
How many vendors? It’s difficult to say for sure, but a search on the term XMEye via the e-commerce sites where Xiongmai’s white-labeled products typically are sold (Amazon, Aliexpress.com, Homedepot.com and Walmart) reveals more than 100 companies that you’ve probably never heard of which brand Xiongmai’s hardware and software as their own. That list is available here (PDF) and is also pasted at the conclusion of this post for the benefit of search engines.
SEC Consult’s technical advisory about their findings lists a number of indicators that system and network administrators can use to quickly determine whether any of these vulnerable P2P Xiongmai devices happen to be on your network.
For end users concerned about this, one way of fingerprinting Xiongmai devices is to search Amazon.com, aliexpress.com, walmart.com and other online merchants for the brand on the side of your device and the term “XMEye.” If you get a hit, chances are excellent you’ve got a device built on Xiongmai’s technology.
Another option: open a browser and navigate to the local Internet address of your device. If you have one of these devices on your local network, the login page should look like the one below:
Another giveaway on virtually all Xiongmai devices is pasting “http://IP/err.htm” into a browser address bar should display the following error message (where IP= the local IP address of the device):
According to SEC Consult, Xiongmai’s electronics and hardware make up the guts of IP cameras and DVRs marketed and sold under the company names below.
What’s most remarkable about many of the companies listed below is that about half of them don’t even have their own Web sites, and instead simply rely on direct-to-consumer product listings at Amazon.com or other e-commerce outlets. Among those that do sell Xiongmai’s products directly via the Web, very few of them seem to even offer secure (https://) Web sites.
SEC Consult’s blog post about their findings has more technical details, as does the security advisory they released today.
In response to questions about the SEC Consult reports, Xiongmai said it is now using a new encryption method to generate the UID for its XMEye devices, and will not longer be relying on MAC addresses.
Xiongmai also said users will be asked to change a devices default username and password when they use the XMEye Internet Explorer plugin or mobile app. The company also said it had removed the “default” account in firmware versions after August 2018. It also disputed SEC Consult’s claims that it doesn’t encrypt traffic handled by the devices.
In response to criticism that any settings changed by the user in the Web interface will not affect user accounts that are only accessible via telnet, Xiongmai said it was getting ready to delete telnet completely from its devices “soon.”
KrebsOnSecurity is unable to validate the veracity of Xiongmai’s claims, but it should be noted that this company has made a number of such claims and promises in the past that never materialized.
Johannes Greil, head of SEC Consult Vulnerability Lab, said as far as he could tell none of the proclaimed fixes have materialized.
“We are looking forward for Xiongmai to fix the vulnerabilities for new devices as well as all devices in the field,” Greil said.
Here’s the current list of companies that white label Xiongmai’s insecure products, according to SEC Consult:
WNK Security Technology
Update, 3:44 p.m.: Updated story to include Xiongmai’s statement.
A rendering of headquarters?
Moral of this story don’t trust cheap electronic goods from China .
Or, please name any Chinese products that can be trusted?
All my cameras are the cheap Chinese ones. The trick is to block them at the firewall. I see hundreds of thousands of denied requests to access WAN connections. Simple deny all from their static IP’s I set deals with it. Then the feeds go to my securityspy (IP cam software) machine and I view the feeds from there via HTTPS. You all are freaking out like you should never buy these cheap cameras, but the quality and price ratio cannot be beat. You just have to know a few things in regard to secure network config to protect yourself, which clearly you don’t.
@The Sunshine State
Very true, but this is the “elephant in the room” that was talked about yesterday. It’s far too late to change that.
Unless you bring back electronic manufacturing back to the United States and start employing people again in factories .
Well then prices go skyrocketing.
Yes they would, on the other hand, just how many people actually need a smart phone, DVR, etc? The reason these devices are proliferating is because they are *CHEAP*. Cheap enough that people are buying them as commodities and aren’t even aware they’re actually dangerous. The most Average Joe is aware of them flooding his uplink is perhaps his Netflix stream is flaky, which Netflix gets undeserved blame. Or he’s got Comcast in which case because of horrific service anyway, never notices. Point is, most people make erroneous assumptions about computer technology and take no effort to learn otherwise. The only way to keep dangerous products out of their hands is to price them high enough that only people that need them are going to buy them. For everyone else they’re luxuries, or technically skilled enough to build their own.
That said, moving all electronics manufacturing back to the US won’t solve this issue either. There are plenty of US companies that pump and dump electronics into the domestic market and never offer support for any firmware/software problems after initial warranties, and only spotty support during limited warranty periods.
This is a problem that will require multiple layers of effective regulations, enforcement, and education from supply chain, design, marketing and retail, to users. Something the US populace and politicians have only rarely shown the social and political will to enact – usually only after some spectacular fatal disaster(s).
You can bring the manufacturing back to USA, but there’s no way you’ll force customers to buy these particular low-end goods from USA instead of the cheapest provider from China.
The Sunshine State,
Given the history of manufacturing in the United States, I have little confidence that goods manufactured here would be any more secure. It won’t happen until there’s a monetary incentive to make a secure product. There’s certainly no such incentive now.
Now there’s a product that deserves a tariff.
I use AOMEI Backupper software. Anyone have any insecurity dirt to dish on that Chinese product?
I use AOMEI Backupper software. Anyone have any security-related dirt to dish up on that Chinese product?
The company that distributes it has a website that was registered by a Savvy Investments LLC in Wyoming. It always worries me when there’s more than one corporate name tied to a product I want to use.
I got that info here: https://whois.domaintools.com/backup-utility.com
The website uses Google Analytics, just like KOS does to measure website traffic. Google is working with the Chinese government to track all of its citizens’ web searches.
Thus, the software is a partner of China. 🙂
“Bringing manufacturing back to the USA” isn’t going to change anything. Remember the story of King Canute? The tide has changed.
We have global stores like Amazon hosting unlimited numbers of merchants selling this crap to a host of consumers who only ever choose on price. Here in Europe we watch the protectionism in the US with interest, bordering on obsession, and wonder how long the mirage of bringing the jobs back and making america great again can last.
Well done Brian for reporting these stories, don’t stop doing what you’re doing.
I wonder if it would be possible at some point to publish a list of firms/products that are prioritize security… Or just a few names. It would help if you were willing to pay extra for security.
Of course, it two are mentioned in the article:
“Since then, two of those firms — Huawei and Dahua — have taken steps to increase the security of their IoT products out-of-the-box.”
And that is already a help. Thanks!
A few other better known brands: Pelco, FLIR, Hikvision
Hikvision (and Dahua) and subsidiaries/rebranded equipment is specifically banned for US .gov purposes, as of 08/13/2019.
FWIW, of late, I only see FLIR and Axis in military installations. I’m a fan of Axis since they use a lot of open standards, notably ONVIF.
Wow, this sucks,
I personally have a Zmodo PoE cam/DVR setup and was surprised they are part of this issue.
I have my cams firewalled and only 2 ports open to dvr for remote viewing capability and it does not use cloud viewing option, only direct IP. I of course set my own username/pw but was not aware of hardcoded pw’s.
I will have to probe my DVR using default passwords given. I knew the security basically sucked on the system but was not concerned about someone seeing the outside of my house nor am aware of any cases of network traversal using the dvr (still not mentioned but will research more) I was not aware remote firmware could be uploaded either and do not want to be part of a botnet
And why the hell is telnet even included these day?
People need to firewall the routers that connect to open internet from their houses, then inspect traffic across their local LANs.
And don’t get me started with wireless! I set up a Kali workstation in my kitchen (it’s central to the house) to get some idea of what the neighborhood looked like in wireless…..it told me a neighbor had a NEST thermostat and some other things…..
But the weirdest thing I saw was that it kept detecting a very powerful signal from an (according to MAC address) LG device — but only intermittently. The only LG device I own is a dishwasher that this computer was immediately above, but I had installed that dishwasher myself — and it certainly hadn’t mentioned wireless connections.
I finally asked the workman who I had adjusting the drip irrigation outside the kitchen window what brand of cellphone he used, and, yes, it was LG.
The IoIT — Internet of Insecure Things — strikes again.
I found a quirk in QSEE dvr’s that use the serial number for the P2P. If I took the serial number and just advanced by one number, using default login, I gained access to about 25 remote dvr’s and with admin privileges.
Did you try going in the other direction with that number?
Always do what I do with Chinese equipment. I out it in an isolated and firewalled network. I did the exact same with Misecu IP cameras before I knew about security issues. I assume if hardware is that cheap nobody will care about supporting it and releasing updated firmware.
This is the main reason why I don’t have any security cameras around my house. It does help that there is a street light in front of my house.
Home security cameras deter theft and vandalism, as street lights do.
But they do so much more:
They document theft, accidents, and vandalism and help catch the bad guys, while preventing false claims against your property.
They provide single people with an alibi, should a crime occur nearby.
They’ll show that your date left your house alive. And with all his/her limbs.
Indoor cameras are especially good at catching abusive caretakers and babysitters.
And they help you figure out whose dog keeps pissing in your azaleas.
If you’re security conscious, consider a style/brand that can operate and record to a box in your home without wifi, Ethernet, or cloud storage. Two examples: Night Owl’s wired cameras and Harbor Freight’s Cobra Surveillance.
I’m just waiting for a white-hat hacker out there who understands Mirai to make their own worm that starts bricking every unsecured piece of junk Xiongmai makes. Cheap or not, even the most frugal won’t buy them once it’s known that they die in a month.
Now that is a unique and great idea.
It would force customers back to whomever they purchased from; Once that retailer started seeing 100% returns on that product they would have to send all that defective product back . . . And eventually, it would put that insecure company into bankruptcy.
My thoughts exactly. A device this insecure has got to be easy to brick. Therefore, someone will do it.
‘In Technology, whatever can be done will be done’ – Andrew S. Grove, president, CEO, and chairman of the board of Intel Corporation.
While probably possible, Mirai was not persistent so I would guess it’s source wouldn’t be the best place to start. That said, one would lose the “white” designation if they did something like this. Gray Hat is more applicable and, I’m sure, some would say Black is even more apt.
This was actually done a couple of years back (in regard to a different vulnerability with different hardware).
There was some discussion of the ethics involved, but he never got that much penetration…..and, when he heard the Feds were involved, he turned it off.
There is only one way to fix this.
Governments must decide it is a national threat on put a stop on all import and sales.
And when you are awakened by your 50 year old Timex Windup Alarm Clock and have to wait for your stovetop percolator to brew your morning coffee, you will see the absurdity and fallacy of your MAGA president as well as your posted statement!
Bro? You’re the only one who said anything about Trump.
What an absurd and nationalist view. Good luck with that.
We don’t allow the import of children’s toys with lead-based paint because it’s dangerous. Why not disallow the import of crappy cameras with dangerous software in them. How is this an absurd or a nationalist view? I’m sure every country has bans on items that are dangerous in one way or another.
The crappy quality and default settings on IoT devices certainly do not help, but the true problem is the absolutely false promise that you can just go around making stuff available anywhere over the Internet without having security problems. Anything that exchanges packets with random parties on the Internet is going to be aggressively attacked. Very few things are up to being aggressively attacked. Even if the product is well-designed and implemented, it still must be configured intelligently. We see this constantly with cloud-based services – the AWS leaky bucket of storage du jour stories, the SaaS and online services that get their accounts hacked, etc. The Internet is a dangerous neighborhood, and you need to keep your doors locked, windows barred, security systems armed, and cameras watched. People thinking that they, as amateurs, can securely set up something they can get at from anywhere is a deeply problematic lie that’s been promulgated by these companies. Even so-called professionals screw it up regularly. For now, if you want remote access use a good VPN that you control.
I find it disturbing, and slightly amusing, that China has found a way to video surveil a multitude of unsuspecting people, and companies, and have the targets pay for the installation!
@Mike Agreed. That’s what I was thinking–this is company could be complicit in a threat to national security. These things have audio/mic on-board, too, right? Oh, and I love this company name: WNK Security Technology — Might as well be “Wink-Wink Security LOLZ, Inc.”
Not just China. SEC Consult points out that the video streams go through Amazon Web Services. That means the US government can probably compel Amazon to let them see the video too.
When viewed in combination with Brian’s prior story, it’s tempting to believe the PRC has multiple threads in its plan to sew seeds of future internet chaos and remote control.
In a previous article, Mr. Krebs made the useful point that too many companies put information in the cloud without having a good reason for doing so.
[[The same day I alerted them, All American took down its bucket of unsecured speaker contract data, and apologized profusely for the oversight (although I have yet to hear a good explanation as to why this data needed to be stored in the cloud to begin with).]]
Of course, having an explanation for the need to put something in the cloud implies the perception that such an action carries risk. That perception is, unfortunately, often absent. Putting information in the cloud, or indeed even putting it on a computer connected to the internet brings risk but many people don’t understand that fact.
In the same way, many people buy smart devices without having a good reason for doing so. That’s because many people (probably the vast majority) have no perception that buying smart devices has a downside or that it brings risk.
Basic security concepts:
1. Understand the risks.
2. Avoid risk exposure whenever possible.
3. Mitigate risk exposure when risk exposure can not be avoided.
Most people never even try to accomplish step 1.
A marketer of mobile phones must go through certification in many national markets. (The marketer–Apple, Samsung– not the original device manufacturer–Foxconn, ASUS, etc.)
But not marketers of cheap IoT (or IoWG — Internet of Worthless Garbage).
Is it time for Walmart to step up and start screening the stuff they sell for infosec problems?
To sell anything that plugs into power in the US, you need approval from Underwriters Labs. They’re a certification association jointly sponsored by fire-insurance companies. If an insured party uses unapproved electric gizmos, they break the terms of their insurance policy. It’s been this way for well over a century.
Do we need a similar kind of certification for IoWC products, and even for laptops and servers? I say yes, we do.
Maybe Good Housekeeping could start giving their “seal of approval” (1) for good devices, guiding the market without government involvement.
Or BK could market the KSOA: krebs seal of approval.
Krebs already exercises power over the markets and banks. Wise consumrs will pay attention. I will not buy system listed on this page.
We as consumrs must demand integrity and values from business leadrs with the power of the dolar. amazon facebook and google will pay attention only if money involve.
You can choose to look at this as an arm of the Chinese government doing an admirable job of being paid by Western consumers to set up an impromptu spy network on themselves.
Said spy network can also be set up remotely to perform DoS services on demand, plus acting as blind proxies for the real state-sponsored hacking teams that must be kept fully-deniable.
Good luck taking it down, it’s already in place. We wanted the cheap so badly, we invited the Dragon into our daily lives – and paid them handsomely for the privilege of doing it. A masterful setup job on the part of the Chinese government.
No, I don’t seriously believe this. Hanlon’s Razor applies: “Never attribute to malice that which is adequately explained by stupidity.” We sure are stupid. It’s also had the same effect as malice, because that net is now *there* for anyone to use.