The U.S. government — along with a number of leading security companies — recently warned about a series of highly complex and widespread attacks that allowed suspected Iranian hackers to siphon huge volumes of email passwords and other sensitive data from multiple governments and private companies. But to date, the specifics of exactly how that attack went down and who was hit have remained shrouded in secrecy.
This post seeks to document the extent of those attacks, and traces the origins of this overwhelmingly successful cyber espionage campaign back to a cascading series of breaches at key Internet infrastructure providers.
Before we delve into the extensive research that culminated in this post, it’s helpful to review the facts disclosed publicly so far. On Nov. 27, 2018, Cisco’s Talos research division published a write-up outlining the contours of a sophisticated cyber espionage campaign it dubbed “DNSpionage.”
The DNS part of that moniker refers to the global “Domain Name System,” which serves as a kind of phone book for the Internet by translating human-friendly Web site names (example.com) into numeric Internet address that are easier for computers to manage.
Talos said the perpetrators of DNSpionage were able to steal email and other login credentials from a number of government and private sector entities in Lebanon and the United Arab Emirates by hijacking the DNS servers for these targets, so that all email and virtual private networking (VPN) traffic was redirected to an Internet address controlled by the attackers.
Talos reported that these DNS hijacks also paved the way for the attackers to obtain SSL encryption certificates for the targeted domains (e.g. webmail.finance.gov.lb), which allowed them to decrypt the intercepted email and VPN credentials and view them in plain text.
On January 9, 2019, security vendor FireEye released its report, “Global DNS Hijacking Campaign: DNS Record Manipulation at Scale,” which went into far greater technical detail about the “how” of the espionage campaign, but contained few additional details about its victims.
About the same time as the FireEye report, the U.S. Department of Homeland Security issued a rare emergency directive ordering all U.S. federal civilian agencies to secure the login credentials for their Internet domain records. As part of that mandate, DHS published a short list of domain names and Internet addresses that were used in the DNSpionage campaign, although those details did not go beyond what was previously released by either Cisco Talos or FireEye.
That changed on Jan. 25, 2019, when security firm CrowdStrike published a blog post listing virtually every Internet address known to be (ab)used by the espionage campaign to date. The remainder of this story is based on open-source research and interviews conducted by KrebsOnSecurity in an effort to shed more light on the true extent of this extraordinary — and ongoing — attack.
The “indicators of compromise” related to the DNSpionage campaign, as published by CrowdStrike.
PASSIVE DNS
I began my research by taking each of the Internet addresses laid out in the CrowdStrike report and running them through both Farsight Security and SecurityTrails, services that passively collect data about changes to DNS records tied to tens of millions of Web site domains around the world.
Working backwards from each Internet address, I was able to see that in the last few months of 2018 the hackers behind DNSpionage succeeded in compromising key components of DNS infrastructure for more than 50 Middle Eastern companies and government agencies, including targets in Albania, Cyprus, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Saudi Arabia and the United Arab Emirates.
For example, the passive DNS data shows the attackers were able to hijack the DNS records for mail.gov.ae, which handles email for government offices of the United Arab Emirates. Here are just a few other interesting assets successfully compromised in this cyber espionage campaign:
-nsa.gov.iq: the National Security Advisory of Iraq
-webmail.mofa.gov.ae: email for the United Arab Emirates’ Ministry of Foreign Affairs
-shish.gov.al: the State Intelligence Service of Albania
-mail.mfa.gov.eg: mail server for Egypt’s Ministry of Foreign Affairs
-mod.gov.eg: Egyptian Ministry of Defense
-embassy.ly: Embassy of Libya
-owa.e-albania.al: the Outlook Web Access portal for the e-government portal of Albania
-mail.dgca.gov.kw: email server for Kuwait’s Civil Aviation Bureau
-gid.gov.jo: Jordan’s General Intelligence Directorate
-adpvpn.adpolice.gov.ae: VPN service for the Abu Dhabi Police
-mail.asp.gov.al: email for Albanian State Police
-owa.gov.cy: Microsoft Outlook Web Access for Government of Cyprus
-webmail.finance.gov.lb: email for Lebanon Ministry of Finance
-mail.petroleum.gov.eg: Egyptian Ministry of Petroleum
-mail.cyta.com.cy: Cyta telecommunications and Internet provider, Cyprus
-mail.mea.com.lb: email access for Middle East Airlines
The passive DNS data provided by Farsight and SecurityTrails also offered clues about when each of these domains was hijacked. In most cases, the attackers appear to have changed the DNS records for these domains (we’ll get to the “how” in a moment) so that the domains pointed to servers in Europe that they controlled.
Shortly after the DNS records for these TLDs were hijacked — sometimes weeks, sometimes just days or hours — the attackers were able to obtain SSL certificates for those domains from SSL providers Comodo and/or Let’s Encrypt. The preparation for several of these attacks can be seen at crt.sh, which provides a searchable database of all new SSL certificate creations.
Let’s take a closer look at one example. The CrowdStrike report references the Internet address 139.59.134[.]216 (see above), which according to Farsight was home to just seven different domains over the years. Two of those domains only appeared at that Internet address in December 2018, including domains in Lebanon and — curiously — Sweden.
The first domain was “ns0.idm.net.lb,” which is a server for the Lebanese Internet service provider IDM. From early 2014 until December 2018, ns0.idm.net.lb pointed to 194.126.10[.]18, which appropriately enough is an Internet address based in Lebanon. But as we can see in the screenshot from Farsight’s data below, on Dec. 18, 2018, the DNS records for this ISP were changed to point Internet traffic destined for IDM to a hosting provider in Germany (the 139.59.134[.]216 address).
Source: Farsight Security
Notice what else is listed along with IDM’s domain at 139.59.134[.]216, according to Farsight:
The DNS records for the domains sa1.dnsnode.net and fork.sth.dnsnode.net also were changed from their rightful home in Sweden to the German hosting provider controlled by the attackers in December. These domains are owned by Netnod Internet Exchange, a major global DNS provider based in Sweden. Netnod also operates one of the 13 “root” name servers, a critical resource that forms the very foundation of the global DNS system.
We’ll come back to Netnod in a moment. But first let’s look at another Internet address referenced in the CrowdStrike report as part of the infrastructure abused by the DNSpionage hackers: 82.196.11[.]127. This address in The Netherlands also is home to the domain mmfasi[.]com, which Crowdstrike says was one of the attacker’s domains that was used as a DNS server for some of the hijacked infrastructure.
As we can see in the screenshot above, 82.196.11[.]127 was temporarily home to another pair of Netnod DNS servers, as well as the server “ns.anycast.woodynet.net.” That domain is derived from the nickname of Bill Woodcock, who serves as executive director of Packet Clearing House (PCH).
PCH is a nonprofit entity based in northern California that also manages significant amounts of the world’s DNS infrastructure, particularly the DNS for more than 500 top-level domains and a number of the Middle East top-level domains targeted by DNSpionage.
TARGETING THE REGISTRARS
Contacted on Feb. 14 by KrebsOnSecurity, Netnod CEO Lars Michael Jogbäck confirmed that parts of Netnod’s DNS infrastructure were hijacked in late December 2018 and early January 2019 after the attackers gained access to accounts at Netnod’s domain name registrar.
Jogbäck pointed to a statement the company published on its Web site on Feb. 5, which says Netnod learned of its role in the attack on January 2 and has been in contact with all relevant parties and customers throughout this process.
“As a participant in an international security co-operation, Netnod became aware on 2 January 2019 that we had been caught up in this wave and that we had experienced a MITM (man-in-the-middle) attack,” the statement reads. “Netnod was not the ultimate goal of the attack. The goal is considered to have been the capture of login details for Internet services in countries outside of Sweden.”
In an interview with this author on Feb. 15, PCH’s Woodcock acknowledged that portions of his organization’s infrastructure were compromised after the DNSpionage hackers abused unauthorized access to its domain name registrar.
As it happens, the registrar records for both pch.net and dnsnode.net point to the same sources: Key-Systems GmbH, a domain registrar based in Germany; and Frobbit.se, a company in Sweden. Frobbit is a reseller of Key Systems, and the two companies share some of the same online resources.
Woodcock said the hackers phished credentials that PCH’s registrar used to send signaling messages known as the Extensible Provisioning Protocol (EPP). EPP is a little-known interface that serves as a kind of back-end for the global DNS system, allowing domain registrars to notify the regional registries (like Verisign) about changes to domain records, including new domain registrations, modifications, and transfers.
“At the beginning of January, Key-Systems said they believed that their EPP interface had been abused by someone who had stolen valid credentials,” Woodcock said.
Key-Systems declined to comment for this story, beyond saying it does not discuss details of its reseller clients’ businesses.
Netnod’s written statement on the attack referred further inquiries to the company’s security director Patrik Fältström, who also is co-owner of Frobbit.se.
In an email to KrebsOnSecurity, Fältström said unauthorized EPP instructions were sent to various registries by the DNSpionage attackers from both Frobbit and Key Systems.
“The attack was from my perspective clearly an early version of a serious EPP attack,” he wrote. “That is, the goal was to get the right EPP commands sent to the registries. I am extremely nervous personally over extrapolations towards the future. Should registries allow any EPP command to come from the registrars? We will always have some weak registrars, right?”
DNSSEC
One of the more interesting aspects of these attacks is that both Netnod and PCH are vocal proponents and adopters of DNSSEC (a.k.a. “DNS Security Extensions”), which is a technology designed to defeat the very type of attack that the DNSpionage hackers were able to execute.
Image: APNIC
DNSSEC protects applications from using forged or manipulated DNS data, by requiring that all DNS queries for a given domain or set of domains be digitally signed. In DNSSEC, if a name server determines that the address record for a given domain has not been modified in transit, it resolves the domain and lets the user visit the site. If, however, that record has been modified in some way or doesn’t match the domain requested, the name server blocks the user from reaching the fraudulent address.
While DNSSEC can be an effective tool for mitigating attacks such as those launched by DNSpionage, only about 20 percent of the world’s major networks and Web sites have enabled it, according to measurements gathered by APNIC, the regional Internet address registry for the Asia-Pacific region.
Jogbäck said Netnod’s infrastructure suffered three separate attacks from the DNSpionage attackers. The first two occurred in a two-week window between Dec. 14, 2018 and Jan. 2, 2019, and targeted company servers that were not protected by DNSSEC.
However, he said the third attack between Dec. 29 and Jan. 2 targeted Netnod infrastructure that was protected by DNSSEC and serving its own internal email network. Yet, because the attackers already had access to its registrar’s systems, they were able to briefly disable that safeguard — or at least long enough to obtain SSL certificates for two of Netnod’s email servers.
Jogbäck told KrebsOnSecurity that once the attackers had those certificates, they re-enabled DNSSEC for the company’s targeted servers while apparently preparing to launch the second stage of the attack — diverting traffic flowing through its mail servers to machines the attackers controlled. But Jogbäck said that for whatever reason, the attackers neglected to use their unauthorized access to its registrar to disable DNSSEC before later attempting to siphon Internet traffic.
“Luckily for us, they forgot to remove that when they launched their man-in-the-middle attack,” he said. “If they had been more skilled they would have removed DNSSEC on the domain, which they could have done.”
Woodcock says PCH validates DNSSEC on all of its infrastructure, but that not all of the company’s customers — particularly some of the countries in the Middle East targeted by DNSpionage — had configured their systems to fully implement the technology.
Woodcock said PCH’s infrastructure was targeted by DNSpionage attackers in four distinct attacks between December 13, 2018 and January 2, 2019. With each attack, the hackers would turn on their password-slurping tools for roughly one hour, and then switch them off before returning the network to its original state after each run.
The attackers didn’t need to enable their surveillance dragnet longer than an hour each time because most modern smartphones are configured to continuously pull new email for any accounts the user may have set up on his device. Thus, the attackers were able to hoover up a great many email credentials with each brief hijack.
On Jan. 2, 2019 — the same day the DNSpionage hackers went after Netnod’s internal email system — they also targeted PCH directly, obtaining SSL certificates from Comodo for two PCH domains that handle internal email for the company.
Woodcock said PCH’s reliance on DNSSEC almost completely blocked that attack, but that it managed to snare email credentials for two employees who were traveling at the time. Those employees’ mobile devices were downloading company email via hotel wireless networks that — as a prerequisite for using the wireless service — forced their devices to use the hotel’s DNS servers, not PCH’s DNNSEC-enabled systems.
“The two people who did get popped, both were traveling and were on their iPhones, and they had to traverse through captive portals during the hijack period,” Woodcock said. “They had to switch off our name servers to use the captive portal, and during that time the mail clients on their phones checked for new email. Aside from that, DNSSEC saved us from being really, thoroughly owned.”
Because PCH had protected its domains with DNSSEC, the practical effect of the hijack against its mail infrastructure was that for roughly an hour nobody but the two remote employees received any email.
“For essentially all of our users, what it looked like was the mail server just wasn’t available for a short period,” Woodcock said. “It didn’t resolve for a while if they happened to be checking their phone or whatever, and each person thought well that’s funny, I’ll check it back in a while. And by the time they checked again it was working fine. A bunch of our staff noticed a brief outage in our email service, but nobody thought enough of it to discuss it with anyone else or open a ticket.”
But the DNSpionage hackers were not deterred. In a letter to its customers sent earlier this month, PCH said a forensic investigation determined that on Jan. 24 a computer which holds its Web site user database had been compromised. The user data stored in the database included customer usernames, bcrypt password hashes, emails, addresses, and organization names.
“We see no evidence that the attackers accessed the user database or exfiltrated it,” the message reads. “So we are providing you this information as a matter of transparency and precaution, rather than because we believe that your data was compromised.”
IMPROVEMENTS
Multiple experts interviewed for this story said one persistent problem with DNS-based attacks is that a great deal of organizations tend to take much of their DNS infrastructure for granted. For example, many entities don’t even log their DNS traffic, nor do they keep a close eye on any changes made to their domain records.
Even for those companies making an effort to monitor their DNS infrastructure for suspicious changes, some monitoring services only take snapshots of DNS records passively, or else only do so actively on a once-daily basis. Indeed, Woodcock said PCH relied on no fewer than three monitoring systems, and that none of them alerted his organization to the various one-hour hijacks that hit PCH’s DNS systems.
“We had three different commercial DNS monitoring services, none of which caught it,” he said. “None of them even warned us that it had happened after the fact.”
Woodcock said PCH has since set up a system to poll its own DNS infrastructure multiple times each hour, and to alert immediately on any changes.
Jogbäck said Netnod also has beefed up its monitoring, as well as redoubled efforts to ensure that all of the available options for securing their domain infrastructure were being used. For instance, the company had not previously secured all of its domains with a “domain lock,” a service that requires a registrar to take additional authentication steps before making any modifications to a domain’s records.
“We are really sad we didn’t do a better job of protecting our customers, but we are also a victim in the chain of the attack,” Jogbäck said. “You can change to a better lock after you’ve been robbed, and hopefully make it more difficult for someone to do it again. But I can truly say we have learned a tremendous amount from being a victim in this attack, and we are now much better off than before.”
Woodcock said he’s worried that Internet policymakers and other infrastructure providers aren’t taking threats to the global DNS seriously or urgently enough, and he’s confident the DNSpionage hackers will have plenty of other victims to target and exploit in the months and years ahead.
“All of this is a running battle,” he said. “The Iranians are not just trying to do these attacks to have an immediate effect. They’re trying to get into the Internet infrastructure deeply enough so they can get away with this stuff whenever they want to. They’re looking to get as many ways in as possible that they can use for specific goals in the future.”
RECOMMENDATIONS
John Crain is chief security, stability and resiliency officer at ICANN, the non-profit entity that oversees the global domain name industry. Crain said many of the best practices that can make it more difficult for attackers to hijack a target’s domains or DNS infrastructure have been known for more than a decade.
“A lot of this comes down to data hygiene,” Crain said. “Large organizations down to mom-and-pop entities are not paying attention to some very basic security practices, like multi-factor authentication. These days, if you have a sub-optimal security stance, you’re going to get owned. That’s the reality today. We’re seeing much more sophisticated adversaries now taking actions on the Internet, and if you’re not doing the basic stuff they’re going to hit you.”
Some of those best practices for organizations include:
-Use DNSSEC (both signing zones and validating responses)
-Use registration features like Registry Lock that can help protect domain names records from being changed
-Use access control lists for applications, Internet traffic and monitoring
-Use 2-factor authentication, and require it to be used by all relevant users and subcontractors
-In cases where passwords are used, pick unique passwords and consider password managers
-Review accounts with registrars and other providers
-Monitor certificates by monitoring, for example, Certificate Transparency Logs
Awesome as always. Thanks Brian. Just out of curiosity, how would you relate DNSSEC and the attacks you lay out here, to DNS cache poisoning, as described at GRC.com, and Steve Gibson’s “DNS Spoof” testing tool?
https://www.grc.com/dns/dns.htm
DNS cache poisoning most often happens at the more local level, i.e. at the user’s machine or router. The attacks profiled in this story involve compromising DNS settings at a far higher level, one that goes well beyond the control of the end user.
“The Iranians are not just trying to do these attacks to have an immediate effect”.
I have looked through the article, but I cannot find any evidence that points to “Iranians” as the perpetrators.
Is there any, or is this just more political character assassination?
Check out the CrowdStrike blog post linked in the story. I did not print a lot of the information I have, mainly because the story was already very long, and in any case this is likely not the last one I will write on the topic. But also, I tried not to give away too many details that would help the attackers in the future to improve their attacks. That said, it was clear from my interviews with many people not named in this story that there are multiple clues pointing to a fairly significant effort by some specific threat actors.
Careful with CrowdStrike, they are just a private wing of the FBI… look at all the connections to the FBI, it is scary.
Look at how they handled the DNC hack and didn’t stop it for what 19 days or something insane but yet only took what 2 hours to identify “the Bears”.. scary.
CrowdStrike will be exposed one of these days….
Please no conspiracy theories and other nonsense. Someone needs to lay off the AM radio and Maga hats.
“Please no conspiracy theories and other nonsense.” Seems as though you need to heed your own advice. No need to throw MAGA hats into this.
Why all the noise about MAGA (Make America Gay Again) hats?
No MAGA hat here (don’t/won’t own one) and no conspiracy just facts from CrowdStrike own reports… but hey if you want to act like a fool be my guest 🙂
Also educate yourself on who works at CrowdStrike… Most C-levels are ex-FBI and also look at the co-founders backgrounds… not exactly clean.
One simple example for you that is straight facts that cannot be argued with.
Steven Chabinsky is “General Counsel and Chief Risk Officer” for the cybersecurity technology firm CrowdStrike. Chabinsky spent 15 years working for the FBI prior to working at
CrowdStrike. Chabinsky joined the FBI in 1995 as an attorney in the Office of the General Counsel
where he initially focused on employment law and personnel litigation. In 1998, Chabinsky was
selected as the Principal Legal Advisor to the multi-agency National Infrastructure Protection
Center (NIPC) and became Senior Counsel to the FBI’s Cyber Division upon its creation in 2002.
While at the FBI, Chabinsky was:
– Deputy assistant director of the FBI’s Cyber division
-Focused on protecting the United States from cyber-attack, cyber
espionage, online child exploitation, and Internet fraud.
-Homeland Security Act of 2002
-National Strategy to Secure Cyberspace of 2003
-National Strategy to Secure Cyberspace of 2008
-National Security Presidential Directive 54
– Between 2007 and 2009, Chabinsky served in the Office of the Director of National
Intelligence (ODNI) in various capacities, including Acting Assistant Deputy Director of
National Intelligence for Cyber, Chairman of the National Cyber Study Group, and Director
of the Joint Interagency Cyber Task Force.
-April 13, 2016 Chabinsky was presidentially appointed to the White House Commission
on Enhancing National Cybersecurity. (Executive order 13718) The Commission provided
recommendations to the President to strengthen cybersecurity in the public and private
sectors, while protecting privacy, fostering innovation and ensuring economic and national
security.
(source https://obamawhitehouse.archives.gov/the-press-office/2016/04/13/president-obama-announces-more-key-administration-posts)
Lets be clear, there are “conspiracies”, where people with power gather. The idea there “arent any” is just weird.
Being picky about who you buy your stuff and your data from is a duty you have in the modern world.
Crowdstrike? And they conclude it’s the main US target for regime change in that region of the world? After their handling of the DNC servers I don’t trust them any further than I can throw them. They are like the CIA, you have to assume they are lying unless they have hard facts to back it up.
Not that I would put it past Iran to do something like this but there is another country that should be on there..but it’s not
“I have looked through the article, but I cannot find any evidence that points to “Iranians” as the perpetrators.
Is there any, or is this just more political character assassination?”
LOL; do you honestly think Brian is committing political character assassination? Unless you don’t work in the security space, your comment is laughable. The Chinese, Iranians and Russians are always at it Ahmed, I mean, Tom Welsh.
hahahaha, ahmad, im dead!!!!
Excellent research Brian. I hope the 20% that uses DNSSec is all in the USA. As a Cyber Security Analyst, there is a reason for any anomaly. If your mail exchange is broken for an hour, investigate it. We all need to do more than protect our systems. We need to verify where our data is going out to also.
I know with my my domain name, I use 2FA to access the website interface, along with DNSSEC and Registry Lock is enabled
“However, he said the third attack between Dec. 29 and Jan. 2 targeted Netnod infrastructure that was protected by DNSSEC and serving its own internal email network. Yet, because the attackers already had access to its registrar’s systems, they were able to briefly disable that safeguard “.
That doesn’t make sense. It should not be possible to disable DNSSEC by turning off DNSSEC on the server zone. For DNSSEC to provide any value, it requires configuration for both DNS servers and enforcement on client computers. On Windows, client enforcement is performed using the Name Resolution Policy Table (NRPT). If DNSSEC is not enforced on the client, it doesn’t really provide any value. If DNSSEC is enforced on the client, disabling DNSSEC for the zone(s) on a DNS server would cause a name resolution failure on the client. If any of the victim organizations had DNSSEC-enabled domains but did not enforce DNSSEC on their client computers they shot themselves in the foot.
All it takes to kill DNSSEC is to get rid of the DS record. I’ve had to walk many a company through why their brand new DNSSEC configuration didn’t really work and it was almost always because they had no corresponding DS record. People just don’t do their homework anymore: “But we signed the zone!”
Cached DS records for the parent would likely be in resolvers for a while, and thus would cause DNSSEC failures for clients that were validating. In particular, if CAs issuing DV certs were doing DNSSEC validation as recommended in the ACME spec, they should have (a) not issued the CERTS and (b) raised alarms during this period.
The mechanisms are there to prevent these types of attacks, but you have to use them.
https://tools.ietf.org/html/draft-ietf-acme-acme-18#section-11.2
“Why CISA issued our first Emergency Directive”
By Christopher Krebs, Director
Any relation?
Nope
Iran & Russia seems to be getting prepare for cyber warfare.
Hope our government (Home Land Security, FBI, CIA) will protect our nation from this. I really wish & pray that President Trump would asked Brian Krebs to be the head of our government cyber terrorist Department!
“Hope our government (Home Land Security, FBI, CIA) will protect our nation from this.”
You hope in vain. We’re on our own.
The problem is, nobody wants the FBI or DHS coming in and instructing them how to pick good passwords or how not to get phished. They can offer help, but they don’t have the legal authority to force anybody in the private sector to not be idiots. And none of us would want them to have that anyway.
So everyone is responsible for locking up their own bicycles. That’s just the way it is, and the way it will be.
185.20.187[.]8 blocked, unknown why, by Forcepoint, Fortinet, Kaspersky (etc)
2019-02-15
http://www.virustotal.com/en/url/0054576de4b5c983f49d77046800ed029b9d3aebdc6334261386c9aac4be39a5/analysis
2018-12-03
http://www.virustotal.com/en/url/55263f4edcd02ab7e9c391f1a36c4e84ff9868133a40020e9db2046fba42a11f/analysis
Talos: “The attackers redirected the hostnames to the IP 185.20.187[.]8 for a short time.” Nov. 6. Sept. 13, etc.
C2 (Command and control) Server, along with two other IPs (all on same Netherlands network).
It amazes me that we could fail to practice data hygiene that was recommended years ago! I mean, DNSSEC is pretty old. Registry Lock has been available for some time as well.
Multi-factor Authentication is a pretty new trend but it’s a key part to the cyber kill chain.
Hope we learn from this.
Registry Lock is actually not that well known and people confuse it with the transfer lock. Add in the fact that it costs money and you have your answer.
What do the brackets [ ] around the dot of the last octet of an IP address mean?
They are designed to prevent web crawlers from crawling into those (criminal) servers. And to prevent mobile web browsers from turning them into links, which could end up tricking innocent readers into visiting the sites.
The [.] is intended to ensure that the link doesn’t become clickable. Therefore it is a protection mechanism for the readers of this site (among other things)
Probably to keep the IP address from displaying as a link in most browsers. A mosfeature that may be a problem when discussing malicious sites as Brian does.
And thanks for another great article, Brian!
Thanks. Yes, as others have said, the brackets effectively prevent any site, browser or email clients from making these addresses into a clickable link.
I would add to the list of practices to get certificates that contain proof of identity, not just a domain.
A certificate that says “this is example.com” is meaningless in the face of a DNS hijack.
One that says “Example Corporation, 1234 Fake Street, Santa Clara, CA” and comes from an authority that requires proof of identity documents is going to be more secure. Especially if users are used to seeing the full name in the location bar and it suddenly changes to just “example.com” at some point.
Good idea!
Too bad you can’t set “levels” of certificate checking in any client software. It would be nice to only allow certs for a list of root domain names to be Organization Validated (OV) or Extended Validated (EV).
Some browser add-on software allows cert pinning, but that gets annoying quickly with many legitimate certs that often rotate.
Better would be to say “only allow OV and above for this domain”.
No indicator of how end users protect themselves. I just did a repave after watching DNS issues for quite some time. For once I did not use my DNS rotator script – I have a script that picks a random dns server at random intervals. Brutal but seems to prevent obvious dns malfeasance. Bit fed up with all of this, the internet is deliberately kept borken IMHO. Iran should have sanctions imposed, such as removing all Iranian students/professors from Universities until they learn to be good citizens. Same with Chinese etc… LOL look at any ‘sensitive’ workplace and the names of the shakers and movers, all vetted perfect too… State sponsored stooges and spies in all the most sensitive of places, while domestic students of equal or better ability have to struggle with debt and loan, if they can find a place at all! stupidity of the highest order!
It is somewhat more complicated for end-users. You can set your DNS settings to something like Google’s DNS — 8.8.8.8 and 4.4.2.2 — which should enforce DNNSEC when the domain signs it as such. But things can get trickier when you are not on your own network, and your mobile data provider’s DNS settings kick in. I’m not sure there is a way for end users to control what DNS settings get used in that case.
>>I’m not sure there is a way for end users to control what DNS settings get used in that case.
1.1.1.1 offers their Android app that may address that question to a degree. Wouldn’t VPNs (in particular those that permit the user to use a DNS server of their choice) put the question to bed, concerning the use of insecure (e.g., hotel) public networks?
Actually, no. Say you are at Hilton, and their system has been hacked. To access the web, you have to sign in to Hilton, a cookie is processed on your device. Is that Hilton’s cookie or the bad actors modified cookie? That cookie modifies your system to be part of their system. What did they modified in your system?
Good article, I liked it. Did not like the identification of the host, but, everyone points back to lack of security protocals for visiting outside of your area. Especially tethering of phone to mobile phone services. More secure then place you are staying systems. But there are bad actors in phone systems also.
“…when you are not on your own network, and your mobile data provider’s DNS settings kick in. I’m not sure there is a way for end users to control what DNS settings get used in that case.”
Changing to another network provider is an option, if the user influences the paying of the network provider like is the case for network provider service to a personal device. Which may entail changing to another device though i.e. Verizon, AT&T, Sprint, T-Mobile, etc.
I think Brian was referencing WiFi hotspots more so than cellular data providers.
I keep mobile data turned off. Simplest solution.
SMS and calling work without mobile data, out on the road. Everything else can wait until my device is on a trusted wifi network, configured to use a DNS filtering service.
It has the side effect of making my phone bill cheap and my life less stressful.
DNS over HTTPS/TLS [1] can help some portion of things.
Getting mail clients to not eagerly connect to links while they’re only partially established is a real problem. It was a problem a decade ago. Probably the easiest solution is to not use email clients (i.e. use a browser).
The other half of the solution is to rely on a major vendor for email (i.e. Google [Gmail]/MS [Office365]). By relying on a major vendor to host your mail on their normal domains you’re putting your eggs into a basket that has better scrutiny than your average bear. When someone hijacks google.com or gmail.com, people around the world will notice. When someone hijacks your-localized-domain-that-only-5-people-use.com, most people won’t notice or care. Also, both vendors have 2FA and other best practices and support enforcing them.
[1] https://developers.google.com/speed/public-dns/docs/dns-over-tls
“I’m not sure there is a way for end users to control what DNS settings get used in that case.”
Of course there is. They just make it obscure and complicated to diddle that so as to keep ordinary stupid end-lusers from hurting themselves.
If you know what you are doing, you can usually accomplish just about anything.
I hear where you’re coming from. I’ve worked as a consultant in several places that were dominated by Visa immigrants from Asia. They’ve kind of done to the white collar (IT anyways) what the Latinos have done to the blue collar workers. I have nothing against them personally, but it irks me that employers are cutting corners this way and American citizens are either unemployed or have to take huge cuts in wages to land a job.
On the DNS issue, I manually set my Android DNS settings to something more secure on my home and other known networks, but on a public network, there is no way to do that (perhaps a 3rd party app?).
I’m an German/Irish-American and my wife is Iranian, her parents were born in Iran, she was born actually in another country and moved here as an infant, and they are all legal citizens. She is an art teacher and is really bad at technology, and had nothing to do with this Her family is very nice and they all love this country, having lived here for over 30 years. Sanctioning them or other Iranians living and working in the Untied States would be similar to what we did with the Japanese internment during WW2.
I urge people to remember to separate the people in power of a government ordering such attacks, and those that live under that rule, especially in an oppressed society.
Reverse example:
The United States (Government, i.e. the military, or the CIA) carries out a cyber attack on another country, let’s say Russia. In Russia, they frame this as “The U.S. (as a whole) have attacked us, and any and all American citizens are a part of this.” Any Americans living in Russia are fired, kicked out of universities, deported, or even possibly jailed and their property seized.
Would that be fair the 30,000 Americans that live in Russia, even though they had nothing to do with the attack, just to send a message to the government of the place where they are citizens?
Sure, but when people are looking for pretexts for future actions, those niceties and distinctions go by the wayside. So, instead of the nuanced and reasonable “suspected Iranian government-backed hackers”, you get the more direct and visceral “Iranians”. Bolton and company are very grateful as they plot their assault on Iran.
Defending DNS is of course important and good. However to the point of users protecting themselves, this is fundamentally a question of: am I connected to the server where I established my account?
We can use Yubikey or equivalent for the server to authenticate the user. *BUT WE NEED* a reverse Yubikey, where my key also confirms the server is the one where I established my account.
I believe the technology of a Yubikey is applicable here. But the user experience is AWOL!! I want to plug in my key and ask the key+browser to confirm: based on encrypted exchange, the server I are communicating with is indeed in possession of the encryption key I previously established there.
Please!!!
More organizations should deploy DNSSEC but most don’t understand how it works and why it is valuable.
Maybe this will help the education issue:
Secure Domain Name System (DNS) Deployment Guide
https://csrc.nist.gov/publications/detail/sp/800-81/2/final
Very interesting! No wonder because this is related to it:
https://blog.powerdns.com/2018/03/22/the-dns-camel-or-the-rise-in-dns-complexit/
Very interesting post!
I mean some of the people working at Netnod are extremely skilled and experienced in the infrastructure of the Internet but they still got hacked.
On a sadder note; I believe that the registrars are the weakest link. Many of them still don’t even support MFA!
agreed
Why isn’t CAA records listed as a recommendation? If PCH and Netnod had DNS CAA records set, to say Digicert, then the attacker would not have been able to issue a certificate from Let’s Encrypt.
CAA fields are a good idea, but once someone controls your DNS, they can replace the CAA, so unless you’ve managed to get your CAA field cached *everywhere* for a *long* time, you’re still in trouble.
Crazy, especially if this is the sign of things to come.
Having EPP credentials alone, would not suffice. They also would have to have the certificate a registrar uses to send messages and would need to be sending the messages from the proper IP addresses.
Aside from 2FA, did any of the affected have a privileged account management solution in place? It seems like modifying root DNS records ought to require more approval than a username and password and ought to be strictly audited and reviewed.
That’s exactly where I’m at! Some very big DNS providers (dinosaurs of the industry) don’t even offer 2 factor. How is that possible!?! If there’s something that should absolutely have 2-factor in front of it, this is one of them. No MFA, no 3rd party authentication hooks to get MFA from an ID provider, no logs provided except for on demand and no alerting when things change. Having to write a script to go do digging and look for a diff in results from last time is a good option outside of DNS sec to notice. You would think at this point, these guys would provide MFA, even turn it on by default but they won’t. Sad state of affairs.
Hey Brian, thanks for shedding light on this!
Question, http://www.cnss.gov has been down for several weeks.
Would you happen to have any info on that? I cannot find info anywhere discussing this site being down. Honestly, I’m pretty sure it went down during the shutdown, or right after it ended….which is suspect.
ok..it’s like they said “uh oh, someone noticed”
It was out this morning before I sent my comment…and now it is back. I’ll totally call that a coincidence.
Seems to have an invalid SSL cert….
For people wondering what that server was: https://web.archive.org/web/20181107213502/https://www.cnss.gov/cnss/
Glad to see this get some of the substantive even handed coverage it deserves. It was kind of weird that this was all out there in passive DNS and cert transparancy log but there was no public debate yet which if we didn`t have it would be a real missed opportunity.
> I Am extremely nervous personally over extrapolations towards the future
Yeah, but is the operator of one of the 13 root servers nervous enough? Sure middle eastern governments and the US government are gonna be spied on, people get killed based on that information and geopolitical balances shift based on this.. bad.. but you know, spies gonna spy.
But the “Shamoon 1” “Shamoon 2” and “Shamoon 3” actor suspected of ties to Iran has been busy not just spying but sabotaging the oil sector left and right, with no regard for western collateral damage.
I am not saying netnod was pwned more than they transparently explained in their press release… but lets ask ,hypothetically, are we okay with the idea of Iran having silently pwned a root server? What if some Israeli/US politician wants to distract from some domestic stuff with some random symbolic bombing because “backwards Iranians cant retaliate anyway”?
What if one in 13 root server networks suddenly starts giving bogus answers that get cached for hours? Sure its one worst case scenario, but with more and more stuff casually moving on-line, if-only as VPNs that connect to names not IP`s, how bad are we talking about?
Two more good tips:
The google story, is a nice long TL;DR description of a great initiative… but here is the nickle summary, if you are responsible for any domain more serious than nyan.cat pop a RSS/atom link like this
https://crt.sh/atom?q=krebsonsecurity.com
in your your outlook, or phone or browser or SOC software security update feedreader. Probably nothing will happen for years, but maybe one day you get a message on a certificate you didn’t request yourself.
And don’t just follow the excellent tips yourself, if you run into someone who runs something critical give them a little audit before your next face-to-face meeting and bring it up, friends don’t let friends miss out on DNSSEC.
Up next, a quick fix for BGP, should be easy enough right?
http://www.saipem.com/sites/SAIPEM_en_IT/con-side-dx/Press%20releases/2018/Cyber%20attack%20update.page
https://www.energyvoice.com/oilandgas/188750/petrofac-confirms-system-security-breach/
+1
Thank you! What a great idea to use RSS+CERT feed. Very creative.
Are there recommended registrars and DNS providers that have all the features and a good track record on security?
I don’t have an answer for this.
Cloudfare was going to become a registrar, they come w/ some strict caveats, but they might not be terrible.
I’ve been looking into Gandi, they at least support 2FA.
I’d be interested in hearing answers to this question.
Fascinating article & great work as always!
One thing I didn’t see called out specifically was a name in the middle of a chart: nl.tunnelbear-ios.com. Was that IP actually part of /used by the tunnelbear VPN provider at some point? That could be another attack vector of interest…
i have some questions to all readers (posted the same question on schneier blog right now):
1-how can i find out if my dns has dns-sec enabled/is supported?
now, supposing that i’m using a dns-sec enabled server
2-the client side (windows) must support dns-sec to be used right, so does windows support dnssec?
3-it’s something system-wide so that if windows supports this it’s “extended” to all the apps or every app must support it? if it must be app-level, firefox and thunderbird support it? (without plugins)
4-is it secure that just works like tls certificates that can’t be abused or it’s a bad solution? (i know that tls certs can be abused too but misinsurance can be mitigated, noticed and it’s rare; private key theft it’s not a tls problem; and if someone steal your domain and gets a legit cert it’s again not a tls/ca infrastructure problem it’s outside scope; if you click ignore on the warning “someone is intercepting the connection” it’s your fault)
5-i know that in tls if someone mess with the connection i will see a warning, they can’t intercept my data without my consent, the “best” attack they can do is dos (because warning or dropping traffic on port 443). what about dns-sec? what it aim to prevent? mitm?
6-what is the scope of dns-sec? tls is data integrity, autenticity and secrecy
tl;dr: i’d like to know more about dns-sec because to me looks like something obscure and i can’t find easy explained informations on the internet
thanks a lot for any answer
1. https://dnssec.vs.uni-due.de/
2. Yes
Here’s a place to check any domain for DNSSEC and also many other free net tools: https://viewdns.info/
Q ? == Why doesn’t Amazon’s Route 53’s DNS service ALSO support DNSSEC?? Must be a great reason, but I haven’t been able to find it. Here’s a quite today from their support site for DNSSEC:
“Route 53 supports DNSSEC for domain registration but does not support DNSSEC for DNS service. If you want to configure DNSSEC for a domain that is registered with Route 53, you must use another DNS service provider.”
— Source: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-configure-dnssec.html
“route 53” I suspect is more commonly referred to as a port (which is a route of sorts):
http://www.grc.com/port_53.htm
3. in general apps get the system resolver unless they choose to work around it. Firefox is the kind of thing that is likely to work around the system resolver.
There are plugins [1] for firefox
4. If the registrar is hacked or the credentials for the registrar are hacked or if the canonical DNS provider is hacked, DNSSEC won’t help much. It isn’t a terrible solution, but it isn’t a silver bullet.
5. Afaiu DNSSEC is more or less form of MITM protection. It ensures that the DNS data your client is receiving matches data that the official DNS server (according to the registrar) authored. And yes, DNSSEC like most things can result in DoS if you configure your system to fail-secure.
6. DNSSEC afaiu is integrity and authenticity, but not secrecy [2]. In fact, by nature, it leaks some information about the shape of your DNS records.
For secrecy, you should look at DNS over TLS or DNS over HTTPS.
[1] https://addons.mozilla.org/en-US/firefox/addon/dnssec/
[2] https://www.cloudflare.com/dns/dnssec/dnssec-complexities-and-considerations/
Thanks for your answer. But, if the end users DNS resolver doesn’t validates DNSSEC signatures how can DNSSEC help to make sure that users are connecting to the good ip address? Also, is the dnssec the only way for website owners / sys admins to mitigate that attack?
Thanks.
thanks to everyone for the answers.
i’m interested in this.
i guess that even if i set in windows 1.1.1.1 as dns windows doesn’t validate dnssec, i think i’ll have to configure dns over tls/https (tls should be better because has fewer layers/overhead) using a program and set system resolver as 127.0.0.1 in this way everything should be fine and mitm should not be possible
Every so often, somebody will email me and ask me to send them my PGP key. I don’t, because I don’t have one.
I don’t have one because I think of the notions of “email” and “security” as being fundamentally unrelated and totally orthogonal to one another. And this whole story from Brian just kind of reconfirms that belief for me. If I have something (anything) which is seriously confidential and that I either need to tell or hear, I’ll do it via voice over my landline, thank you very much. I mean how many million times do we need to read in the press about somebody having their emails copied illicitly before people wise up and realize that there are about eight gazillion potential ways that somebody can hack and get hold of your emails and that it is only when literally EVERYTHING is locked down as tight as a drum that you have even a prayer in hell of having your emails remain confidential?
PGP’s about as good as it gets. If you adequately protect your secret key. And if your correspondents are equally careful about ensuring that the public key they have really belongs to you.
Yeah … we’re all screwed.
s/regional registries/TLD registries/
Hi Brian,
I’ve been re-reading your article. What I fail to understand is the motivation to modify A records for name servers of NETNOD and PCH. I understand the modus operandi, but most domain names you refer to, do not use NETNOD nor PCH. So why changing those name servers?
“…the global “Domain Name System,” which serves as a kind of phone book for the Internet”. I emphisis “global phone book”.
It’s not only name servers that were affected.