A visualization of the Internet made using network routing data. Image: Barrett Lyon, opte.org.
Imagine being able to disconnect or redirect Internet traffic destined for some of the world’s biggest companies — just by spoofing an email. This is the nature of a threat vector recently removed by a Fortune 500 firm that operates one of the largest Internet backbones.
Based in Monroe, La., Lumen Technologies Inc. [NYSE: LUMN] (formerly CenturyLink) is one of more than two dozen entities that operate what’s known as an Internet Routing Registry (IRR). These IRRs maintain routing databases used by network operators to register their assigned network resources — i.e., the Internet addresses that have been allocated to their organization.
The data maintained by the IRRs help keep track of which organizations have the right to access what Internet address space in the global routing system. Collectively, the information voluntarily submitted to the IRRs forms a distributed database of Internet routing instructions that helps connect a vast array of individual networks.
There are about 70,000 distinct networks on the Internet today, ranging from huge broadband providers like AT&T, Comcast and Verizon to many thousands of enterprises that connect to the edge of the Internet for access. Each of these so-called “Autonomous Systems” (ASes) make their own decisions about how and with whom they will connect to the larger Internet.
Regardless of how they get online, each AS uses the same language to specify which Internet IP address ranges they control: It’s called the Border Gateway Protocol, or BGP. Using BGP, an AS tells its directly connected neighbor AS(es) the addresses that it can reach. That neighbor in turn passes the information on to its neighbors, and so on, until the information has propagated everywhere [1].
A key function of the BGP data maintained by IRRs is preventing rogue network operators from claiming another network’s addresses and hijacking their traffic. In essence, an organization can use IRRs to declare to the rest of the Internet, “These specific Internet address ranges are ours, should only originate from our network, and you should ignore any other networks trying to lay claim to these address ranges.”
In the early days of the Internet, when organizations wanted to update their records with an IRR, the changes usually involved some amount of human interaction — often someone manually editing the new coordinates into an Internet backbone router. But over the years the various IRRs made it easier to automate this process via email.
For a long time, any changes to an organization’s routing information with an IRR could be processed via email as long as one of the following authentication methods was successfully used:
-CRYPT-PW: A password is added to the text of an email to the IRR containing the record they wish to add, change or delete (the IRR then compares that password to a hash of the password);
-PGPKEY: The requestor signs the email containing the update with an encryption key the IRR recognizes;
-MAIL-FROM: The requestor sends the record changes in an email to the IRR, and the authentication is based solely on the “From:” header of the email.
Of these, MAIL-FROM has long been considered insecure, for the simple reason that it’s not difficult to spoof the return address of an email. And virtually all IRRs have disallowed its use since at least 2012, said Adam Korab, a network engineer and security researcher based in Houston.
All except Level 3 Communications, a major Internet backbone provider acquired by Lumen/CenturyLink.
“LEVEL 3 is the last IRR operator which allows the use of this method, although they have discouraged its use since at least 2012,” Korab told KrebsOnSecurity. “Other IRR operators have fully deprecated MAIL-FROM.”
Importantly, the name and email address of each Autonomous System’s official contact for making updates with the IRRs is public information.
Korab filed a vulnerability report with Lumen demonstrating how a simple spoofed email could be used to disrupt Internet service for banks, telecommunications firms and even government entities.
“If such an attack were successful, it would result in customer IP address blocks being filtered and dropped, making them unreachable from some or all of the global Internet,” Korab said, noting that he found more than 2,000 Lumen customers were potentially affected. “This would effectively cut off Internet access for the impacted IP address blocks.”
The recent outage that took Facebook, Instagram and WhatsApp offline for the better part of a day was caused by an erroneous BGP update submitted by Facebook. That update took away the map telling the world’s computers how to find its various online properties.
Now consider the mayhem that would ensue if someone spoofed IRR updates to remove or alter routing entries for multiple e-commerce providers, banks and telecommunications companies at the same time.
“Depending on the scope of an attack, this could impact individual customers, geographic market areas, or potentially the [Lumen] backbone,” Korab continued. “This attack is trivial to exploit, and has a difficult recovery. Our conjecture is that any impacted Lumen or customer IP address blocks would be offline for 24-48 hours. In the worst-case scenario, this could extend much longer.”
Lumen told KrebsOnSecurity that it continued offering MAIL-FROM: authentication because many of its customers still relied on it due to legacy systems. Nevertheless, after receiving Korab’s report the company decided the wisest course of action was to disable MAIL-FROM: authentication altogether.
“We recently received notice of a known insecure configuration with our Route Registry,” reads a statement Lumen shared with KrebsOnSecurity. “We already had mitigating controls in place and to date we have not identified any additional issues. As part of our normal cybersecurity protocol, we carefully considered this notice and took steps to further mitigate any potential risks the vulnerability may have created for our customers or systems.”
Level3, now part of Lumen, has long urged customers to avoid using “Mail From” for authentication, but until very recently they still allowed it.
KC Claffy is the founder and director of the Center for Applied Internet Data Analysis (CAIDA), and a resident research scientist of the San Diego Supercomputer Center at the University of California, San Diego. Claffy said there is scant public evidence of a threat actor using the weakness now fixed by Lumen to hijack Internet routes.
“People often don’t notice, and a malicious actor certainly works to achieve this,” Claffy said in an email to KrebsOnSecurity. “But also, if a victim does notice, they generally aren’t going to release details that they’ve been hijacked. This is why we need mandatory reporting of such breaches, as Dan Geer has been saying for years.”
But there are plenty of examples of cybercriminals hijacking IP address blocks after a domain name associated with an email address in an IRR record has expired. In those cases, the thieves simply register the expired domain and then send email from it to an IRR specifying any route changes.
While it’s nice that Lumen is no longer the weakest link in the IRR chain, the remaining authentication mechanisms aren’t great. Claffy said after years of debate over approaches to improving routing security, the operator community deployed an alternative known as the Resource Public Key Infrastructure (RPKI).
“The RPKI includes cryptographic attestation of records, including expiration dates, with each Regional Internet Registry (RIR) operating as a ‘root’ of trust,” wrote Claffy and two other UC San Diego researchers in a paper that is still undergoing peer review. “Similar to the IRR, operators can use the RPKI to discard routing messages that do not pass origin validation checks.”
However, the additional integrity RPKI brings also comes with a fair amount of added complexity and cost, the researchers found.
“Operational and legal implications of potential malfunctions have limited registration in and use of the RPKI,” the study observed (link added). “In response, some networks have redoubled their efforts to improve the accuracy of IRR registration data. These two technologies are now operating in parallel, along with the option of doing nothing at all to validate routes.”
[1]: I borrowed some descriptive text in the 5th and 6th paragraphs from a CAIDA/UCSD draft paper — IRR Hygiene in the RPKI Era (PDF).
Further reading:
Trust Zones: A Path to a More Secure Internet Infrastructure (PDF).
switch to IPFS
It seems like a steep learning curve. https://docs.ipfs.io/how-to/configure-node/#profiles
How to make it more accessible?
Brian, thanks for the mentions, and for all that you do. Note, by “mandatory reporting of such breaches”, I meant the attacks Dan Geer talked about in his essay. That is, I believe we need mandatory reporting of cybersecurity attacks above “some severity threshold”, which should include reporting when BGP hijacks were a means to achieving the attack. The U.S. Congress recently proposed such a harm-based threshold ( https://aboutbgov.com/YIK ). Route hijacks are one of many means toward inducing such harm. But I do not believe route hijack events are the appropriate granularity for mandatory reporting.
-MAIL-FROM is not even authentication, it is only identification. I would assume some level of identification is involved with both -CRYPT-PW and -PGPKEY and the password or PGP key signed message would then be authentication.
I was not aware such a lack of authentication for such an important function was generally available until 2012 and still in use at all until now.
It seems to me that even -CRYPT-PW is questionable considering the importance of the function. I realize it my be a monumental task to convert all to -PGPKEY but sometimes you just gotta do what you gotta do.
Unfortunately, no – if the message to the IRR auto-processor contains the correct cleartext password or is appropriately signed with PGP, it doesn’t matter where it comes from.
The 2012 mention is slightly inaccurate. LEVEL3 has been discouraging use of MAIL-FROM in their IRR guide since at least 2012 – the oldest I could find. The only other IRR operator on which I could find a date of MAIL-FROM removal is RADb which was May of 2015.
PGPKEY is not particularly difficult – you email a key-cert object to the IRR, and once it’s accepted, you add auth: PGPKEY foo to the maintainer object. Then, once you’re sure PGPKEY auth works, you update the maintainer object to remove auth: CRYPT-PW.
So the solution for an insecure email MAIL-FROM address is to send your key-cert to the IRR by email?
if you’re the contact for an IRR and you don’t have SPF and DKIM configured that’s your funeral
“Seventy percent of attacks use letter-off misspelled domains.” -some blog
Can you eliminate “duplicate” domain PEBCAK error with SPF DKIM? (?)
Depends on the quality of your PEBCAK?
“The East India Company was no apparition though; it was the template for many subsequent corporations […] Liberals betray themselves […] the moment they turn a blind eye to this kind of hyper-concentrated power. […] This is why trading in apples does not come even close to trading in shares. Large quantities may produce, at worse, lots of bad cider, but large amounts of money invested in liquid shares can release demonic forces that no market or state can control.”
― Yanis Varoufakis, Another Now: Dispatches from an Alternative Present
Built of cobbled hubris and copied code we sail a vast uncharted ocean of extremely hungry sharks,
no “plan” at any point but sink or swim, and no rudder. Greater speed the only virtue and destination.
You can imagine how it goes, when it does. The looks on people’s faces in the inconvenient reality.
Wherever this course leads, faster. More coal to the boilers now. Release date approaches.
The problem with RPKI as a security mechanism is that they handed the problem of securing BGP over to a bunch of X.509 fanboys, and so the result was something that looked suspiciously like X.509…. no, I stand corrected, it _is_ X.509. It’s an attempt to use an X.509 certificate in a way that was never designed for, combined with the mass of other problems that make X.509 such a joy to use.
“The Internet is Held Together With Spit & Baling Wire”-
Holy Heckfire, y’dont say??
Man, wait until The Suits hear about this and Tooth Decay…all He**’s going to break loose!
++++++++++
Yep, any system that rides on this much legacy clutter if about as stable as a paper boat in the North Atlantic.
—————-
Seriously, I’m thankful for your work, Brian, on this T-Day weekend! You go, man!
And for many years I thought it was held together with bubble gum and silly putty. Thanks for the update.
Good old, Lumen! They always do the right thing after trying every other possibility path.
I think we need to remember that the “internet” was not really designed, it pretty much evolved over time. And a lot of that early “time” was when it was only for educational, research or military traffic and we didn’t think much about spam, randsomware and cyber attacks. Once it became commercialized the speed at which it expanded is staggering, and when something grows at that speed some things can get overlooked. Unfortunately strong security was one of them.
Anyway, we are where we are. Deciding what to do is the question. And herding cats comes to mind.
I wouldn’t call the process “evolved,” as that implies “better able to survive.” I would simply say the changes in Internet management over the years have been and are being made almost randomly and without any real central authority.
…evolved is exactly the right term – not all evolved traits survive, although it’s a common mistake to think that “survival of the fittest” means that all evolved traits are “good” for survival, in fact they are not and many “evolved” traits die away…
…as to “management” it works exactly as envisioned by Cerf, et al, through the RFC process and you should understand that before making incorrect comments…
Well a wise-man or collection of individuals offered a solution i believe some 20 years ago called ipv6… wonder when It’ll be fully implemented. I believe if there’s a global collective agreement to adopt ipv6 as the standard it’ll circumvent a lot of today’s issues ..
That same committee of wiseacres, in their infinite wisdom decided to intentionally break backward compatibility with IPv4 for no real good reason except for hubris, thus ensuring that government and business never saw a compelling reason to adopt it and kludged “solution” after “solution” onto IPv4 because it was easier to do and cost less than the supposedly technically “superior” IPv6.
Academics can’t get it through their neckbeards that average users and businesses don’t care about technical superiority or elegance in solutions. They want something that works and is mostly compatible with legacy systems. I’m quite sure IPv6 will be adopted, possibly around 2100 or so.
distributed uncompartmentalized internet as we know it will exist in 2100? 2032 even?
IPv4 vs IPv6 is irrelevant in this context. IRR records exist for IPv4 & IPv6 and could have been forged for either.
IPSEC is a fundamental part of IPv6. So with IPv6-only delivered email it would be possible to extract certificate metadata of the the sending system and associate this metadata with the received email for whitelisting purposes. 70,000 possible end-points with 2 year certificates translates to a whitelist that requires less than 100 changes a day. This is not more effort than a large IRR could keep up with. The reality is that the number of orgs using IRRs is orders of magnitude less than 70,000 and we are looking at small number of daily whitelist changes per IRR. Spoofing email headers is much easier than circumventing a whitelist of connections secured by IPSEC.
A few points from someone who monitors a small number of name & number resources for supply chain security.
Forging an IRR record does not in and of itself redirect traffic. Loosely speaking, any BGP speaker can announce itself as the preferred provider to a number resource of interest [1]. IRR’s host the data that BGP recipients use to create filters to control receipt of BGP announcements. The ability to forge IRR records influences the acceptance of fraudulent BGP announcements. A considerable amount of work has been done to improve routing hygiene and security. Interested parties should review Mutually Agreed Norms for Routing Security (MANRS) & the references therein, FCC, IETF, NIST, NANOG meeting archives, ARIN, & RIPE resources. See NTT’s Job Snijders’ experiences deploying RPKI in their global network [2].
I think Lumen should have restricted “Mail From” IRR updates to the set of providers currently using them and collaboratively determined a drop dead date for the technology, instead of implementing a breaking change. Yes, those resource owners are using weak processes & should modernize. I don’t think the appropriate step is less number resource security in the meantime.
[1] Historic incidents:
https://bgpmon.net/popular-destinations-rerouted-to-russia/
https://arstechnica.com/uncategorized/2008/02/insecure-routing-redirects-youtube-to-pakistan/
http://bgpstream.com/
[2] https://www.nanog.org/news-stories/nanog-tv/nanog-79-webcast/lessons-learned-ntts-rpki-deployment/
A few points from someone who monitors a small number of name & number resources for supply chain security.
Forging an IRR record does not in and of itself redirect traffic. Loosely speaking, any BGP speaker can announce itself as the preferred provider to a number resource of interest [1]. IRR’s host the data that BGP recipients use to create filters to control the receipt of BGP announcements. The ability to forge IRR records influences the acceptance of fraudulent BGP announcements. A considerable amount of work has been done to improve routing hygiene and security. Interested parties should review Mutually Agreed Norms for Routing Security (MANRS) & the references therein, FCC, IETF, NIST, NANOG meeting archives, ARIN, & RIPE resources. See NTT’s Job Snijders’ experiences deploying RPKI in their global network [2].
I think Lumen should have restricted “Mail From” IRR updates to the set of providers currently using them and collaboratively determined a drop dead date for the technology, instead of implementing a breaking change. Yes, those resource owners are using weak processes & should modernize. I don’t think the appropriate step is less number resource security in the meantime.
[1] Historic incidents:
https://bgpmon.net/popular-destinations-rerouted-to-russia/
https://arstechnica.com/uncategorized/2008/02/insecure-routing-redirects-youtube-to-pakistan/
http://bgpstream.com/
[2] https://www.nanog.org/news-stories/nanog-tv/nanog-79-webcast/lessons-learned-ntts-rpki-deployment/
Facebook went offline?! Dammit!
Wonder if a government maintained read-only database was kept as a backup and published as an ICE database that would always be out of date but would get 75% of the backbone back up and running if things went to crap?
We have to move away from the use of unencrypted emails , they have been a problem for years now.
Al Gore here to describe the inter-webs, since i created it!
it was created for coms during disaster recover if a nuke war ever happened..
it was not designed for commerce or cat videos.
Hold my beers al snore said the University of Caucasians Lost among Asians.
Ucla created the web and it is held together by greed, fear, narcissism, and lots and lots of porn.
when and if ww3 occurs,
the redundancy design will be put to the test and instead of nocs and it departments frantically trying to rebuild the web,
it will be only authoritarian govs backed by greedy mega corps armed with black ai
installing, maintaining, and controlling it.
many thanks to Brian and the other trained hands who have offered cogent details, and who have obviously been around the track a few times. I can only hope in blind faith that somehow all these systems which have evolved or developed over time, and out of a hit or miss collage of concepts, will get gradually refined. Human history being what it is, the prognosis, according to many of the commenters, is not so rosy. As a lay person, but a student of history, humans have demonstrated we are not the sharpest tool in the shed, and humility is not our strong suit either. The 7 deadly sins seem to be alive and well, so I do not have any ready answers.
Somewhat off-point, the fact that radio bands and frequencies are allocated somewhat arbitrarily depending on which gov is doing it and which companies are buying up the leases of the bandwidth tells you something about how humans work. There is only one electro-magnetic-spectrum, but it would take real world-wide cooperation to realign all these systems to conform to one standard. Sounds like an age old problem, but being an ignorant optimist, I can only hope and pray, a critical mass of high-minded professionals will iron it all out, eventually. Thanks again to all of you professionals who know what you are doing and know what should be done. In my darker moments, the way some companies behave reminds me of the old ad for Arpegge perfume, “promise her anything, but give her Arpegge”, a cynical statement when looked at in hindsight. thank you, Brian.
One would think that this is exactly where block chain or similar technology would be useful. If there is an immutable chain of transactions, you couldn’t have the Facebook fiasco. The problem however, IMHO, is that the system is a state machine and there are now states that weren’t visualized in it’s inception that now are causing problems. Simple, secure, cheap. Pick two.
This makes me think of another rabbit trail of how BGP is relevant to the Great Firewall, and geopolitical strategic use of the internet…
Real question here – Do providers actually scrub BGP announced prefixes against the IRR? And furthermore, do they then implement restrictions (think route-maps) to actually restrict what prefixes they will accept from your AS ?
What is described in this article is a much-needed improvement to maintaining a sort of ‘chain of custody’ of who rightfully SHOULD be announcing a prefix; but there seems little work done to prevent the actual announcement of that prefix from anyone else.
If I were to start announcing 8.8.8.8 /32 with a fake AS-path on my peering with Lumen, would that prefix actually get dropped, or would I get a flood of DNS traffic (till my router caught fire, that is)? I know there would be repercussions, but what if I was a bad actor at my company, or what if my company’s been hacked… What if I announced an e-commerce’s AS? Or a bank, for that matter?
All the more reason to pay attention to the rules about what you will accept via BGP, and keep them as tight as possible…
> Do providers actually scrub BGP announced prefixes against the IRR? And furthermore, do they then implement restrictions (think route-maps) to actually restrict what prefixes they will accept from your AS ?
Yes.
> If I were to start announcing 8.8.8.8 /32 with a fake AS-path on my peering with Lumen, would that prefix actually get dropped, or would I get a flood of DNS traffic
Yes, it would get dropped by BGP filtering.
Unless….you added a route: object of 8.8.8.8/32 to IRR with origin: youras — which you could do. Then once filters rebuild from IRR, you would actually be permitted to advertise that /32 prefix to the Lumen router to which you directly peer, and BGP filtering would not drop that announcement, because it’s on your list. IRR is not a perfect system.
However, Lumen drops RPKI invalid routes – see: https://blog.lumen.com/lumen-enhances-routing-security-with-resource-public-key-infrastructure-rpki/ – and since 8.8.8.8/32 is part of actually 8.8.8.0/24, and the /24 has a valid and signed ROA indicating its authentic origin is AS15169 — and therefore, even though you slipped it past the BGP filter goalie, it will still get dropped because the ROA isn’t valid for the announcement from an AS other than AS15169.
AK
Nice choice of an example 🙂 : Short answer … yes it is possible. In 2014 someone fat-fingered a route of 8.8.8.0/24 and not only disrupted Google search, but DoS’ed the internet of the entire country:
https://arstechnica.com/information-technology/2014/03/google-dns-briefly-hijacked-to-venezuela/
Unless I’ve missed something in reference to the title verbatim, I believe you’ve mixed your metaphors.
Made of Spit and Tissue Paper – In the context of cheap motel room walls.
Held together with Duct Tape and Bailing Wire — In the context of kluging a fix to keep something together.
Kudos for writing about the important topic and giving the article an amazing title. Routing security is indeed a foundation of Internet security that should be widely considered by network operators around the world, but the interconnected nature of the Internet means it is a collective action problem.
The Mutually Agreed Norms for Routing Security (MANRS) is a community-driven initiative, supported by the Internet Society, that provides a set of best practices based on existing norms for network operators to improve the security of the global Internet routing system. About 700 network operators around the world take part in the fast-growing initiative today. One of the requirements of MANRS is to facilitate routing information via IRR or RPKI.
If people knew that even some high-end telecom equipment are held together by a bunch of perl scripts supporting their day-to-day communications.. they would be amazed..
Thaaaaanksss big to Brian! Because security is hard, and users are lazy, and so making systems which are secure even for ordinary users takes way too much time and effort, so too many companies just hack together something slapdash and hope nothing goes terribly wrong.Thanks for the update.
IRRs could improve security by limiting email automation to just IPv6 received emails and whitelisting based on IPSEC certificate metadata. They each deal with orders of magnitude less than 70,000 other organizations and that number grows slowly, so there is not a scaling problem in this aspect of the internet. One or a few employees would be needed to keep up with the daily whitelist changes caused largely by certificates expiring which could also be partly automated between larger organizations. This would provide one more layer of security that is relatively easily enabled via IPv6.
Continuing to operate desite risks… what a concept. Any chance that some of these folks could migrate into public health?