On Monday, a former Amazon employee was arrested and charged with stealing more than 100 million consumer applications for credit from Capital One. Since then, many have speculated the breach was perhaps the result of a previously unknown “zero-day” flaw, or an “insider” attack in which the accused took advantage of access surreptitiously obtained from her former employer. But new information indicates the methods she deployed have been well understood for years.
What follows is based on interviews with almost a dozen security experts, including one who is privy to details about the ongoing breach investigation. Because this incident deals with somewhat jargon-laced and esoteric concepts, much of what is described below has been dramatically simplified. Anyone seeking a more technical explanation of the basic concepts referenced here should explore some of the many links included in this story.
According to a source with direct knowledge of the breach investigation, the problem stemmed in part from a misconfigured open-source Web Application Firewall (WAF) that Capital One was using as part of its operations hosted in the cloud with Amazon Web Services (AWS).
Known as “ModSecurity,” this WAF is deployed along with the open-source Apache Web server to provide protections against several classes of vulnerabilities that attackers most commonly use to compromise the security of Web-based applications.
The misconfiguration of the WAF allowed the intruder to trick the firewall into relaying requests to a key back-end resource on the AWS platform. This resource, known as the “metadata” service, is responsible for handing out temporary information to a cloud server, including current credentials sent from a security service to access any resource in the cloud to which that server has access.
In AWS, exactly what those credentials can be used for hinges on the permissions assigned to the resource that is requesting them. In Capital One’s case, the misconfigured WAF for whatever reason was assigned too many permissions, i.e. it was allowed to list all of the files in any buckets of data, and to read the contents of each of those files.
The type of vulnerability exploited by the intruder in the Capital One hack is a well-known method called a “Server Side Request Forgery” (SSRF) attack, in which a server (in this case, CapOne’s WAF) can be tricked into running commands that it should never have been permitted to run, including those that allow it to talk to the metadata service.
Evan Johnson, manager of the product security team at Cloudflare, recently penned an easily digestible column on the Capital One hack and the challenges of detecting and blocking SSRF attacks targeting cloud services. Johnson said it’s worth noting that SSRF attacks are not among the dozen or so attack methods for which detection rules are shipped by default in the WAF exploited as part of the Capital One intrusion.
“SSRF has become the most serious vulnerability facing organizations that use public clouds,” Johnson wrote. “The impact of SSRF is being worsened by the offering of public clouds, and the major players like AWS are not doing anything to fix it. The problem is common and well-known, but hard to prevent and does not have any mitigations built into the AWS platform.”
Johnson said AWS could address this shortcoming by including extra identifying information in any request sent to the metadata service, as Google has already done with its cloud hosting platform. He also acknowledged that doing so could break a lot of backwards compatibility within AWS.
“There’s a lot of specialized knowledge that comes with operating a service within AWS, and to someone without specialized knowledge of AWS, [SSRF attacks are] not something that would show up on any critical configuration guide,” Johnson said in an interview with KrebsOnSecurity.
“You have to learn how EC2 works, understand Amazon’s Identity and Access Management (IAM) system, and how to authenticate with other AWS services,” he continued. “A lot of people using AWS will interface with dozens of AWS services and write software that orchestrates and automates new services, but in the end people really lean into AWS a ton, and with that comes a lot of specialized knowledge that is hard to learn and hard to get right.”
In a statement provided to KrebsOnSecurity, Amazon said it is inaccurate to argue that the Capital One breach was caused by AWS IAM, the instance metadata service, or the AWS WAF in any way.
“The intrusion was caused by a misconfiguration of a web application firewall and not the underlying infrastructure or the location of the infrastructure,” the statement reads. “AWS is constantly delivering services and functionality to anticipate new threats at scale, offering more security capabilities and layers than customers can find anywhere else including within their own datacenters, and when broadly used, properly configured and monitored, offer unmatched security—and the track record for customers over 13+ years in securely using AWS provides unambiguous proof that these layers work.”
Amazon pointed to several (mostly a la carte) services it offers AWS customers to help mitigate many of the threats that were key factors in this breach, including:
–Access Advisor, which helps identify and scope down AWS roles that may have more permissions than they need;
–GuardDuty, designed to raise alarms when someone is scanning for potentially vulnerable systems or moving unusually large amounts of data to or from unexpected places;
–The AWS WAF, which Amazon says can detect common exploitation techniques, including SSRF attacks;
–Amazon Macie, designed to automatically discover, classify and protect sensitive data stored in AWS.
William Bengston, formerly a senior security engineer at Netflix, wrote a series of blog posts last year on how Netflix built its own systems for detecting and preventing credential compromises in AWS. Interestingly, Bengston was hired roughly two months ago to be director of cloud security for Capital One. My guess is Capital One now wishes they had somehow managed to lure him away sooner.
Rich Mogull is founder and chief technology officer with DisruptOPS, a firm that helps companies secure their cloud infrastructure. Mogull said one major challenge for companies moving their operations from sprawling, expensive physical data centers to the cloud is that very often the employees responsible for handling that transition are application and software developers who may not be as steeped as they should in security.
“There is a basic skills and knowledge gap that everyone in the industry is fighting to deal with right now,” Mogull said. “For these big companies making that move, they have to learn all this new stuff while maintaining their old stuff. I can get you more secure in the cloud more easily than on-premise at a physical data center, but there’s going to be a transition period as you’re acquiring that new knowledge.”
Image: Capital One
Since news of the Capital One breach broke on Monday, KrebsOnSecurity has received numerous emails and phone calls from security executives who are desperate for more information about how they can avoid falling prey to the missteps that led to this colossal breach (indeed, those requests were part of the impetus behind this story).
Some of those people included executives at big competing banks that haven’t yet taken the plunge into the cloud quite as deeply as Capital One has. But it’s probably not much of a stretch to say they’re all lining up in front of the diving board.
It’s been interesting to watch over the past couple of years how various cloud providers have responded to major outages on their platforms — very often soon after publishing detailed post-mortems on the underlying causes of the outage and what they are doing to prevent such occurrences in the future. In the same vein, it would be wonderful if this kind of public accounting extended to other big companies in the wake of a massive breach.
I’m not holding out much hope that we will get such detail officially from Capital One, which declined to comment on the record and referred me to their statement on the breach and to the Justice Department’s complaint against the hacker. That’s probably to be expected, seeing as the company is already facing a class action lawsuit over the breach and is likely to be targeted by more lawsuits going forward.
But as long as the public and private response to data breaches remains orchestrated primarily by attorneys (which is certainly the case now at most major corporations), everyone else will continue to lack the benefit of being able to learn from and avoid those same mistakes.
How is it that the intruder was even able to access the misconfigured Web Application Firewall in the first place? Isn’t some sort of authentication necessary to do this? If not, or if the method of authentication is weak (using a password such as “password123”), wouldn’t some form of strong authentication for accessing the WAF have prevented the breach, despite the other misconfigurations? Or did the misconfiguration of the WAF itself disable any authentication necessary to access the WAF?
Most likely, the WAF was intentionally exposed to the Internet and was expected to transparently pass requests to servers running on EC2 instances inside the AWS account VPC.
This article implies that the intruder was able to cause the WAF to trigger AWS commands to enumerate and dump the contents of the S3 buckets. This would imply that the WAF was itself running on an EC2 with an overly-permissive IAM role.
Another possibility is that the intruder used an additional hop and caused the WAF to pass HTTP requests to an EC2 server inside the VPC, and that server executed the S3 commands. A misconfiguration on the WAF could allow requests to be passed to a server that wasn’t intended to be exposed, and thus have a permissive role.
Actually, on reading related articles in more depth, the consensus appears to be that the WAF was intentionally set up to “redirect” incoming requests (really, more like a reverse proxy). Due to the misconfiguration, the intruder was able to send a request to the local metadata service, which returned the temporary IAM credentials the WAF was using. Then, the intruder used those creds to make API calls to the S3 service and dump all the data.
This would imply two bad configurations: Whatever was on the WAF that allowed the redirection to the metadata service, and the overly-permissive role assigned to the EC2 instance and allowed all the S3 access.
This is a common, easy, well-understood attack. The configuration errors are very easy mistakes to make.
I think it’s more so that they sent a request through the WAF that was a SSRF attack that called the metadata service on the EC2s running the WAF application to retrieve credentials. These credentials are for the role the EC2 is running under and had too permissive policies. Those credentials allowed the hacker to list buckets contents and download data from the attackers machines not from machines in Capital Ones VPC.
Yes, it looks that way. Sadly, your reply pointing that out appeared before my reply pointing that out.
Looking at the earlier article, it looks like she typically also used creds to set up EC2 instances and set up other services to exfiltrate. Just pulling straight off the S3 buckets would have worked for Capital One, though.
Regardless of HOW this happened, wasn’t the personal data that was stolen encrypted? Or was it poorly encrypted and able to be brute-forced?
Leaving it unencrypted would be a huge security violation. My guess is that it was encrypted on S3 but the role on the EC2 had KMS decrypt permissions attached.
Almost certainly. The data was clearly not programmatically encrypted by the application. Given the SSRF risks and just “whoopsie” permissions issues with configuring S3 properly, and issues dealing with IAM roles – as mentioned by Krebs and the linked article from Evan, that stuff is complex and confusing, sensitive data should never be stored in S3 unencrypted. Figure out a way to programmatically encrypt it and protect the hell out of those keys. It just was not hard enough for this hacker to get to this data.
In public articles, Capital One’s CIO stated quite clearly, that Capital One created a product called “Cloud Custodian” that would “automatically encrypt all Capital One data” going into AWS. (not some..ALL).
Is there any evidence the data is not still encrypted by CapOne? There is reporting that data was exfiltrated. Nothing says it was in a format actually useful to the hacker.
should has a deny block in S3 bucket policy, just use the IAM and encrypts are not enough. Remember your data is on public-clouds.
I agree, simple S3 policies are not enough, more restrictions to make access more specific can help. Even if the instance credentials are compromised, e.g. the S3 vpc-endpoint whitelisting may deny outside access.
Te basic 4 control family veracity were not there. If any of the 4 mentioned control families were weak, the threat is inevitable to become reality.
Always try to have Control veracity at highest when it comes to:
1, AAA (authentication, authorization, access)
2, Channel Security or crypto in transit
3, Content Security or crypto at rest (hoping it is separated from 2)
4, Audit (lastly this is where we all miss the signs of security “tickles” that we get before we fall over).
It is easy to blame the vendors or admins or…
The truth of the matter is that we are all vulnerable to low resources, authority, and budget to have the controls above maintained and operated. Far too many times when we are successful the first question from MGT is what do those guys do behind closed doors.
Something I never see addressed. The executives of all these companies that have had break-ins, was any of THEIR personal information included? Do they keep their info sand boxed away from the customer databases? If they can keep their own information safe then why can they not keep ALL information safe?
In the deposition, it mentioned that someone saw the Capital One files on Github and informed Capital One at which point they began their investigation. It also states that she mentioned encrypting the information prior to putting it on Github. So, this also raises the question, how secure is encrypted information on Github?
GitHub isn’t designed for security per se.
It’s designed for distributed coordination of development on source code. Generally speaking, it’s a way to share code and make changes to code.
If you choose to encrypt your content before submitting it to GitHub, then it’s up to you wrt how you choose to encrypt it.
Connections to GitHub themselves are transport encrypted w/ TLS/SSH, but that’s mostly just to prevent MitM attacks. Generally speaking anyone who is authorized can read the content of a given repository, and I think most repositories are public.
Isn’t anyone concerned that by notifying customers through various means if they were involved will only create a much larger phishing campaign with hackers moving forward? Why is Capital One not establishing a search index similar to what Equifax did?
Equifax’s response to their data breach was more horrible than the breach itself. It isn’t and shouldn’t be up to the victim to find out whether they are affected or not. The responsibility of notifying the victim was Equifax’s. By setting up a website for users to input 6 of the 9 digits in their social security number is just asking for someone to setup a rogue website to collect social security numbers.
most of capitalone should have migrated away from mod_security around 2 years ago
2 years ago capitalone switched their WAFs away from mod_security