On Monday, a former Amazon employee was arrested and charged with stealing more than 100 million consumer applications for credit from Capital One. Since then, many have speculated the breach was perhaps the result of a previously unknown “zero-day” flaw, or an “insider” attack in which the accused took advantage of access surreptitiously obtained from her former employer. But new information indicates the methods she deployed have been well understood for years.
What follows is based on interviews with almost a dozen security experts, including one who is privy to details about the ongoing breach investigation. Because this incident deals with somewhat jargon-laced and esoteric concepts, much of what is described below has been dramatically simplified. Anyone seeking a more technical explanation of the basic concepts referenced here should explore some of the many links included in this story.
According to a source with direct knowledge of the breach investigation, the problem stemmed in part from a misconfigured open-source Web Application Firewall (WAF) that Capital One was using as part of its operations hosted in the cloud with Amazon Web Services (AWS).
Known as “ModSecurity,” this WAF is deployed along with the open-source Apache Web server to provide protections against several classes of vulnerabilities that attackers most commonly use to compromise the security of Web-based applications.
The misconfiguration of the WAF allowed the intruder to trick the firewall into relaying requests to a key back-end resource on the AWS platform. This resource, known as the “metadata” service, is responsible for handing out temporary information to a cloud server, including current credentials sent from a security service to access any resource in the cloud to which that server has access.
In AWS, exactly what those credentials can be used for hinges on the permissions assigned to the resource that is requesting them. In Capital One’s case, the misconfigured WAF for whatever reason was assigned too many permissions, i.e. it was allowed to list all of the files in any buckets of data, and to read the contents of each of those files.
The type of vulnerability exploited by the intruder in the Capital One hack is a well-known method called a “Server Side Request Forgery” (SSRF) attack, in which a server (in this case, CapOne’s WAF) can be tricked into running commands that it should never have been permitted to run, including those that allow it to talk to the metadata service.
Evan Johnson, manager of the product security team at Cloudflare, recently penned an easily digestible column on the Capital One hack and the challenges of detecting and blocking SSRF attacks targeting cloud services. Johnson said it’s worth noting that SSRF attacks are not among the dozen or so attack methods for which detection rules are shipped by default in the WAF exploited as part of the Capital One intrusion.
“SSRF has become the most serious vulnerability facing organizations that use public clouds,” Johnson wrote. “The impact of SSRF is being worsened by the offering of public clouds, and the major players like AWS are not doing anything to fix it. The problem is common and well-known, but hard to prevent and does not have any mitigations built into the AWS platform.”
Johnson said AWS could address this shortcoming by including extra identifying information in any request sent to the metadata service, as Google has already done with its cloud hosting platform. He also acknowledged that doing so could break a lot of backwards compatibility within AWS.
“There’s a lot of specialized knowledge that comes with operating a service within AWS, and to someone without specialized knowledge of AWS, [SSRF attacks are] not something that would show up on any critical configuration guide,” Johnson said in an interview with KrebsOnSecurity.
“You have to learn how EC2 works, understand Amazon’s Identity and Access Management (IAM) system, and how to authenticate with other AWS services,” he continued. “A lot of people using AWS will interface with dozens of AWS services and write software that orchestrates and automates new services, but in the end people really lean into AWS a ton, and with that comes a lot of specialized knowledge that is hard to learn and hard to get right.”
In a statement provided to KrebsOnSecurity, Amazon said it is inaccurate to argue that the Capital One breach was caused by AWS IAM, the instance metadata service, or the AWS WAF in any way.
“The intrusion was caused by a misconfiguration of a web application firewall and not the underlying infrastructure or the location of the infrastructure,” the statement reads. “AWS is constantly delivering services and functionality to anticipate new threats at scale, offering more security capabilities and layers than customers can find anywhere else including within their own datacenters, and when broadly used, properly configured and monitored, offer unmatched security—and the track record for customers over 13+ years in securely using AWS provides unambiguous proof that these layers work.”
Amazon pointed to several (mostly a la carte) services it offers AWS customers to help mitigate many of the threats that were key factors in this breach, including:
–Access Advisor, which helps identify and scope down AWS roles that may have more permissions than they need;
–GuardDuty, designed to raise alarms when someone is scanning for potentially vulnerable systems or moving unusually large amounts of data to or from unexpected places;
–The AWS WAF, which Amazon says can detect common exploitation techniques, including SSRF attacks;
–Amazon Macie, designed to automatically discover, classify and protect sensitive data stored in AWS.
William Bengston, formerly a senior security engineer at Netflix, wrote a series of blog posts last year on how Netflix built its own systems for detecting and preventing credential compromises in AWS. Interestingly, Bengston was hired roughly two months ago to be director of cloud security for Capital One. My guess is Capital One now wishes they had somehow managed to lure him away sooner.
Rich Mogull is founder and chief technology officer with DisruptOPS, a firm that helps companies secure their cloud infrastructure. Mogull said one major challenge for companies moving their operations from sprawling, expensive physical data centers to the cloud is that very often the employees responsible for handling that transition are application and software developers who may not be as steeped as they should in security.
“There is a basic skills and knowledge gap that everyone in the industry is fighting to deal with right now,” Mogull said. “For these big companies making that move, they have to learn all this new stuff while maintaining their old stuff. I can get you more secure in the cloud more easily than on-premise at a physical data center, but there’s going to be a transition period as you’re acquiring that new knowledge.”
Image: Capital One
Since news of the Capital One breach broke on Monday, KrebsOnSecurity has received numerous emails and phone calls from security executives who are desperate for more information about how they can avoid falling prey to the missteps that led to this colossal breach (indeed, those requests were part of the impetus behind this story).
Some of those people included executives at big competing banks that haven’t yet taken the plunge into the cloud quite as deeply as Capital One has. But it’s probably not much of a stretch to say they’re all lining up in front of the diving board.
It’s been interesting to watch over the past couple of years how various cloud providers have responded to major outages on their platforms — very often soon after publishing detailed post-mortems on the underlying causes of the outage and what they are doing to prevent such occurrences in the future. In the same vein, it would be wonderful if this kind of public accounting extended to other big companies in the wake of a massive breach.
I’m not holding out much hope that we will get such detail officially from Capital One, which declined to comment on the record and referred me to their statement on the breach and to the Justice Department’s complaint against the hacker. That’s probably to be expected, seeing as the company is already facing a class action lawsuit over the breach and is likely to be targeted by more lawsuits going forward.
But as long as the public and private response to data breaches remains orchestrated primarily by attorneys (which is certainly the case now at most major corporations), everyone else will continue to lack the benefit of being able to learn from and avoid those same mistakes.
What we’ve learned is that the amount they have to pay if they lose should be based on a set fee for the attorneys and a per verified claimant dollar ammount. As it stands today, no one is going to get $125, but will probably end up closer to $0.01. What I propose is that they pay a base cost and then set amout to every verified claimant. In the Eperian case, $125 is ridiculously low and should be $10000. Multiply this times the number of members in the class, and you get over $2.6T. I think that would be fair fine for Equifax. Also, as a penalty, any information posessed by all other credit bureaus that originated at Experian must be deleted. I think that’s fair, too.
Amen, brother.
Agreed! Although I don’t think either of those companies should be fined into bankruptcy (which they could evade damages by filing Chapter 11), sanctions should have a punitive, deterrent impact. More importantly, the individuals behind the egregious behavior should have more personal liability (equifax execs for their response and insider trading, and the arrested suspect behind the Cap 1 hack). Not seeing the motivation for anything to change…yet again.
Let’s not take this to such an extreme! Fining the corporation of Equivax $2.6T would be great but not achievable.
Fining each of the senior management some large proportion of $1T based on their positions during this security lapse/heist would be very satisfactory.
Who is the goofball Govt attorney who let Equifax only allow 31 million for refunds against the 125 fee? That person should be fired immediately.
The amt should have been north of 200 million.
I get your idea but it’s pretty ridiculous. We live in an age where breaches such as this is common. Are you saying that every company that gets breached needs to be bankrupted? The amount they gave out was definitely low but there isn’t a need to go overboard.
Heck, if the amount is high, I could see people repeatedly targeting specific companies to bankrupt them…
I am still going to submit… and hope. I froze my credit, spending $20, $10 each at Esperian and Transunion. I did this right after the Equifax breach before the law passed to make freezes free.
I doubt I’ll even cover that amount. Life Lock and other monitoring isn’t worth anything… “hey, someone opened a credit card in your name, you should do something about that!”
Where’s GDPR for US? Why are we trailing so far behind Europe? For as long as these giant corporations can get away with all these breaches, and can elect (buy) their politicians, nothing will change!
Also can’t agree more with this statement. We need to round up all the lawyers and … ‘em.
“But as long as the public and private response to data breaches remains orchestrated primarily by attorneys (which is certainly the case now at most major corporations), everyone else will continue to lack the benefit of being able to learn from and avoid those same mistakes.”
In the USA, digital privacy is being handled at the state level, not federal. For example, California passed the California Consumer Privacy Act (CCPA). It isn’t being enforced yet, and there are still some poorly defined parts of it. I think other states may be working on similar laws. The problem is that when you are an national or international company, and there are multiple compliance laws that may contradict each other, which ones do you follow? And, what is going to keep GDPR & CCPA from being used by shyster lawyers to harass and extort companies?
Look on the bright side. Most of us will be getting free (worthless) credit reports for the rest of our creditable lives. Given how our accounts are plundered by the banks/financials, we won’t have to worry about having any credit in 2-3 years.
“Look on the bright side. Most of us will be getting free (worthless) credit reports for the rest of our creditable lives.”
Maybe and maybe not. They are worthless if hackers can still find someway to setup accounts in our name even if we enable freezes with every CRA. I’d get a report showing that someone still created an account in my name, that’s wonderful, now I’d have to fight that. But the catch is – enabling such. I did freeze my Equifax account, among others, last year. Now when I try to setup the freeze via my.equifax.com, the endless loop begins. They let me setup an account but won’t let me access it, wrong emai or password is the notice every time I try to log in.
I change the password by coping the info from a text file and paste it into the box for such, that it accepts. Try to log in with the same password, no go. Call Equifax (hour plus wait) they reset the password, I create a new one, once again it says my password is wrong. Gone on all week, many calls, no joy, they blame it on all the Capital One customers contacting them. And, I’m a Cap 1 customer so now what do I do? Equifax won’t allow me to secure my account, And Capital One lets everyone into my wallet, bank account and who knows what other info has been leaked.
The only bright side is that so far…no one has used any of my personal info illegally. But with all the hacks over the years, how much longer before a Nation State simply releases all the hacked data just to disrupt the US economy? Getting a free credit report that shows you’ve opened up 20 accounts, all now delinquent, in the past year will not help you.
Bright side, maybe Capital One will finally scrap their what’s in your wallet ad.
I haven’t seen ANY reporting that Capital One ACCOUNTS were breached. The reports have stated information submitted for new accounts is what was obtained. Yes, this information is personal and should have been secured and could lead to nefarious uses but your existing accounts were not breached.
Harp on Capital One for having the breach but lets keep to the known facts.
Do we know what role Purple Rain plays into this? Back in 2017 Capital One was in the news for beefing up their Cyber Security via ‘Purple Rain’, but I see no mention of that in this article anywhere.
Oh my
https://www.datanami.com/2017/04/21/purple-rain-bolsters-security-intelligence-capital-one/
Yeah, but I am so skeptical about AI. Sorry but today AI without a human is pretty dumb.
You teach Elastic to find one or two requests and you don’t know what to look for. Uncharacteristic? Today we know that metadata request through public interface is a bad thing. But we didn’t know it yesterday. And when we know we can fix WAF.
What I an saying that AI comes with human experience. You cannot teach AI of something that you did not experience yourself.
And if you receive this “uncharacteristic” packet. What do you do? Block?
What if it is legit and will break legit functionality.
All those are very complex things.
“What I an saying that AI comes with human experience. You cannot teach AI of something that you did not experience yourself.”
No… you are not understanding what AI is. Perhaps you are thinking of the bastardized buzzword of AI that many companies sell in marketing pitches.
AI has machine learning. Read up… it can and does learn outside of human experience.
As I usually like to do whenever I see news of a security breach at a company, I did a little looking into Capital One and how much they care about security.
First I checked if they run a bug bounty program, rewarding those who find and alert them of security issues with their systems. They do not.
Next I looked into their job openings. They have an opening for Director of Cybersecurity Operations, which at least shows they have people in their organization whose job it is to secure and monitor their systems. Whether this is the laughably minimal ‘patch patrol’ most companies think is adequate is unclear. They also have openings for software engineers. This is where the rubber hits the road from my point of view. Any company that cares about security will prioritize hiring developers who know how to build secure systems and follow secure coding practices. The word security does not even appear in the job listings for software engineers.
Any claim Capital One makes to care about information security should be viewed with extreme suspicion. As should their systems themselves.
I guess you didn’t read the complaint. They had a responsible disclosure program. That is a large part of why it was discovered.
It’s laughable that one thinks a large banks’s sole monitoring of cloud storage of customer data should bevleft to a big bounty program. First, this wasn’t due to a bug. It was due to negligence in performing the most basic of security configurations. Second, for many months their ineffective Cyber Security Threat Intel teams apparently failed to even be aware of public forum postings made by the hacker who apparently was literally begging to be caught. And lastly, their high tech security operations center failed to detect the breach, the persistent 4 month presence of the hacker, and the unauthorized transfer of gigs upon gigs of data.
But yet the CEO and CIO will surely still receive the usual huge bonuses for 2019, I’m sure.
Interesting stand. We have been looking into the legal side of the matter to start with. How liable is Capital One in terms of the security they have hired Amazon Web Services to take care of their digitl automated infrastructure.
What will be the legal position of the inflicted Capatal One customer knowing that all mentioned data have been exposed on the internet over six weeke before Capatial One has been tipped of(!?!) and notified the FBI.
Since digital automation is 100% Predictable, as matter, and over 85% of professionals in digital automation seems to have no idea of that, this wil become a serious legal issue in terms of individual liability and accountability.
As a think tank we are monitoring this case as well as the Hertz – Accenture case closely. Herts is suing Accenture for misconduct, as we understand of the Hertz lawyer and lawsuit, they are aiming for a jury trial.
In case of a jury trial the contract isn’t that much leading but the fact how a supplier has acted to satisfaction. If the jury can easily be convinced that ‘matter’ digital automation, in order to function flawless, is to be exact/100% predictable, it will bear serious consequences for any professional in and with the digital automation sphere.
More to be found …
bit.ly/2T5eQX5
Who’s we?
Copy/paste that dude’s name and “ax0#io” into google, replacing # with a dot. The first results claim he’s some consultant.
I’ve never come across some consultant with a weird website name like that, so I ain’t gonna click it.
“First I checked if they run a bug bounty program, rewarding those who find and alert them of security issues with their systems. They do not.”
Except they do: https://www.capitalone.com/applications/responsible-disclosure/ & it worked to catch this person.
A Bug Bounty program and a Responsible Disclosure program are different things.
A Bug Bounty program is a program where you are inviting researches to test your apps and you are compensating them somehow. Typically, compensation is based on the vulnerability identified. The scope of a Bug Bounty program is typically pretty strict.
A Responsible Disclosure program allows a researcher who identifies a bug to report it. You aren’t inviting researchers to test, you are simply allowing for the disclosure of vulnerabilities through a controlled process and offering some degree of safe harbor if some sort of test was run.
So, this is starting to make some more sense, because modsecurity would actually be running on the same instance as the application, making the AWS metaservice APIs accessible, but I still don’t understand how a configuration issue with a WAF would introduce an SSRF vulnerability. A vulnerability in the WAF yes, a configuration issue though?
The SSRF attack means a comprise was made on the server side and then redirected to the metadata URL . To prevent this the WAF should have been only be able to access the particular domains of the required Apache server and then a whitelist should have been created against the other web pages that are not in use or required i.e the HTTP metadata URL scheme. In this way SSRF would not have been possible.
In SSRF the exploit is usually part of some vulnerable query parameters. It’s not a redirect…the server fetches the resource. i.e. the server acts as an HTTP client, fetches the resource, and returns it to the original client.
I think that blaming the WAF is a bit of a red herring.
There was an account associated with the instance that had too broad of access.
There was an application vulnerable to SSRF.
There was a WAF that might have prevented exploitation.
That first one is probably the biggest issue.
Yes agreed. I also think there far far to much focus on SSRF here and not on protecting the actual data itself. An analogy would be someone picking a lock to enter a house (SSRF) but the valuables would be stored in a safe (encryption)
The SSRF exploit is only part of the problem.
Once the attacker had access to S3 via the API, the data should have been protected via SSE-C (customer managed server side encryption) which they would need to pass a rotated key as part of the request. Had this been in place its unlikely they would have been able to access any data.
https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
It looks to me like Capitol One were using SSE-S3 (AWS managed server side encryption) which is basically abstracted and transparent to the requester.
It maybe that the application in place that used the EC2 Role did not support using SS3-C and thus`SS3-S3 had to be used but irrespective of this I don’t think there is a full understanding of server side encryption and what this actually means when you have a data breach.
I agree with both of your assessments. The biggest issue (and one I see in ~70% of the clients I work with as a Cloud Security consultant) is that they granted too broad of permissions. “We need this to work – we’ll fix it later” – grants full admin to role, never revokes privileges.
The encryption would have certainly solved the issue, but even then, granting a write-only policy would have been preferred, with proper encryption added as a second layer of defense.
I truly wish AWS would change their certification program too. I had to recertify this year and IAM was virtually unmentioned in my testing (except the Security Specialty exam). In the Associate exams, they mentioned using roles versus users but never really went over policies or even touched on least-privilege and how to enforce that. Yet we wonder why people aren’t implementing security…
The impetus behind CCPA is to bankrupt violators. Had this occurred after 1/1/2020 the California CCPA fines alone would have been $750,000,000. PIPEDA allows for $100,000 CAN per occurrence. But Capital One is a victim just as much as we all are. Every single Capital One employee responsible for migrating them to the cloud jumped ship in 2016 and is now making the same mistake elsewhere.
I’ve long been a credit card customer of CapitalOne and whenever a charge is made to the card, I get an email notifying me of the charge. (Of course, software does that.)
What I haven’t received is an email notifying me of the data-breach. Or an email suggesting how it might impact me.
And the other thing I didn’t get was a warm and fuzzy feeling that CapitalOne cares about my financial health.
I think they’re sending out snail mail. Our daughter banks there, and a thin (read notification letter) mailing has arrived for her as of today.
Now the thrill of securing sales force running on aws really makes me anxious. So many places to screw yourself on security misconfiguration.
But if you pay them they will happily sell you plenty of add ons to implement their product correctly.
Buyer beware. Rings true.
I am not surprised a WAF is the issue. It could be me, but I don’t find WAF documentation to be very good. After getting way too many false positives, I ended up just using a scheme using regular expression matching and the Nginx map feature.
Note that blocking datacenters from accessing your website greatly reduces the volume of attacks. It is hardly security, but you look dumb when you get hacked from an IP that lacks eyeballs.
Some of this is not in my wheel-house, but good article anyway , since their is always new things to learn. :–)
Lucky me, since I do my best to not patronize companies that pay naming rights fees for stuff, I am not a customer of theirs. Of course, even though my bank is such a john, they might get hit next. (I was a customer long before they decided to try to influence people to buy their products by naming something).
I dun geddit. What I keep hearing is the defacto fact that no system is secure. One should assume (I do) that the most serious compromises remain unreported, or more likely haven’t even been detected yet. One conclusion? This whole internet thing is a can of worms and can Never work safely. NEVER. Safe solution? No more internet, it don’t work. Quit. It.. It’s time to go back to Kafkaesque solutions for handling big data.
Let’s try solving industrial problems that we can and should handle, like the vast conspiracy to ruin all temporarily neglected battery-powered devices, like flashlights. Now, this decades old leaky battery thing is something we CAN fix if we have the national will. Internet, No. Dependable flashlights, Yes!
Are you having a stroke?
We can make a difference. We all know bad guys spend all day probing good guys and all night collaborating to make tomorrow’s probes better. Let’s start a movement to adopt open with security and beat bad guys at their own game.
Here’s my part.
https://www.dgregscott.com/a-radical-not-so-new-idea-to-stop-the-daily-barrage-of-data-breaches/
This is good stuff…thanks! As a non-security-pro I am always looking for thing explained clearly.
ouch, the ROI on C1 buying a legit enterprise WAF instead of “FREE” open source WAF from MOD!!
better posture? probably but lols, they’d have someone else to sue!
“At Capital One, the use of AWS was to serve as proof of the bank’s forward-thinking approach to security, with Alexander once telling an AWS summit they could protect data “more securely in the public cloud than even in our own data centers.”
And yet, there never was a single cap1 breach during the decades of data being in the private data centers.
TJ and Greg,
You both hit the nail on the head. Some things just don’t belong in a cloud, online, where it can be remotely accessed by evil cretins.
This push for cloud computing is to outsource basic responsibility and have more of the company’s work handled by long-distance employees at half the prevailing wages. It’s a gimmick to save money.
I favor tax breaks for companies to maintain their own data centers locally and bring back any outsourced jobs. At the same time, I’d tax every byte of corporate data transmitted to the cloud.
Readership1, I’m not so sure it’s that black and white. The companies that operate public clouds gain lots of economies of scale and they can run server farms lots more efficiently than any enterprise organization I’ve seen.
But there ain’t no free lunch and I’ve seen too many decision-makers go to the cloud for the wrong reasons. I’m not ready to hammer Capital One for going to AWS. For all we know, that move may have enabled a bunch of business advantages.
If we’re going to pound the table, let’s pound the table for Capital One to adopt open and spill details so everyone else can improve.
– Greg
Greg S,
Fair counterpoint.
I certainly am not “hammering” Cap1 for leveraging the cloud. But as someone who is privy to inside information, I know for a fact that Capital One intentionally chose to not take a careful, methodical approach to going in the cloud in order to not make mistakes like they did.
Instead, the entire tech org (including their CyberSecurity team and leadership) was forced to go as quick and as fast as possible to AWS, and if that meant sacrificing a very thorough approach to security in the process, then that’s the risk the CIO was willing to accept.
Faster you get to cloud, the more profit you realize AND the more kudos the CIO would ultimately receive as being a “forward thinking, leaning forward, leaning in, disruptor”.
Again: internally at Capital One? The CIO no longer referred to Capital One as a Bank, but referred to it as a “tech company” that needed to act like..a tech company.
Those were mandates.
Brian – thank you for the great information, as usual. I was hoping you would come through on this! These breaches have mega downstream effects and as you’ve stated it would be much more helpful to have details so others can tighten up their controls. Glad we’ve got you to take care of that for the rest of us who need it.
The keyword not mentioned by the Security Experts here are Resource Policies. AWS access logic assigns highest precedence to Resource Policies. Essentially, one can attach a policy to bucket saying ONLY certain principals are permitted access and also, what actions are permitted.
Even if ever there was an excessively privileged id – and there sure will be when operating at a scale Cap1 or other big orgs. do, the resource policy will prevent access.
KEY LESSON: Define Resource Policies for all the services that support them. Industry as a whole has never managed to get privileges right and there are no hopes that this will be get where it needs to be anytime sooner.
Regardless the technical intrinsics, there seem to be a few legal issues at stake here. According to the court documents, it is very lightly that a former AWS employee, who is arrested, has stated that she used old AWS credentials to obtain entrance to the particular firewall.
Here is an entire other matter survfacing. If it is fact that AWS has a poor to no governance in changing password and credentials policy for staff leaving AWS, how liable does that make AWS? Since a contract only is a contract but not complying to the most essential governance in digital automation could make AWS liable.
Still many digital automation professionals don’t understand they are accountable for act and conduct in terms that every facet and instance thinkable, in order to function, is to be exact. And Exact here means 100% Predictable.
That also means that professionally any professional in digital automation, regardless if one is any administrator, programmer, engineer, consultant, in any discipline in digital automation, is accountable for ones act and conduct, also ‘lack of’…
There is another case going on on similar grounds, Hertz vs Accenture where Hertz lawyer is contemplating to go for a jury trial.
Most interesting to have a peek on the legal site of it since the question very well may be how liable and accountable AWS is in this matter and what the legal position of the Capital One customer is.
to be continued
Sigh, still no acknowledgement that cred based security is not enough. The s3 bucket should have only been listable from inside a vpc, whitelisted ips, a proxy or something. The fact that we just keep on allowing our apis to be hittable by Tor exit nodes because IP level security is “hard”, inconvenient, or not “cloud native” is the issue. If AWS had some damn security around it’s sprawling endpoints we wouldn’t be here.
There’s been no mention here in the UK media of Capitol One problems. Is their UK clients base small? Perhaps our GDPR plays a major part in that? My bank account was hacked on the 5th May – the notorious TV licence fraud. Santander, my bank, still have no idea how the hackers bypassed their security – they confirm it wasn’t anything that I did.
But the majority of the fraud loan/credit applications that I received as a result came from two sources: Newday (provider of 40% of UK credit cards/car loans – mainly the cheaper end & those with bad credit history) and Amazon (fronting for Newday). Incidentally. one of the Amazon hackers was very generous, putting his £700 loan into my bank account. I made them wait 3 months before returning it.)
Everything that anyone would like to know about the Capital One breach and essentially all other massive security breaches is summed up exceptionally well in a set of short videos available on YouTube. Just go there and search for:
The Expert comedy
View all of the search hits that come up and that feature an Asian engineer guy in a light blue shirt.
These videos depict the reality of the day-to-day life of the typical software and/or network and/or application and/or database engineer, and THIS is the actual problem. Our tasks, schedules, and project goals and designs are largely or entirely driven by managerial morons and their even less clueful marketing department bootlickers, none of whom. in general, actually have the first approximation of a clue. And worse, they are steadfastly IMPERVIOUS to all attempts to give them clue, because they are all utterly convinced of their own brilliance and infallibility.
Watch the videos. They explain everything.
I am see a lot of confusion. In fact, much of the confusion is caused by Cap 1 and Amazon trying to deflect blame – and probably a number of PR Firms trying to confuse the situation.
Brian’s first post on the Cap 1 data breach indicated a insider threat: ‘Ray Watson, a cybersecurity researcher at cloud security firm Masergy, said the Capital One incident contains the hallmarks of many other modern data breaches. “The attacker was a former employee of the web hosting company involved, which is what is often referred to as insider threats,” Watson said..”‘ -Brian Krebs
https://krebsonsecurity[.]com/2019/07/capital-one-data-theft-impacts-106m-people/
That “insider threat” makes sense. Brian’s second post on the Cap 1 breach seems to reinforce that idea.
Brian interviews Evan Johnson [who works for a competitor Cloudfair]:
“There’s a lot of specialized knowledge that comes with operating a service within AWS, and to someone without specialized knowledge of AWS, [SSRF attacks are] not something that would show up on any critical configuration guide…” -Evan Johnson
https://krebsonsecurity[.]com/2019/08/what-we-can-learn-from-the-capital-one-hack/
[links broken]
The idea that some guy on the inside of Amazon had a good idea of how to break into to AWS servers seem reasonable. Now, Even Johnson says that Amazon is a fairly easy target for hackers:
“What are the drawbacks to the metadata service? … Accessing the credentials is easy! The metadata service provides temporary credentials that services can pull out of thin air…It’s extremely easy to access the IAM Role temporary credentials that are provided by the metadata service. This is an example of how easy it is access the metadata service. There is no authentication and no authorization to access the service…All HTTP Requests are trusted… It is a dangerous assumption to offer only an HTTP based metadata service because the world has shifted where most services are communicating via HTTP. My rule of thumb is that anywhere a hostname or IP address is accepted as user-input there is a 50% chance that SSRF is possible, and an attacker could potentially make requests to the metadata service…”- Evan Johnson, manager of the product security team at Cloudflare
https://ejj[.]io/blog/capital-one
Judging from the above, I would say Amazon is not very safe! And, I would say the obvious solution for Cap 1 includes the Non-Use of Amazon’s over advertised “Cloud Service” for any critical banking data at all. Further, Cap 1 should probably move their customer’s Credit Applications to a much more secure solution as quickly as possible. Lastly, any major bank should carefully consider all risks associated with all Cloud Services until proven Very Safe.
This has been going on for far too long. Not that I know how to implement this or even if it is feasible, but I would think that in the world of AI that the major players could afford to do the research and the cost to plug in all the known issues into a specialized software solution designed to lock down security and prevent this kind of breach. At some point it would seem that this kind of software would be superior than one dimensional human “experts”.
Maybe the reason this keeps reoccurring is because the business model is driven by profit and the penalties are not severe enough to require companies that deal with financial data to give a rip. How about making the monetary penalties so overwhelming that they put security first in every part of their business?
If they were forced to perform or die in bankruptcy something tells me they would make the effort or voluntarily close down the business. If they refuse they deserve to die in bankruptcy, after all, they gather our data without our permission and sell it to a 3rd party for a profit and they also loan money for interest. Why does anyone care if they survive. I sure don’t.
Capitol One, and everyone who learns from their mistakes, are very fortunate. This crime harkens back to yesteryear when hackers did their deeds for bragging rights and prestige, not for the money.
OK, the big question is what did the Cap One hacker do with the data? Maybe it was smart enough not to release it into the public. If released, help it along with a castration….without anesthesia. Until there are real consequences for this stuff, without presidential pardons (Obama) it will become more prevelant.
I see now that it was released into the public domain. These people should fry…..death penalty for hacking AND releasing people’s private information.
Appreciating the commitment you put into your site and detailed information you provide. It’s good to come across a blog every once in a while that isn’t the same outdated rehashed information. Excellent read! I’ve bookmarked your site and I’m including your RSS feeds to my Google account.
Would this not happen if hosted privately, instead of public cloud?
An opensource software was the opening door, which could have also happened internally. But then, the amount of data exposed due to it is the problem, where public cloud provider may have to answer.
Or may be the application miss to give granular access control, knowing of the shortcomings in the metadata services. It is easy to pin point that now, as this has already happened.
Is it a mistake in application implementation or security flaw of public cloud vendor?
Was the shortcoming of metadata services not known before development?
I think the bank may have to answer this hack.
This is a wake up call for others. Are you wasting time and money with too much focus on paper policies, or really getting down to practical implementations of security?
One aspect of the issue, is rushing. CIO’s goal of being first bank to be fully in cloud by 2020 has caused intense pressure on internal security teams and prevented careful, methodical, relentless focus on security practices, especially around monitoring of assets containing customer data. Other than making claims that cloud is more secure than private data centers, the CIO rarely (ever?) mentions security and protection of customer data in any of his interviews, AWS ReInvent talks, etc. (He actually doesn’t call Cap1 a bank anymore, he calls Cap1 a Software Company). Also, to have gone 127 days without any hint that your customer data was flowing out of your door as a bank, is wholly unacceptable.
I agree with your comment, the Equifax breach was not in the Cloud. To the contrary it appears that because Capital One went with Modsec as the firewall instead of AWS’s WAF offering, a vulnerability was introduced. It’s likely Modsec was installed/built “on” an EC2 instance (anyone correct me if I’m wrong). I say that because I don’t see any offerings in Marketplace.
That being said, comments that state it was the rush to the Cloud that was the factor…that could be very well true. It depends how all of this was architected, how they setup/integrated their security controls/logging, etc.
I think that’s the key: planning and implementation.
As with any on-premises solution, all of this needs to take place as well. So I agree with your comment, this would have happened on-premises as well.
I was a CIO for 20 years (now a COO). Security was always top of mind and I spent time everyday reading the latest security issues, talking to staff, etc.
First we all have to remember that humans screw up. As a CIO I had no idea when they did and some mistakes were larger than others.
Where I still differ with companies like Capital One is the cloud. We have very sensitive data and do not use the cloud. I feel we have more control and frankly do it at a lower cost overall.
What we all have to face is how do we detect intruders who have made it through whatever wall we have built around our systems. That leaves some very difficult challenges since data can be stolen in seconds.
Companies like DarkTrace offer interesting products that sniff out different network patterns. (Not selling for them but like the product). While DarkTrace works pretty well, it will have an occasional false positive and you have to decide if you want it to block new patterns.
I’m not apologizing for Capital One, Equifax, etc. My smaller company is way ahead of them. But that sticky issue with human error will always be there. Capital One has the resources to catch something like this – I worry that we have enough solid talent to protect all of this data.
Dave, the solution to catch those human errors is to adopt open for all things security related. Share everything you do in forums, present it at conferences, publish it, and subject it all to public scrutiny. Expect everyone else to do the same. Bad guys do it – why not good guys? Beat ’em at their own game, catch the configuration mistakes early, and continuously improve all those skillsets for your people.
– Greg
The dollar amount for security breach fines is meaningless and will do nothing to fix the problem. The real solution is a combination of PERSONAL financial liability for the executives in charge and CRIMINAL charges filed against them.
Until they have personal (not corporate) skin in the game, no amount of fine will change behavior. As someone else said, if a fine is large enough, the corporation will simply file for bankruptcy protection. And at lower amounts, it’s the shareholders who will ultimately pay the fine, not the executives.
Imagine the next Board meeting at most large companies after an executive at another company is personally fined and sent to jail.
Scott,
Reasons why your idea would fail and is idiotic:
A. It would penalize companies for being located in the US, as no other jurisdiction does this. That “giant sucking sound” is your idea removing millions of jobs, worsening everyone’s lives, instead of helping.
B. It would criminalize mistakes, as well as fraud and negligence. There are already laws against fraud and financial regulations against negligence, so this is really a way to hurt those who make mistakes.
Mistakes should not be criminalized; if it were, your grammar would land you in Leavenworth.
C. Your idea would punish executives for doing what they should be doing, hiring knowledgeable experts with IT skills to execute their ideas. Higher management are not subject matter experts and can’t reasonably be expected to know a WAF from a WAN. All they can do is hire a second team to audit the first, or an outside consultant, and rely on their reports.
That’s standard business practice. And you would punish them all for not catching mistakes.
And then you’d punish the IT workers, too, for carrying out the executives’ ideas. Jerk.
D. No industry succeeds in which errors are punished. The military, healthcare, and aeronautics industries have greatly reduced errors (and deaths) by instituting safe harbor rules for reporting errors and helping each other fix problems.
E. The jails are not big enough to handle everyone you seek to incarcerate. From new hires to CEOs, you would be locking up millions of non-violent people.
F. The economy would instantly collapse. No company or executive would take risks, so innovation would end. Profits that help fund everything from schools and roads to your retirement would be curtailed. Companies would have to reduce entry level jobs, to pay more to retain their frightened top talent, leaving millions of low paid folks unemployed. Or they’d just leave the US entirely.
G. Corporate fines never change corporate behavior. The cost just gets shifted to consumers and that harms consumers. Oh, it’s great for government, but lousy for the people that government serves.
“Mistakes should not be criminalized”
Actually, in your haste to attempt to belittle the prior poster(vs simply posting a respectful rebuttal), you posted an opinion that is fundamentally flawed.
Because if a driver mistakenly ran over your family? I’m sure you’d be in full support of the DA/Prosecutor pressing criminal charges against the driver for his/her mistake.
Observer,
Unless the driver engaged in malicious, neglectful, reckless, immoral, or careless behavior, I would not ask for criminal charges.
I’d bring a lawsuit, of course.
I might even resort to punching the driver.
But not every bad outcome deserves incarceration.