Hundreds of millions of Facebook users had their account passwords stored in plain text and searchable by thousands of Facebook employees — in some cases going back to 2012, KrebsOnSecurity has learned. Facebook says an ongoing investigation has so far found no indication that employees have abused access to this data.
Facebook is probing a series of security failures in which employees built applications that logged unencrypted password data for Facebook users and stored it in plain text on internal company servers. That’s according to a senior Facebook employee who is familiar with the investigation and who spoke on condition of anonymity because they were not authorized to speak to the press.
The Facebook source said the investigation so far indicates between 200 million and 600 million Facebook users may have had their account passwords stored in plain text and searchable by more than 20,000 Facebook employees. The source said Facebook is still trying to determine how many passwords were exposed and for how long, but so far the inquiry has uncovered archives with plain text user passwords dating back to 2012.
My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
“The longer we go into this analysis the more comfortable the legal people [at Facebook] are going with the lower bounds” of affected users, the source said. “Right now they’re working on an effort to reduce that number even more by only counting things we have currently in our data warehouse.”
In an interview with KrebsOnSecurity, Facebook software engineer Scott Renfro said the company wasn’t ready to talk about specific numbers — such as the number of Facebook employees who could have accessed the data.
Renfro said the company planned to alert affected Facebook users, but that no password resets would be required.
“We’ve not found any cases so far in our investigations where someone was looking intentionally for passwords, nor have we found signs of misuse of this data,” Renfro said. “In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We want to make sure we’re reserving those steps and only force a password change in cases where there’s definitely been signs of abuse.”
A written statement from Facebook provided to KrebsOnSecurity says the company expects to notify “hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users.” Facebook Lite is a version of Facebook designed for low speed connections and low-spec phones.
Both Github and Twitter were forced to admit similar stumbles in recent months, but in both of those cases the plain text user passwords were available to a relatively small number of people within those organizations, and for far shorter periods of time.
Renfro said the issue first came to light in January 2019 when security engineers reviewing some new code noticed passwords were being inadvertently logged in plain text.
“This prompted the team to set up a small task force to make sure we did a broad-based review of anywhere this might be happening,” Renfro said. “We have a bunch of controls in place to try to mitigate these problems, and we’re in the process of investigating long-term infrastructure changes to prevent this going forward. We’re now reviewing any logs we have to see if there has been abuse or other access to that data.”
Facebook’s password woes come amid a tough month for the social network. Last week, The New York Times reported that federal prosecutors are conducting a criminal investigation into data deals Facebook struck with some of the world’s largest tech companies.
Earlier in March, Facebook came under fire from security and privacy experts for using phone numbers provided for security reasons — like two-factor authentication — for other things (like marketing, advertising and making users searchable by their phone numbers across the social network’s different platforms).
Update, 11:43 a.m.: Facebook has posted a statement about this incident here.
More ammunition supporting Elizabeth Warren’s push to impose stronger regulatory controls on (and possibly antitrust actions against) FB, I would think.
I just do not have Facebook or any of those sites. So, I do not have to worry about stuff like that. I do not have any credit cards, any debt and I do not believe in storing anything personal or important on a computer.
How silly!
so wat u have?
probably real hard assets… you know sh*t that is actually real, not fake like our current monetary system. Gold and Guns baby (and of course some USD even tho everyone knows its not real)
Identity theft and impersonation happens to some people.
Identity theft and impersonation happens to everyone.
And yet you post with what looks like your real name, thats pretty dumb.
Touche’!
End of message.
Liz’s stance on anything is irrelevant. She is powerless, toothless, and desperate. But, otherwise, yea… this.
As compared to Cheeto Mussilini whose toothless rants on twitter are likely to lead him to jail and who has lost over 90% of the cases brought against the WH for illegal laws he tried to pass? She has a lot more power than him.
There is no reason for “antitrust” anything regarding Facebook. There’s nothing to split up. Elizabeth Warren is an idiot.
You could split up Instagram, Whatsapp, and Facebook. All are part of Facebook.
Elizabeth Warren is only 1/1024 relevant.
Gee…that comment certainly triggered a reaction from a few of the more trollish readers, so there must be more than a bit of concern at the prospects of a regulatory crackdown on FB with a prospective change in administrations after 2020.
So, we’re saying that at least 2,000 Facebook employees have had access to account IDs and their associated passwords – the full set of credentials needed for most people to log on. They have no idea where those passwords may have been saved or otherwise exfiltrated from the system by those who accessed them. Yet we have no need to worry at all, and no need to change our passwords? I would recommend every FB user change their password immediately. Especially considering, as someone else wrote earlier, that FB identity federation is used on many non FB sites.
Stay safe, folks.
@Scott
” I would recommend every FB user… quit Facebook …immediately.”
FTFY
I did quit and my profile is still on there. Have tried 3 times and still no luck.
OK, so you quit. You delete your profile. 1) Do all your likes and comments on other user’s accounts disappear? 2) If FB is later served with a subpoena to provide information from your account, is that impossible because the account has been deleted from all the backups? 3) If a disastrous breach occurred whereby FB customer data were snarfed up by hackers, could “deleted” accounts be included in the breach, or not?
I’m sceptical that a FB account, once created, can actually be deleted (expunged, disappeared, vanished) by a simple user request. Perhaps a law enforcement request (think about the witness protection program) could completely wipe all record of such an account from FB’s servers. But, er, I’m sceptical.
Especially when FB has “shadow profiles” on people who haven’t signed up. Their algorithms pluck identifying information about non-members from posts by their members and assemble the profiles from them. The reason, ostensibly, is that it will make the FB user experience much richer for when those non-members finally cave in and sign up.
So even non-members can be victimized by any actions FB takes or doesn’t take.
The EU has privacy laws that include “the right to be forgotten” which forces companies to prove that they have, in fact, expunged your info/data.
Does anyone know how FB complies with those regs for EU citizens? Are they somehow exempt because the function in the US and are essentially a voluntary membership?
Tom – Agree! I quit FB years ago, around the Snowden times. But I recognize that not everyone wants to do that. So, given that, the best I can do is recommend some password hygiene. There is no amount of preaching I can do that would convince folks to quit FB if they haven’t already decided to do so on their own. Thanks for the reply!
In case anyone is wondering how to create a secure password, here’s a little guide: https://gobestvpn.com/guide/password-creation-bible/
That’s quite a page full of good password tips. However, I believe the best system out there is Diceware. Just google it and then roll the dice!
I do not believe this was an accident. Even first year graduates know you always hash passwords. It is simply not believable that an organization as big as Facebook would allow plaintext unless it was officially sanctioned somewhere.
Regulators should investigate what Facebook was doing with these passwords. What types of users were targeted? Were they journalists? Officials? Competitors?
Or, more likely, this indicates that the bar for becoming a software engineer at FB is much, much lower than ‘first year graduate’.
The post doesn’t say that passwords weren’t hashed in their storage location, but that they were *logged* unencrypted. That’s an easier mistake to make, if you routinely log requests. I dare say some graduates have even made this mistake from time to time. Easyish to do if you’re at one end of the Dunning Kruger curve.
Why in this day and age are unhashed passwords even hitting the wire in the first place?
There ought to be an authentication protocol agreed upon between the browser and the web server, wherein the browser only ever transmits hashed passwords.
Had a system like this been in place, it would not have been possible for this to have happened, since FB would never have received any plaintext passwords in the first place.
Am I missing something?
It is still very common for the browsers to send the password in clear text as part of an HTTP request. It could be encrypted by the TLS, but that doesn’t protect them from being seen on the server side. There are a few sites that encrypt the passwords before they hit the wire using Javascript but those are not very common.
It doesn’t matter where it was stored in plain text. If you work on Facebook, you should be aware of it. Plus, why the fuck do logs store passwords?
>”There are a few sites that encrypt the passwords before they hit the wire using Javascript but those are not very common.
That’s the problem. It’s being left up to the individual sites to implement. This all needs to be built into the browser, and defined as an open standard. That way everyone can use it and benefit.
You know how Chrome is deprecating weak TLS configurations, and indeed aims to deprecate plain old HTTP itself? Plaintext auth should suffer the same fate.
Some commenters say “Oh, but the server needs to see the plaintext so it can validate that it contains enough special characters, etc.” I disagree. I think that the browser can handle this as well. If they’re that concerned about it serverside, they can always expend a bit of compute power to try and break the hashes, like an attacker would.
I’d dare to guess that most websites are not encrypting the password as it is entered by the user. Sometimes you’ll see one that hashes it client side and sends that, but that’s got its own set of problems.
Remember that the encryption used to prevent someone from intercepting the password isn’t leaving it encrypted on the server that is receiving it
Secure websites need to validate passwords server side as all client-provided content should be treated as untrusted. If a user selected “password” as a password (common) and it was securely hashed client-side before being sent to the server, there would be no way for the server to check that the password was sufficiently secure. Instead, the password is transmitted in the original request (sent encrypted over https and decrypted server-side), checked against the rules imposed by the site, hashed (if password storage is done correctly) and stored. If monitoring software is also present on the server (as is the case here), then it may also have access if mistakes are made. It pays to be careful, and it seems Facebook weren’t careful here.
If the client hashes the password, then the hash itself effectively becomes the password. This is known as “pass the hash”. The practical security effect of client hashing is the same a storing it in clear text on the server.
The hash cannot be computed in the browser. Any decent password management system is going to use a salt for each password. In order to compute the hash, you have to have both the plaintext password and the salt. And since sending the salt to an unauthenticated browser is a really bad idea, that only leaves computing the hash on the server.
Protocols like SRP implement a way of authenticating without sending the password over the wire. You can happily log all the request/response data for the entire process and there won’t be a password in there anywhere.
I also agree about your point of implementing this sort of thing in the browser. It would not be hard to do. There would need to be matching logic on the server-side, which could be handled in-language, or (preferred) contributed to the common open source server platforms, such as Apache and Nginx. Load balancers and cache fronts don’t need to worry about this, they just pass it all through.
It doesn’t say that they ONLY logged. It also says that they STORED the passwords unencrypted. So, regrdless of carrying the password unencrypted in requests, it conveys that they had also stored it in their DBs unencrypted.
Wrong. The article says they were stored plaintext. They were discovered because of log output.
Renfro said. “In this situation what we’ve found is these passwords were inadvertently logged…”
Logs ARE stored. Not stored in the database that houses account information like the hashed passwords.
FB still hashed the stored passwords.
Web traffic logs might get sent to several places for storage. And if they don’t actively scrub/mask the log entries containing passwords, they would have this exact problem.
As stated previously, TLS is really the industry standard for encrypting in transit, even passwords. Once decrypted on the webserver, there are checks that need to happen. No good, secure, or smart way to hash on the client side. Never trust client.
Entirely possible but, even so, their work would be monitored by someone higher up.
As well, password security in large organizations is so fundamental that it’s impossible any staff would not be aware of the need to hash. They would hear it mentioned in general conversation and meetings.
Again, this has nothing to do with the password storage policy. FB does hash their passwords at rest. Logging at the web server, however, needs to explicitly be configured to either not log authentication parameters like passwords, or automatically redact the password from the log entries.
From the description it sounds like passwords were stored properly in the database, but the application logs contained network requests containing passwords.
There’s been a few incidents like this where log applications didn’t automatically redact sensitive data in logs
Don’t underestimate the power of stupid.
People are prone to making mistakes, and this one was not a simple mistake.
Regardless don’t forget to set up secure passwords https://gobestvpn.com/guide/password-creation-bible/
Facebook’s employees aren’t even employees. They are all contractors — most of whom are overseas. Manila is their largest operation. I think the number of people contracting FIRMS is 4,500. Facebook monitors aren’t monitoring — they are pretending to be algorithms. Increase user engagement and spy on those FB or their customers ID as high value. FB knows it has a problem with contractors — something happened a few months ago in Manila. Maybe someone stole data… that’s what I inferred. Maybe they are being blackmailed? If an employee is coming forward with a partial story… it is spin. Why say unencrypted email addresses if nothing happened… something happened.
So big question – Were FB username and/or email addresses tied to these plain-text passwords? Or was this just merely a long text list of millions of random passwords?
Obviously 20,000 facebook employees would not be searching the file if it only contained random gibberish.
Big deal?……NOT. If the idiots that use FB to see who’s eating what for breakfast and are responsible for sitting thru a green light while they check their “status”, do not know when they signed up for this stupid forced intrusion and logging of everything and everywhere they go and everyone they ever knew, then they got exactly what they deserve. Of course these idiots are not intelligent enough to not use the same PW on multiple accounts! The government is NOT going to do anything to punish FB, b/c they are using their user profiles to supposedly thwart terrorism…….LOL. WTF…this is AMERICA where ANYTHING GOES!!!
Your comment is ludricous. Are you always way off the mark, or just most of the time? Your replying is so laughable that people must think you’re a bit mental.
Not storing passwords in plaintext is one of the first security measures that a software developer learns. It’s not even Security 101; it’s Security Preschool. You don’t need to be making excuses for these guys. Most of them are pulling in a healthy six figures for a quality of work that is downright embarrassing.
By the way, Facebook is tracking you whether you’ve ever made a profile or not. They compile a hidden profile about you based on the data given to them by your friends/families/coworkers/acquaintances, and also track your device fingerprint through sites that integrate with Facebook plugins.
But this isn’t even about password storage. It is about log storage, which is always plaintext. This is a major incident where passwords were in the logs.
Unfortunately, a very common misconfiguration mistake. There is always a fight between security and privacy. One wants to log everything, and the other wants to log nothing.
Big deal?……NOT. If the idiots that use FB to see who’s eating what for breakfast and are responsible for sitting thru a green light while they check their “status”, do not know when they signed up for this stupid forced intrusion and logging of everything and everywhere they go and everyone they ever knew, then they got exactly what they deserve. Of course these idiots are not intelligent enough to not use the same PW on multiple accounts! The government is NOT going to do anything to punish FB, b/c they are using their user profiles to supposedly thwart terrorism…….LOL. WTF…this is AMERICA where ANYTHING GOES!!!
Your reply is identical to “Muhammid”‘s above. Why is that ‘Adolfo’?
Looking at the timings, would be interesting to work out which troll factory produced the comments……
Simple minds in different troll shops working from a common script, or a single mind (a really stupid one, at that) submitting the exact same blather under a different name a few minutes later by thinking that doing so could fool this crowd that follows BK — not many other options than that.
Damn, now everyone will know what I had for dinner yesterday.
Okay – I LOL’d. 🙂
Mark Z don’t care about users privacy. hes been quoted as saying “users are dumb fks to trust me” or something to that effect.
Didn’t get a email notification on this one?
What???
My account was deleted and I couldn’t get my email keep making me change my email and now I can’t get my mail
Go ahead hack me…I don’t like Facebook anyway…
Facebook has a Password Heaven!!
FYI: In the OWASP ASVS V2 Authentication, Level 1 apps(low risk)doesn’t require “credential encryption” and “secure storage”.
Users may have apps like this only for extremely low risk purpose.
Do not overestimate ASVS L1 apps.
All my information is bogus anyway, so…. who’s worried?
And if they did steal anything, my credit is so bad, they’d give back my identity with a letter of apology, and one hundred dollars.
So when you send credentials to a website (response to a 401 challenge using Basic Auth) you will see an Authorization header with base64 value of username and password. If you are logging headers, which you probably are, you will see that this kind of stuff is more common than you’d think. The only reason they are getting a hard time is because it is facebook. If this was your Credit Union.. you’d totally let them get away with it
I still got my old Facebook open, I forgotten the pass words I call Facebook, and is a wasted the time, I thinking to close the new Facebook I have, and guit I realy don’t need this tipe of things in my life I think is much better how use to be, and living my life like the old way, thanks God I never put my real information,
When will you make it where people who are shut out of Facebook and Messenger be able to recover their accounts?? How long will it be? I’ve been shut out since 6:30 pm March 20th (EDT). I have too many family members and friends to create a new account. For 24 hrs I kept being told I’ve had too many security messages and to try back later. When is later enough??
Amex did the same while I was working there. Sometimes it is just stupid developers
A blast from tbe past, from business insider:
Mark used his site, TheFacebook.com, to look up members of the site who identified themselves as members of the Crimson. Then he examined a log of failed logins to see if any of the Crimson members had ever entered an incorrect password into TheFacebook.com. If the cases in which they had entered failed logins, Mark tried to use them to access the Crimson members’ Harvard email accounts. He successfully accessed two of them.
Media and social media are for fools, children and foolish children. Anything with advertising is junk.
Like, for example, this website which you are posting on? I guess you’re a trash collector then eh?
I have two Facebook accounts that have been trying to delete it ask for my password when I put it in it doesn’t work then I changed my password and it still doesn’t work I’m really confused I want them close because someone login to my account in Seattle Washington and I live in Wrangell Alaska
Hey! I pass through Wrangell once a year, on my way to Coffman Cove.
Get off the plane and onto a Breakaway boat.
I laid over there for the day on one of my trips. The museum was very interesting.
Alex Stamos was head of security at two of the largest internet companies, both of which were storing hundreds of millions of user passwords in cleartext. Yet his reputation as a security expert is still golden. Why?
He is still golden because a lot of his failures have been covered up or misunderstood. His main job is to tell the boss to stay the course and that everything is fine. Lots of CEOs like that in a security chief.
I had to get a new phone so now I’m locked out and Facebook will not send me a new password. I’m very upset in don’t understand why I should be O to access my Facebook my pictures
Unfortunately … not at all surprising.
I mean “how does protecting user credentials generate revenue” …
… answer: It doesn’t … so they DON’T CARE.
Brian: Any word on if the shared secret their 2FA (via auhenticator app) was either bypassed or outright compromised ?
We just changed password but the 2FA appears to be deactivated (@ 1120 EDT)
That’s totally wreckless of Facebook. I don’t know if it’s an employee issue or not but that’s totally unacceptable.
Apparently Facebook is going to pretend that disgruntled employees aren’t a prime attack vector. *sighs*
Grounds for immediate termination
just wanting to thank brian-kreb for choosing an orange link-color (discernible) … unlike facebook’s links which are almost identical to “black” text.
I’m not sure exactly why but this site is loading incredibly slow for me.
Is anyone else having this issue or is it a problem on my
end? I’ll check back later and see if the problem still exists.