Hundreds of millions of Facebook users had their account passwords stored in plain text and searchable by thousands of Facebook employees — in some cases going back to 2012, KrebsOnSecurity has learned. Facebook says an ongoing investigation has so far found no indication that employees have abused access to this data.
Facebook is probing a series of security failures in which employees built applications that logged unencrypted password data for Facebook users and stored it in plain text on internal company servers. That’s according to a senior Facebook employee who is familiar with the investigation and who spoke on condition of anonymity because they were not authorized to speak to the press.
The Facebook source said the investigation so far indicates between 200 million and 600 million Facebook users may have had their account passwords stored in plain text and searchable by more than 20,000 Facebook employees. The source said Facebook is still trying to determine how many passwords were exposed and for how long, but so far the inquiry has uncovered archives with plain text user passwords dating back to 2012.
My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
“The longer we go into this analysis the more comfortable the legal people [at Facebook] are going with the lower bounds” of affected users, the source said. “Right now they’re working on an effort to reduce that number even more by only counting things we have currently in our data warehouse.”
In an interview with KrebsOnSecurity, Facebook software engineer Scott Renfro said the company wasn’t ready to talk about specific numbers — such as the number of Facebook employees who could have accessed the data.
Renfro said the company planned to alert affected Facebook users, but that no password resets would be required.
“We’ve not found any cases so far in our investigations where someone was looking intentionally for passwords, nor have we found signs of misuse of this data,” Renfro said. “In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We want to make sure we’re reserving those steps and only force a password change in cases where there’s definitely been signs of abuse.”
A written statement from Facebook provided to KrebsOnSecurity says the company expects to notify “hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users.” Facebook Lite is a version of Facebook designed for low speed connections and low-spec phones.
Both Github and Twitter were forced to admit similar stumbles in recent months, but in both of those cases the plain text user passwords were available to a relatively small number of people within those organizations, and for far shorter periods of time.
Renfro said the issue first came to light in January 2019 when security engineers reviewing some new code noticed passwords were being inadvertently logged in plain text.
“This prompted the team to set up a small task force to make sure we did a broad-based review of anywhere this might be happening,” Renfro said. “We have a bunch of controls in place to try to mitigate these problems, and we’re in the process of investigating long-term infrastructure changes to prevent this going forward. We’re now reviewing any logs we have to see if there has been abuse or other access to that data.”
Facebook’s password woes come amid a tough month for the social network. Last week, The New York Times reported that federal prosecutors are conducting a criminal investigation into data deals Facebook struck with some of the world’s largest tech companies.
Earlier in March, Facebook came under fire from security and privacy experts for using phone numbers provided for security reasons — like two-factor authentication — for other things (like marketing, advertising and making users searchable by their phone numbers across the social network’s different platforms).
Update, 11:43 a.m.: Facebook has posted a statement about this incident here.
Seldom have I seen a more pathetic example of complete and utter cluelessness than Scott Renfro’s statement? He doesn’t think there is a risk in logging cleartext passwords? He thinks they would know if someone “looked intentionally for passwords”? What signs of abuse is he looking for? (I see questionable facebook accounts all the time.) He doesn’t think ALL of the passwords have been compromised?Where did they find this guy? He needs a different career.
He’s less speaking as a “software engineer” and more as “damage control” for FB. This is a pretty transparent and honestly pathetic attempt at reassuring the userbase that no harm came from this disastrous chain of events. So from our perspective, yes, he is an idiot. From FB’s perspective, he’s lying out his ass to protect the company from larger backlash.
The database was only available to Facebook employees. Apparently, there was some kind of logging in place to track who accessed it and what fields they requested. “Questionable Facebook accounts” don’t even come into it. You obviously don’t know how to parse the scope of this.
Given the competence shown in both the statement and with what happened, I would not be surprised to learn that all the logs for the access to that data was stored in a basic text file on the virtual desktop named something like “Log File for Plain Text Password Access – Pls don’t edit”.
Right. How did they differentiate a regular read of a log for diagnostic purposes from someone intentionally looking for the passwords. What a B.S.
Never mind the fact that ALL passwords evidently at Facebook at encrypted and not hashed. Just horrible all the way around, I am surprised Brian did not ask about this practice for the rest of Facebook.
“Renfro said the issue first came to light in January 2019 when security engineers reviewing some new code noticed passwords were being inadvertently logged in plain text.”
…inadvertently logged in plain text.
What was inadvertent? Software engineers did this deliberately.
Not necessarily. As a software engineer, I inadvertently introduced a bug into a code fix that almost got my company sued. If you don’t thoroughly inspect, review, and test code changes, all kinds of unanticipated side-effects can happen.
It does not matter whether Facebook, Twitter whoever as long as you are dealing with human beings you always have to be vigilant. Life goes on you can’t make these things stop you.
This could be the reason I’ve had notifications that my password had been changed. I had to change my password from FB, Amazon, Spectrum a few CC’s including my bank account. I also receive calls every day from Slovenia which if not mistaken is Russia. Where’s our privacy?
Yes, you’re mistaken. Slovenia is a country in Europe. It’s not part of Russia.
Slovenia is not Russia – it’s part of the former Yugoslavia, and is just south of Austria.
I ended up having to change all my passwords as well!
Then you should be using a password manager and auto generating unique passwords. Don’t use non-unique passwords, very bad practice.
Slovenia is not russia, it’s a small country in europe, right next to italy.
It’s all out there,your kidding your self if u think you have any privacy.
*you’re *yourself *you
And no, it’s not. You silly troll.
Assuming 10% truth, what about the other 90% they aren’t sharing?
FacePalm
This is horrific. I work at a similarly large tech company in the Bay (our offices are actually very close to each other) and this sort of thing is very carefully monitored — *any* sort of personally identifiable information in logs is a huge no-no, monitored for carefully, and if a slippage ever occurs an internal impact review and follow up report are 100% mandatory (followed by evaluation of your position within the company…), and we’re not nearly as hip/modern/cutting-edge as Facebook.
I don’t know why I always expect better of them when I’m continually disappointed time and time again.
This is especially nice given you can use your Facebook identity as federation for signing on to other websites.
Exactly. Which is the very reason why I never do such things. Compartmentalize as much as I reasonably can.
How convenient!
All I want to know is why they were EVER in clear text. And that the highest manager over the person(s) responsible, are fired and banned from every using a computer again.
You type your password in clear text. It’s sent to the server in clear text (though encrypted by ssl). Then the server reads that and hashes it then. Sounds like some programmer logged the password before it was stored.
It’s most likely URLs and/or POST data that ended up being logged into an activity log or such. Maybe even just an httpd access log.
POSTs are not logged in access logs, only gets.
Which is why I wrote the above. If the passwords are sent in a GET, even an access log is enough
2019 and organizations still don’t take the matter of passwords or encrypting personal information seriously.
I work at a software Dev company, a MUCH smaller company, around 100 employees. Back in the 90’s, I saw this type of thing going on.. In the 2000’s this was enforced and they even scanned the network for passwords, social security numbers, credit card numbers to male sure some employee didn’t have a flat file on their workstation woth this data. We had VB scripts that decrypted files to work on issues and we had to delete them as soon as we were done a d those files had to be decrypted and stored on a secure server, not gour own machine. This happening at a company of this size in this decade is beyond crazy! Get ready for the lawsuits!
Really you want the public to believe that no one misused our
passwords! That is why my Facebook has been hacked,
my mother’s Facebook has been hacked, my next door
neighbor Facebook has been hacked, my best friend Facebook has been hacked
and several other people I know all in the last year had their Facebook hacked ! These companies are foolish in their thinking and public relations, they
need to stop thinking that the majority of their customers
are stupid. It is really quite insulting!
This is insane
Nice work Brian! Facebook should offer users a password management feature to show a real commitment to user security! By allowing users to encrypt their passwords on the client side, this issue can be completely eliminated.
As others have mentioned this is basic security. Only hashes are ever stored.
Obviously Facebook from day 1 only interest has been to make money off its users anyhow, anyway they can.
In a normal company the CEO would be removed but I guess this won’t happen at Facebook.
While I have an account, I rarely have used it and generally only used it to view other accounts online since I try to keep a low social profile.
It is a pretty stunning admission.
These are logs, not stored passwords. Logs are not parsed for sensitive data and then hashed before they are written to disk. If anything sensitive data might be redacted with asterisks, but not hashed. There’s no reason for that. But instead it should be left out altogether (not logged) in any form.
Brian —
Could you add a sentence or two to the article about your views on 1) whether industry practices agree with Renfro’s assessment of risk, 2) whether it is likely that there would be evidence of misuse, and 3) whether industry practice would usually include a password reset?
I think it would be helpful for the non-technical press, and the industry in general. Even if your view is that industry wisdom is mixed on a question, it would help.
Thanks as always for your service…
FB, maybe next to MS, is the best example of dubiously bad programming turned into a monopoly and forced into the daily lives of nearly everybody. Big fail accept to those devils walking away with the billions.
Sad..Facebook always looked like a fluke.they did not deserved the money or popularity they got . Recent gotcha’s and scandals all point to zero management and all greed.
Thanks for the investigation. Is it by any chance related to the password reset procedure?
https://security.stackexchange.com/questions/53481/does-facebook-store-plain-text-passwords#comment84577_53483
When you initiate a reset, facebook takes your new password, generates variants of it, and checks if any of the hashes of the new variants match with the old hash. Meaning facebook will reject your password if it’s “too similar to the old one”. This has always struck me as odd and i dont know of any other platform that does that…
Pure speculation on my part of course 🙂
This kind of nonsense will continue until management, at the level of the CEO, CFO, CTO are made liable and face real consequences such as significant fines, lawsuits that go after their own money, or jail/prison time. Maybe all three are required. This always make me think of Karl Marx’s dictum of Bold Capital, with special emphasis on the 100% and 300%…
“Capital eschews no profit, or very small profit, just as Nature was formerly said to abhor a vacuum. With adequate profit, capital is very bold. A certain 10 per cent. will ensure its employment anywhere; 20 per cent. certain will produce eagerness; 50 per cent., positive audacity; 100 per cent. will make it ready to trample on all human laws; 300 per cent., and there is not a crime at which it will scruple, nor a risk it will not run, even to the chance of its owner being hanged.”
This kind of nonsense will continue until people stop being willful victims. If everyone simply stopped using facebook for a month the problems would get cleaned up pretty quickly. It’s unfortunate that so many people are addicted to it.
It’s not just Facebook, though. Facebook is the latest example and that only in information technology. There are many other examples where We the People have been sacrificed to the almighty god of profit, and not necessarily in IT. I’m talking about in pharma, aviation, energy, chemical manufacturing, prosthesis manufacturing, the list is enormous. We all know who the names are, and basically to stop using them would mean to stop living in modern society, IMHO.
The most egregious violation of security practices imaginable and it was completely intentional. Password hashing has been required on every production system since Zuckerberg was learning to read. Why were they storing them in plaintext? Developers couldn’t figure out how to call a hash in a form? This would have gotten you fired from a dotcom in the 90s. But don’t worry, they were safe on the LAN. Facebook’s security team should commit group seppuku on Dumbarton bridge.
I am thinking I need to leave Facebook
Another reason that people need to stop sharing what should be treated as personal information. There is no way at all to totally protect data.
Here is something most don’t ever talk about …
On a pure psychological interest into the thought process of general humans, I can and I’m sure many Facebook employees can as well just simply build an algorithm scan that will show the most consistent password characters and formats. This could be the largest password study ever! (of which I’m not advocating for!)
The password algorithm can easily match what people posted to their profile; like birthdates, first and last names, city, zips, favorite colors, etc. then you compare all of that to the persons password. Then you can prioritize the millions of peoples compared password data to see what the worlds most common characters and format’s of passwords are
I promise you the hackers can use that to their advantage to build future hacking software. It would be worth millions of dollars to sell this data to the wrong person.
Anything and everything you’ve ever heard in the past on common password choices is just assumptions based on chatter and sure they have likely been accurate but this would be definitive beyond any previous Measure!
My situation is a female employee of Facebook accessed my account and took screen shots of my private messages and showed them to my boyfriend who is her hisband’s collage buddy.
Totally invested my privacy and everyone whom I messaged.
Facebook has been storing passwords in plain text since 2004, if not earlier. And not just passwords that log you in, but passwords that you type incorrectly.
There was a report that said Zuck actually hacked into a Crimson reporter’s email address using the incorrect passwords logged into FB. He perused through the passwords stored in plaintext available to him!
And by the latest GDPR, that is not illegal, and outside of regulation to disclose if that data is somehow stolen. Why? Because the data may or may not belong to the user, and therefore outside of the agreement the user has with FB.
“nor have we found signs of misuse of this data”. I beg your pardon? That is not how it goes! What I want is evidence of no misuse, not a lack of evidence of misuse.
Oh, and assume the hackers are smarter than you, and knows how to hide their tracks.
My situation is a female employee of Facebook accessed my account and took screen shots of my private messages and showed them to my boyfriend who is her hisband’s collage buddy.
Totally invasion of my privacy and everyone whom I messaged. I know who this woman is and know she knows she broke the law.
I was posting on my own page when in the middle of setting it up I WAS MYSTERIOUSLY LOGGED OUT … given what was in the news this morning I thought it prudent to change my password immediately … I returned to my post on my page AND WAS IMMEDIATELY BLOCKED BY FACEBOOK FOR A WEEK …. WOW !!!!!!!!!!!! They claim it was “SUSPICIOUS ACTIVITY” ….. DOUBLE WOW !!!!!
What good is LifeLog, er um, Facebook if you can’t access all of it,,,,