By now, it’s difficult to summarize all of Facebook’s privacy, misuse, and security missteps in one neat description. It just got even harder: On Thursday, following a report by Krebs on Security, Facebook acknowledged a bug in its password management systems that caused hundreds of millions of user passwords for Facebook, Facebook Lite, and Instagram to be stored as plaintext in an internal platform. This means that thousands of Facebook employees could have searched for and found them. Krebs reports that the passwords stretched back to those created in 2012.
Organizations can store account passwords securely by scrambling them with a cryptographic process known as hashing before saving them to their servers. This way, even if someone compromises those passwords, they won’t be able to read them, and a computer would find it difficult—even functionally impossible—to unscramble them. As a prominent company with billions of users, Facebook knows that it would be a jackpot for hackers, and invests heavily to avoid the liability and embarrassment of security mishaps. Unfortunately, though, one open window negates all the padlocks, bolts, and booby traps money can buy.
“As part of a routine security review in January, we found that some user passwords were being stored in a readable format within our internal data storage systems,” Pedro Canahuati, Facebook’s vice president of engineering, security, and privacy wrote in a statement. “Our login systems are designed to mask passwords using techniques that make them unreadable. To be clear, these passwords were never visible to anyone outside of Facebook and we have found no evidence to date that anyone internally abused or improperly accessed them.”
Canahuati says that Facebook has now corrected the password logging bug, and that the company will notify hundreds of millions of Facebook Lite users, tens of millions of Facebook users, and tens of thousands of Instagram users that their passwords may have been exposed. Facebook does not plan to reset those users’ passwords.
For such a prominent target, Facebook has had relatively few technical security failures, and in this case appears not to have been compromised. But the company’s track record was severely marred by a breach in September, in which attackers stole extensive data from 30 million users by compromising their account access tokens—authentication markers generated when a user logs in.
That breach indirectly helped Facebook discover the trove of plaintext passwords and the bugs that caused them to be there; the incident motivated a security review that caught the lapse. “In the course of our review, we have been looking at the ways we store certain other categories of information—like access tokens—and have fixed problems as we’ve discovered them,” according to Canahuati.
“It’s good that they’re being proactive,” says Lukasz Olejnik, an independent cybersecurity adviser and research associate at the Center for Technology and Global Affairs at Oxford University. “But this is a big deal. It seems like they found the issue during an audit, so maybe their past mistakes plus new privacy regulations are making these checks more standard.”
Facebook told WIRED that the exposed passwords weren’t all stored in one place, and that the issue didn’t result from a single bug in the platform’s password management system. Instead, the company had unintentionally and incidentally captured plaintext passwords across a variety of internal mechanisms and storage systems, like crash logs. Facebook says that the scattered nature of the problem made it more complicated both to understand and to fix, which the company says explains the nearly two months it took to complete the investigation and disclose the findings.
A company operating at Facebook’s enormous scale needs to keep network traffic logs to better understand and trace bugs, outages, and other incidents that may crop up. Those logs will inevitably pull in whatever network data happens to be flowing by. That Facebook caught passwords in that process makes sense; the question is why Facebook retained logs that included sensitive data for so long, and why the company was apparently unaware of its contents.
“The data that’s captured incidentally as part of debugging, and operating at the network scales they do is not uncommon,” says Kenn White, a security engineer and director of the Open Crypto Audit Project. “But if Facebook retains that for years it raises a lot of questions about their architecture. They have an obligation to protect these debug logs and audit and understand what they’re retaining. In some ways that’s the most sensitive data they hold, because it’s raw and unmanaged.”
Twitter dealt with a very similar plaintext password-logging bug last May; it too didn’t require users to reset their passwords, saying it had no reason to believe that the passwords were actually breached. Similarly, Facebook says its investigation hasn’t revealed any signs that anyone intentionally accessed its hundreds of millions of errant passwords to steal them. But whether you get a password notification from Facebook or not, you might as well go ahead and change it as a precaution.
To do so on Facebook desktop, go to Settings → Security and Login → Change Password. On Facebook for iOS and Android, go to Settings & Privacy → Settings → Security and Login → Change Password. On Facebook Lite for Android, go to Settings → Security and Login → Change Password. Changing your account password on either main Facebook or Facebook Lite changes it for both.
On Instagram, go to Settings → Privacy and Security → Password to change your password. Instagram and Facebook do not use the same password, but can be linked to log into one with the other.
While you’re at it, the easiest way to keep track of and manage your passwords so you can easily change them after incidents like this is to set up a password manager. Go get one now.
Facebook says that the plaintext password issue is now fixed, and that it doesn’t think there will be long-term impacts from the incident, because the passwords were never actually stolen. But given the company’s apparently endless stream of gaffes, it’s difficult to know what will come next.
“I get that they are working at mind-boggling scale,” White says. “But these are the crown jewels right there.”