OneLogin, an online service that lets users manage logins to sites and apps from a single platform, says it has suffered a security breach in which customer data was compromised, including the ability to decrypt encrypted data.
Headquartered in San Francisco, OneLogin provides single sign-on and identity management for cloud-base applications. OneLogin counts among its customers some 2,000 companies in 44 countries, over 300 app vendors and more than 70 software-as-a-service providers.
A breach that allowed intruders to decrypt customer data could be extremely damaging for affected customers. After OneLogin customers sign into their account, the service takes care of remembering and supplying the customer’s usernames and passwords for all of their other applications.
In a brief blog post Wednesday, OneLogin chief information security officer Alvaro Hoyos wrote that the company detected unauthorized access to OneLogin data.
“Today we detected unauthorized access to OneLogin data in our US data region. We have since blocked this unauthorized access, reported the matter to law enforcement, and are working with an independent security firm to determine how the unauthorized access happened and verify the extent of the impact of this incident. We want our customers to know that the trust they have placed in us is paramount.”
“While our investigation is still ongoing, we have already reached out to impacted customers with specific recommended remediation steps and are actively working to determine how best to prevent such an incident from occurring in the future and will update our customers as these improvements are implemented.”
OneLogin’s blog post includes no other details, aside from a reference to the company’s compliance page. The company has not yet responded to request for comment. However, Motherboard has obtained a copy of a message OneLogin reportedly sent to its customers about the incident, and that missive contains a critical piece of information:
“Customer data was compromised, including the ability to decrypt encrypted data,” reads the message OneLogin sent to customers.
According to Motherboard, the message also directed customers to a list of required steps to minimize any damage from the breach, such as generating new API keys and OAuth tokens (OAuth being a system for logging into accounts), creating new security certificates as well as credentials; recycling any secrets stored in OneLogin’s Secure Notes feature; and having end-users update their passwords.
Gartner Inc. financial fraud analyst Avivah Litan said she has long discouraged companies from using cloud-based single sign-on services, arguing that they are the digital equivalent to an organization putting all of its eggs in one basket.
“It’s just such a massive single point of failure,” Litan said. “And this breach shows that other [cloud-based single sign-on] services are vulnerable, too. This is a big deal and it’s disruptive for victim customers, because they have to now change the inner guts of their authentication systems and there’s a lot of employee inconvenience while that’s going on.”
KrebsOnSecurity will likely update this story throughout the day as more details become available.
Update 7:54 p.m ET: OneLogin posted an update to its blog with more details about the breach:
“Our review has shown that a threat actor obtained access to a set of AWS keys and used them to access the AWS API from an intermediate host with another, smaller service provider in the US. Evidence shows the attack started on May 31, 2017 around 2 am PST. Through the AWS API, the actor created several instances in our infrastructure to do reconnaissance. OneLogin staff was alerted of unusual database activity around 9 am PST and within minutes shut down the affected instance as well as the AWS keys that were used to create it.”
Customer impact
“The threat actor was able to access database tables that contain information about users, apps, and various types of keys. While we encrypt certain sensitive data at rest, at this time we cannot rule out the possibility that the threat actor also obtained the ability to decrypt data. We are thus erring on the side of caution and recommending actions our customers should take, which we have already communicated to our customers.”
Does OneLogin use two factor authentication ?
Just the facts , Krebs ! :=)
They claim to use machine learning to decide when to ask for 2FA: https://www.onelogin.com/product/adaptive-authentication
I can’t tell from one of the endorsements whether they allow site mandated 2FA…
@ timeless – too bad they didn’t use “machine learning” to prevent the breach in the first place! Hopefully this will be at least one lesson for the “machine”. 😉
My thoughts exactly
“Computers are useless — they can only give you answers.” — Pablo Picasso
Hyperbole, but thought-provoking hyperbole!
Does it matter? Do you have any reason to believe this hack could have been prevented by using 2FA?
Yes, a customer administrator on OneLogin can mandate 2FA for her users of the service.
Ollie, AWS keys don’t support MFA. They simply shouldn’t have active keys for long and should enforce that somehow
Not true. You can protect a AWS key with a second factor token: https://aws.amazon.com/iam/details/mfa/
My workplace enforces that all at users must use MFA; I use this feature every day.
Good to know
IAM Users =/= Keys. MFA works for the Console (the browser), however not the client (Python aws-cli). Depending on the level of access the key has, it can basically be the “keys to the castle” if compromised. The key is that of an API key so send it in your GET/POST/PUT/DELETE request and be done with it, even with MFA turned on for users.
There should be no reasons for keys unless you have services depending on AWS functionality (storing to S3, manipulating EC2 instances, etc…) and even then, they should be a strict as possible in all policies.
You can use MFA for apis.
It’s not clear to me that 2FA would have helped.
2FA might have been in place for user access into OneLogin, but (presumably) the breach gave the attacker access to unencrypted credentials that can be used to log in to third-party sites.
Now, 2FA on *those* websites would certainly help…
Yes, you can turn on MFA in ONeLogin and use SMS, physical security keys, and their mobile app as the second factor.
If the data contained by the password manager is unencrypted when stolen, 2FA access to the password manager doesn’t much matter.
This has the potential to be an utter catastrophe and bodes very poorly to the future of online communications and data storage IMO
When you authenticate yourself and follow that link, you will get this msg.
On Wednesday, May 31, 2017, we detected that there was unauthorized access to OneLogin data in our US data region. All customers served by our US data center are affected; customer data was compromised, including the ability to possibly decrypt some encrypted data. We have since blocked this unauthorized access, reported the matter to law enforcement, and are working with an independent security firm to assess how the unauthorized access happened and to verify the extent of the impact. We want our customers to know that the trust they have placed in us is paramount, and we have therefore created a set of required actions:
If you replicate your directory password to provisioned applications (using the SSO Password feature) or if your users authentication method is OneLogin as a directory, force a OneLogin directory password reset for your users.
You don’t need to reset directory passwords if you don’t use the SSO Password feature or if your users authenticate using Active Directory!
See Password Management.
Generate new certificates for your apps that use SAML SSO.
For information about generating new certificates, see Creating and Applying Certificates.
For information about providing the new certificate to the SAML app, see the app-specific documentation in the App Integration section.
Generate new API credentials and OAuth tokens.
For legacy API keys, see developers.onelogin.com/api-docs/v1-v3/getting-started/using-the-onelogin-api
For current API keys, see developers.onelogin.com/api-docs/1/getting-started/working-with-api-credentials
For OAuth tokens, see developers.onelogin.com/api-docs/1/oauth20-tokens/refresh-tokens
Generate and apply new directory tokens for Active Directory Connectors and LDAP Directory Connectors.
For Active Directory Connectors:
Create a new failover Active Directory Connector instance, following steps 1- 5a in “Adding additional ADC instances for load balancing and failover.”
Copy the installation token for the new failover over the existing primary Active Directory Connector token on the server where the Active Directory Connector instance runs, replacing the contents of the Windows Registry key at
HKEY_Local_Machine\SOFTWARE\Wow6432Node\OneLogin, Inc.\Active Directory Connector\DirectoryToken
Restart the Active Directory Connector.
Switch the new failover Active Directory Connector instance to be the primary (sync) connector, following the instructions in “Failing over a synchronization Active Directory Connector instance manually.”
Delete the old Active Directory Connector instance from OneLogin by following the instructions in “Deleting or disabling your Active Directory Connector instance.”
For LDAP Directory Connectors:
Create a new failover LDAP Directory Connector instance, following steps 1 – 6 in Installing Multiple LDAP Directory Connectors for High Availability.
Copy the token from the new instance to the config file for your existing active LDAP Directory Connector by editing the file conf/ldc.conf and updating the configuration property ldc.api.token. (See steps 9 and 10 in “Installing an LDAP Directory Connector”)
Restart the LDAP Directory Connector.
Switch the new failover LDAP Directory Connector instance to be the active connector, following the instructions in “Switching a standby connector to active.”
Remove the old LDAP Directory Connector instance from OneLogin by clicking Delete on the Basic tab of the LDAP Directory Connector configuration page (Go to Users > Directories, select the directory, go to the Basic tab, and select the instance to delete).
Update the API or OAuth credentials you use to authenticate to third-party directories like G Suite (Google), Workday, Namely, and UltiPro.
A link from a KrebOnSecurity mailing Apr 2017. The link still works, just FYI.
https://krebsonsecurity.com/onelogin/
“Whoa! Forrester study on OneLogin shows 482% ROI. In this impact report from
Forrester, OneLogin’s cloud-based IAM solution changes the game with a
staggering 482% ROI. Get the free report and see why thousands of great
companies are using OneLogin to increase security and decrease IT cost.
Download the Forrester study now to get the details to evaluate the potential
impact on your organization.
Once you click on that link, you will see this:
All customers served by our US data center are affected; customer data was compromised, including the ability to possibly decrypt some encrypted data.
You will also have instructions to:
Generate new certificates for your apps that use SAML SSO.
Generate new API credentials and OAuth tokens
Generate and apply new directory tokens for Active Directory Connectors and LDAP Directory Connectors.
In other words, you have to reset pretty much all of your auth infrastructure.
David
This sort of issue highlights why it’s important for cloud services to make moves in permitting customers to manage their own keys. Certainly, it’s nice having a service that decrypts all your encrypted data for you, but really it’s still very cost-effective to let them store it encrypted and deliver it encrypted either through a server where you control your own keys and can decrypt the data before the customer/user sees it, or by only loading the key ephemerally to decrypt data in sessions. The latter would still have a few obvious weaknesses, but it would be better than someone running database and NAS solutions on flat network with keys sitting nextdoor to the relevant data (if it was even that separated).
For the services that allow provided keys; Many of them keep a copy of your key for their ability to decrypt your data for you. So that’s not really a solution for this problem.
+1. Unfortunately, implementing your suggestion requires consumer smarts. OneLogin tries to provide an experience that minimizes consumers’ required security knowledge. Now we see the consequences.
I agree, though my intent was to underline a broader problem in cloud services being the implicit key manager.
You can be good at supplying a service without needing access to the data. Hell, as part of your service you could setup or assist with the on-premise intermediary for those without the skill tree.
Cloud has scared my from day #1. I guess I was right to be scared – we can NOT all be security experts – the field, combined with crypto, network, and hardware infrastructure knowledge, is just to large.
I’m looking forward to more details about this breach, and a comparison of the two “successful but unsuccessful” breaches in the past of OneLogin’s competitor LastPass.
One two occasions in the past LastPass’ database of customers’ encrypted password vaults was breached, but in neither case was data at-risk (unless the customer had chosen a weak master password).
I want to know what’s different about this OneLogin breach.
It might be difficult to compare, as this product has Active Directory support. But for us individuals and/or SMBs, LastPass does it differently. They have no access to the encrypted blobs in their cloud servers. So I assume that is why their breaches were not considered as serious. On the side of caution, they suggested customers change their master password, but I wonder how an intruder in their cloud server could get those, as they are also encrypted. I think the only reason LastPass suggested changing the master password was because some people have insecure passwords that would be easy to brute force if the encrypted file were copied somehow from the cloud source. So I figure if your master password is in a secure configuration, it would take them forever to crack it.
LastPass is so good at reporting account activity to customer sites and attempts to access the main account, I have little doubt it would be rather difficult to gain control of a LastPass account where the user is paying attention to these alerts. Perhaps someone with more knowledge of how the process works in that cloud environment can weigh in here.
Makes you wonder just how safe other password managers are including the top rated LastPass.
If you weren’t already wondering about how secure your online password manager is you’re doing it wrong.
LastPass does not store the necessary decryption key on their server. Ina breach of LastPass, the criminal would only get an encrypted blob…useless without your master password.
Are you sure? As of a month ago this was not the case. LastPass did store the encryption key on their server and they have been breached before and lost customer data. This was the reason I chose not to use LastPass as our end user solution for password management.
What’s the source for the claim on Lastpass? As far as I’m aware, they never have your Master Password and it never leaves your device. They do store hashes of your master password (which are required to serve up your encrypted vault), but that’s not the same thing.
I use Lastpass in combination with 2FA, first and foremost to protect my password vault, but I also use 2FA for all services I can. For instance Google or Dropbox. Lastpass fills out username/password and I press the button on my YubiKey.
This means that even if someone was able to hack my Lastpass vault, they can still not get in to my account.
Of course nothing is 100% secure but the option to our “normal” world is to dig a hole in the woods and sit there looking at the stars (without any electronics). Plausible but maybe not so fun…
don’t use a pw manager….
They are the equivalent of FormFire… when that eventually gets compromised, millions of health records and PII is going to be taken,. It’s just a matter of time.
The IT security world is divided in two: those who have been hacked and those are yet to be hacked. The second category is getting smaller.
When the best pros in the IT security world (OneLogin, LastPass, RSA, etc) get hacked, what hope is there for the rest of us?
This is a sad day.
I found this old comment from 2011, after the RSA hack:
” Never place such fundamental trust in an entity which prioritizes its own profit above your security. Most cloud services will be a disaster for exactly this reason. ”
Ouch!
What *really* bothers me is decrypting encrypted information. I guess it depends on how things are set up, but I’ve been told previously that various services can’t decrypt my passwords (I have to generate a new one) – has the point of view of these services changed? I guess it’s time to do some research on how these things work…
(thanks for the awesome info, though!)
What no free credit monitoring offered by OneLogin? I thought that was all businesses go to response.
No need to anymore since everyone already is double covered.
From this FAQ it seems OneLogin keeps the keys to decrypt on their systems.
https://support.onelogin.com/hc/en-us/articles/204632494-Password-and-Security-FAQ
Others like LastPass and Dashlane don’t have the keys so they wouldn’t be compromised in the same fashion presuming this is how OneLogin was compromised.
Cloud storage isn’t all bad, but you’ve got to know what kind of security model you’re going with.
True! (+1)
You sure about LastPass? They have had breaches before and because encryption was being done on their cloud, customer passwords were compromised. I met with them just a month ago and for this reason I chose to pursue Dashlane as our possible password management solution.
Both Dashlane and LastPass allow you to install this software on multiple devices, which then have access to your stored passwords. How is the encryption key being passed to the newly added device?
@MV: “How is the encryption key being passed to the newly added device?”
You need to enter your master password on the new device. I haven’t audited the code personally but the claim is the master password is not stored in Lastpass’s cloud, and nothing stored is useful without that password. All decryption is done on the local devices that access the encrypted blob.
Steve Gibsons “Security Now” Podcast did a (now dated) thorough tear down of Lastpass. It is convinced me that Lastpass is the endall-beall single sign on solution, by far.
Lastpass encrypts passwords at the end user. The key(s) never leave the end user. While the encrypted blob of passwords is stored on Lastpass’ cloud, Lastpass (the corporate entity) cannot decrypt your blob that is stored on their servers because they have not stored the secret keys. Only you can decrypt the blob at your end.
There are some great advanced security features that should help strengthen your cloud stored data; Limiting login access to a Geographic region, Disallowing access through TOR, and setting custom Password hashing iterations during encryption (Mine is 32,768 while the default is “only” 5,000) before being sent and stored on Lastpass’ servers.
It really is a very robust tool.
…at least until something like a RAT gets distributed by their servers and harvests local passwords… But that’s a different attack, although it has happened before, just not with these guyz.
“You sure about LastPass? They have had breaches before and because encryption was being done on their cloud, customer passwords were compromised.”
I don’t believe that’s true unless you have access to information that the public does not. As far as I understand, the encryption happens locally on your browser, and only then gets sent to LastPass
The only “security model” that you need for Cloud Storage is, “Never store anything in the Cloud that you wouldn’t mind having plastered all over the front pages of the NYT and your weekly hometown newspaper.
Their website lists a who’s who of large companies, “Over 2,000 enterprise customers globally secure their applications with OneLogin”. This is going to be a lot of work for enterprise IT to clean up and secure.
I work in the IT department of one of those companies. It’s going to be a complete disaster for at least four days to a week.
When my company was shut down “back in the day” by a virus, I figure it wasted $millions, although in those days, the bad guys didn’t get rich.
From the description it appears the actor is little better than a script kiddie – “Get AWS keys to impersonate as a legit actor”
Start reconnaissance – your last resort is initiate forensic analysis from logs (cloudtrail?) They may have had security products installed in their instance as a policy which perform continuous monitoring, baselining workloads and quickly conclude breach detection in real time of their data center.
Alas the description of ~7 hrs of continued malicious activity followed by manual intervention seems like all red lights on the dashboard.
Wonder if the breach findings will be published… in the benefit of cloud computing community. Time will tell…
I also work for one of those companies – it isn’t all that bad. We don’t store our passwords there, though, and only use SSO (not provisioning). Re-issuing the certs is non-impacting.
Let me guess: with the AWS keys they were able to create images from root volume snapshots (never backup your root volume kids), and those root volumes contained all keys for further exploration.
Or perhaps even simpler: just creating EC2 instances in their VPC was sufficient to be trusted, because only trusted actors can do that…
Encrypting data at rest could have worked if the decryption keys were stored offline, but given that the actor had AWS keys, he probably had the decryption keys as well.
Just found this, OneLogin was hacked last year too: https://www.theregister.co.uk/2016/08/31/onelogin_breached_hacker_finds_cleartext_credential_notepads/
It seems OneLogin only recently migrate to AWS, and was helped by threatstack: https://blog.threatstack.com/onelogin-gains-granular-security-control-with-threat-stack-on-aws
Quote: “Threat Stack has literally changed the way OneLogin sees their cloud environment. The Security and Ops teams can automatically audit their AWS configurations to identify new risks as their environment changes. OneLogin can more confidently and easily manage their environment with more granular control over tuning and adjusting security alerts within Threat Stack’s intrusion detection system (IDS). ”
Wow.
Based on this article regarding security company Threatstack in helping them in migratin and protecting them on AWS. Threatstack May the the small vendor involved in them getting hacked. It’s apparent that Threatstack can’t protect anyone from being hacked.
I always had a weird feeling about these uni-password type sites. Sure they are convenient, but it takes only one breach to compromise everything you have under them. Nothing seems safe these days!
Kristen, if you have the time, check out this thorough analysis of Lastpass circa 2010. yes, it’s 7 years old, but what you will hear is that Lastpass has been doing password management right for years.
(download the MP3 audio file and ignore Leo Leporte)
https://www.grc.com/sn/sn-256.htm
Quite simply, this is why I don’t use a password manager. I don’t care which one it is.
OK, so how do you(and others here) “manage” your passwords?
My way: Depending on how often they’re used, many can be memorized (length is more important than complexity). Those that can’t be retained in your mind can be stored in an strong-encrypted container with one really difficult master passphrase for the occasional need. Always kept offline. And always backed up redundantly (in various locations), by hand. (This works for all kinds of vital information which you can’t afford to loose.)
Like this thing I am currently testing: http://www.admin-admin.com The “promise” (well, they do not promise nothing) is very simple. Keep the master repository of passwords and other relevant data offline AND encripted, but in my computer. I like it this way, because as a designer I know all the passwords for the sites I work in.
Thank you… been looking for a desktop app like that for a while…!
I write them down in a paper notepad. Primitive but as foolproof as it gets.
Very good article. Very good arguments both ways. So, did they plug the breach, or just leave the hole and the little dutch boy. It could have been an excellent honeypot to take out whoever. No mention of that though. So, was this another school exercise from China, or the kids from the Ukraine, or Harvard? None of them seem to have an excuse handy, or a pass from the teacher to hand in, but it sounds off as such an attack. Why, easy to block when noticed. No back worm, no spread, no offsite copy, just a halt. Maybe a reattch request?
My only question, what happens when the power fails,and you try to login. I get an encrypted screen, or no screen to try to religion and from where? The server, the handheld? Or from the badguy?
I think many in these threads are failing to see that the information available suggests an operational concern.
It’s likely that the AWS credentials used by the “smaller service provider” had excessive privilege. It’s possible that this provider had broad operational responsibilities, but I think it’s more likely that they provided a service like monitoring or something more narrow. If this is the case, then poor processes and procedures led to excessive privilege.
It’s also likely that the keys were not managed according to best practices (see https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html#use-roles). Managing these properly causes overhead and might not be well supported by third parties if they can’t assume the key they have stored is still useful.
Security is hard. 🙂
2 security breaches in one year at a “security” company means they will go out of business in a few months.
I don’t understand this. I would like to think that my data on my 1password account is encrypted and that since it is on my icloud (rather than a central database) account, gaining access to 1passwords server should do nothing.
I can understand how SMS is fairly easy to hack these days by transferring the phone number to a different phone under the hackers control ( http://bit.ly/2rl5Tg9 ) and even my totp authentication is only a 1 bit cryptographic code, but still…
Geezz-o-pete. For anyone with any bit of basic network and application security, this is so much wasted keystrokes, electrons and neurons. When the marketeers first uttered “you gotta get some of this cloud…” so many years ago (what is it now, 20 years?), public cloud usage has been Russian Roulette with all eight chambers loaded (in the case of my S&W R8) . Been doing this stuff back to vacuum tubes and I get it – economy of scale, elasticity, ubiquitous access, etc. etc. Map your data flows, perform key management studies, decide how much bad press, financial loss and prison time you can stomach and then put your crown jewels in an environment that you do not manage or control. And I am not saying that your self-contained data center is safe, just that convenience verses security is a no-brainer (at least to me…)
^ This!
The same experience here for me since the 80’s. Death by convenience. You could smell it on the new MBAs and the promise of fixed expenses and limited liability.
^This!
You say “cloud computing”, I say “time-sharing”.
It’s been a data security nightmare since it was born. Wikipedia says “The first international conference on computer security in London in 1971 was primarily driven by the time-sharing industry and its customers.”
Details are lacking for the incident breach. Inside source breach could have occurred. False positives have been used in attacks to monitor new logins when breach has not occurred.
Quick report link would be pertinent to deescalate any incident from user or source. Notification to breach are also an issue for verification.
Neutral login would be beneficial and secondary used from source.
One use code is the best application. This is very time consuming and something to work towards.
Pardon my ignorance, but what does this mean? “Neutral login would be beneficial and secondary used from source.” What’s “Neutral” login? And “secondary”? I’m not an IT person, just a wannabe!
I am of the age that committing anything to memory is a high risk and I don’t like managing passwords on paper. Erasures, cross outs, and rewrites hurt my eyes.
I use a password manager. Although they all have flaws and are a single point of failure, I am still convinced that random passwords consistently managed are more secure than anything fallible me will keep up in the long term. Which password manager? I have worked with competitive software long enough to come to the opinion that unless a vendor is obviously sloppy, the risks even out over time. A product that is better one day is worse the next. The best on any given day is a crap shoot.
Second, I keep a residuum of passwords and other keys in a text file on a usb drive that I keep on my physical key ring. When I need it, I plug it in. I have never lost my car keys and my passwords are on the same ring. I can’t leave my desk with the usb plugged in. I occasionally copy the file to another usb drive that I keep in a safe with other important stuff.
This provides a balance of convenience and safety that I can live with with. Breaches like OneLogin must be minimized, but I don’t expect them to be eliminated.