Payroll software provider Apex Human Capital Management suffered a ransomware attack this week that severed payroll management services for hundreds of the company’s customers for nearly three days. Faced with the threat of an extended outage, Apex chose to pay the ransom demand and begin the process of restoring service to customers.
Roswell, Ga. based Apex HCM is a cloud-based payroll software company that serves some 350 payroll service bureaus that in turn provide payroll services to small and mid-sized businesses. At 4 a.m. on Tuesday, Feb. 19, Apex was alerted that its systems had been infected with a destructive strain of ransomware that encrypts computer files and demands payment for a digital key needed to unscramble the data.
The company quickly took all of its systems offline, and began notifying customers that it was trying to remediate a security threat. Over a series of bi-hourly updates, Apex kept estimating that it expected to restore service in a few hours, only to have to walk back those estimates almost every other time a new customer update went out.
Contacted Wednesday by an Apex client who was nervous about being unable to make this week’s payroll for his clients, KrebsOnSecurity reached out to Apex for comment. Ian Oxman, the company’s chief marketing officer, said the ransomware never touched customer data, but instead encrypted and disrupted everything in the company’s computer systems and at its off-site disaster recovery systems.
“We had just recently completed a pretty state-of-the-art disaster recovery plan off-site out and out of state that was mirroring our live system,” Oxman said. “But when the ransomware bomb went off, not only did it go through and infect our own network, it was then immediately picked up in our disaster recovery site, which made switching over to that site unusable.”
Oxman said Apex hired two outside security firms, and by Feb. 20 the consensus among all three was that paying the ransom was the fastest way to get back online. The company declined to specify how much was paid or what strain of ransomware was responsible for the attack.
“We paid the ransom, and it sucked,” Oxman said. “In respect for our clients who needed to get their businesses up and running that was going to be obviously the quicker path.”
Unfortunately for Apex, paying up didn’t completely solve its problems. For one thing, Oxman said, the decryption key they were given after paying the ransom didn’t work exactly as promised. Instead of restoring all files and folders to their pre-encrypted state, the decryption process broke countless file directories and rendered many executable files inoperable — causing even more delays.
“When they encrypt the data, that happens really fast,” he said. “When they gave us the keys to decrypt it, things didn’t go quite as cleanly.”
One of Apex’s older business units — ACA OnDemand — is still offline, but the company is now offering to move customers on that platform over to newer (and more expensive) software-as-a-service systems, and to train those customers on how to use them.
Experts say attacks like the one against Apex HCM are playing out across the world every day, and have turned into a billion-dollar business for cyber thieves. The biggest group of victims are professional services firms, according to a study by NTT Security.
Ransomware victims perhaps in the toughest spot include those offering cloud data hosting and software-as-service, as these businesses are completely unable to serve their customers while a ransomware infestation is active.
The FBI and multiple security firms have advised victims not to pay any ransom demands, as doing so just encourages the attackers and in any case may not result in actually regaining access to encrypted files.
In practice, however, many cybersecurity consulting firms are quietly urging their customers that paying up is the fastest route back to business-as-usual. It’s not hard to see why: Having customer data ransomed or stolen can spell the end of cloud-based business, but just being down for more than a few days can often be just as devastating. As a result, the temptation to simply pay up may become stronger with each passing day — even if the only thing being ransomed is a bunch of desktops and servers.
On Christmas Eve 2018, cloud data hosting firm Dataresolution.net was hit with the Ryuk strain of ransomware. More than a week later on Jan. 2, 2019, this blog reported that the company — which had chosen not to pay the ransom and instead restore everything from backups — was still struggling to bring its systems back online.
One dataresolution.net client said the company didn’t succeed in rebuilding its server or turning over his company’s database stored there until Jan. 9 — 16 days after the ransomware outbreak.
“From my understanding it was another two weeks until all of the clients were rebuilt,” said the customer, who works as an IT manager at a benefits management firm that used dataresolution.net and its now transitioning away from the company. “The vendor never provided any analysis on how it occurred and how they would prevent it from occurring again. Other than different antivirus and not allowing RDP connections to the internet they don’t seem to have put any additional safeguards in place. They did not proactively offer any compensation for the outage. I am in the process of documenting the business financial impact to request a ‘credit’ at the same time as planning on bringing the system in house.”
For its part, Apex is still trying to determine how the ransomware got into its systems.
“That’s where this forensic analysis is still going on,” Oxman said. “For us, the emergency response team literally worked 48 hours straight getting our systems back up, and secondary to that is now trying to figure out what the hell happened and how do we prevent this from happening again. We had just completed a security audit and we were feeling pretty good. Obviously, these cyber hackers found a way in, but I’m sure that’s how every company feels that gets hit.”
Here are a few tips for preventing and dealing with ransomware attacks:
-Patch, early and often: Many ransomware attacks leverage known security flaws in servers and desktops.
-Disable RDP: Short for Remote Desktop Protocol, this feature of Windows allows a system to be remotely administered over the Internet. A ridiculous number of businesses — particularly healthcare providers — get hit with ransomware because they leave RDP open to the Internet and secured with easy-to-guess passwords. And there are a number of criminal services that sell access to brute-forced RDP installations.
-Filter all email: Invest in security systems that can block executable files at the email gateway.
-Isolate mission-critical systems and data: This can be harder than it sounds. It may be worth hiring a competent security firm to make sure this is done right.
-Backup key files and databases: Bear in mind that ransomware can encrypt any network or cloud-based files or folders that are mapped and have been assigned a drive letter. Backing up to a secondary system that is not assigned a drive letter or is disconnected when it’s not backing up data is key. The old “3-2-1” backup rule comes into play here: Wherever possible, keep three backups of your data, on two different storage types, with at least one backup offsite.
-Disable macros in Microsoft Office: Block external content in Office files. Educate users that ransomware very often succeeds only when a user opens Office file attachment sent via email and manually enables Macros.
-Enable controlled folder access: Create rules to disallow the running of executable files in Windows from local user profile folders (App Data, Local App Data, ProgramData, Temp, etc.)
Sites like nomoreransom.org distribute free tools and tutorials that can help some ransomware victims recover files without paying a ransom demand, but those tools often only work with specific versions of a particular ransomware strain.
When will those silly companies learn that it is worth hiring a security professional AHEAD of time to review their system and software rather than paying North Koreans, Russians or Chinese for those services later (by paying the ransom.)
Also anyone who pays that ransom, I hope they realize that they have a very slim chance to get their working system back, if any at all. That Sergei guy that wrote the encryption portion of the ransomware software wasn’t really interested in getting the decryption part right. So please don’t play the ransom. You’re just losing money and encourage the thieves! Learn your lesions ahead of time. And don’t just pile all your money into marketing dept.
From the post: “We had just completed a security audit and we were feeling pretty good.“ Sometimes, it’s just not enough.
pretty poor audit i’d say…
Audits are inherently infected with a perverse incentive conundrum.
As an IT manager the mandate is to pass the audit. So you hope they auditor doesn’t find the stuff under the carpet.
The auditor wants a payday from you next year too. So if he finds all the bad stuff under the carpet and holds your feet to the fire… well buh bye next years pay check.
Here is the site that customers of FirstData use to download their entire Credit Card portfolio every day. TLS 1.0 enabled? Check. Weak ciphers only? Check. No 2FA? Check. PCI Compliant? Oh yea!
https://www.ssllabs.com/ssltest/analyze.html?d=www.mft1.firstdataclients.com&latest
The IT manager shouldn’t be the one to hire or fire the external auditors that should fall to the audit committee or information security officer.
Putting yourself out there as looking for security products or security staff can actually make you a target. I know a company that didnt focus at all on security for over 25 years, nothing. Not one breach and only a handful of security incidents. Then they decided to fosuc on improving security, audits, new technologies, security staff etc, and boom they get breached.
Im not saying that companies shouldnt invest in and focus on security, but sometimes doing so puts you at risk. I personally think someone should betaking a close look at those who work for IR companies and other security vendors as I have a suspicion many of them have blackhats working for them.
**
“I personally think someone should be taking a close look at those who work for IR companies and other security vendors as I have a suspicion many of them have blackhats working for them”
**
I’m not a conspiracy theorist, but I wouldn’t be surprised in the slightest if IR companies didn’t do that already. “Hey, here’s our service, you need it to be secure. Oh… you don’t want it? It would be a real shame if you… suffered from a breach, wouldn’t it?” There’s a reason why a lot of cases of breaches go unsolved. Those IR companies, though they sign NDA’s and other documentation, will eventually learn the ins and outs of the company’s network. They will know exactly just how to get in, get out, and go undetected – heck, they’re the ones usually controlling the monitoring too. I just noticed my initial sentence almost seems inconsistent with the rest of this paragraph… guess I really am a conspiracy theorist then!
Don’t you think it is much more likely that they had the tools to finally see a breach? Just because they didnt notice a breach when they didnt have good security doesnt mean that there was not a breach.
The article doesn’t say how much the ransom was or how the the destructive strain of ransomware breached onto the computer/server .
Bummer !
“We had just recently completed a pretty state-of-the-art disaster recovery plan off-site…when the ransomware bomb went off…it was then immediately picked up in our disaster recovery site…”
Hmmm..is that “state-of-the-art”?? Isn’t that exactly what SHOULDN’T happen?
Whoever devised that “disaster recovery” plan should be fired.
@Red
That’s exactly my thought! It also said:
“But when the ransomware bomb went off, not only did it go through and infect our own network, it was then immediately picked up in our disaster recovery site, which made switching over to that site unusable.”
Wouldn’t that mean that this was all on the same network, i.e., there was no logical segregation in the network? If my thinking is correct, had their network been properly segregated from their DR site, this probably shouldn’t have spread.
Not necessarily. Some people use a tool like Zerto to keep their DR site synchronized with the working servers. If someone doesn’t catch the infection quickly *and* remember to stop the replication, the software will dutifully copy over the changes.
Of the 60ish people who identify as employees of the company on LinkedIn, none of them have titles that would indicate that they do cybersecurity as their full-time role. While it’s understandable that many small firms can’t justify someone in a full-time role, it seems negligent for a company that does HCM and payroll.
https://acaondemand isn’t up but https://www.acaondemand.com/ is.
I’m also wondering how Apex’s backups weren’t able to resurrect the system to at last the prior non-infected day/hour.
And with decent incrementals and DB backups, this should not be any worse than someone tripping over the power cord.
How do mission-critical companies end up looking like a couple of teenagers that think they’ve built the holy grail?
It sounds like they set up a mirroring system which mirrored the infection.
Mirrors are how one is able to do near instant failover.
They also don’t protect against this problem.
To get backups right, the sender must not be able to have full access to the destination (backup) server. It sounds like this wasn’t the case here.
Yeah, I read into that, too. What the guy was describing was an HA system, not a backup system. Or maybe he meant ‘backup’ in the ‘failover’ sense. Nowhere in there did he accurately describe what a real data backup is.
HA is not backup, people.
ransom payed? bitcoins Again?
funny name… oxman: D Why x in the name?
apex is with x too : D
btw
follow the money and you see who is quilty.
When You use btc wallet Hovestly You are going to leave some IP address behind.
IF its Russia then Trump Can call to Putin and Say lets get the bad guys… im sure Every Country Want to Fight against cyber crime… Putin is good person he was police men police Fight against Criminals.
Putin friend is Steven seagal and Steven is police too… im sure if they join together the Steven seagal Will do few neck chops on bad guys.
I would like to become cyber police or Secret service agent good Money and interestisting Job.
I have idea: maybe next Hollywood movie could be that Steven seagal Going to Russia after bad guys who send ransom to usa!
You forgot the most important tip – don’t run Windows.
And oh, step your users through new-school security awareness training.
Basic security training should occur at the same time a new employee is issued his/her credentials and an initial password, and upon first login the system should immediately require a change-of-password to something known only to the user.
And do not stop training the employee’s. Train often, show them examples and drive web & email security in to their skulls until they get as paranoid as we tend to be!!!
I understand that some companies may be reluctant to run their desktops on MacOS and Linux only, particularly if they have hard-to-train employees or asinine requirements.
On another hand, if you insist on running a wide-open Windows desktops, you should at least not keep your files on writeable shares. And you should keep offline backups. Neither of which is hard or expensive. So it’s hard to feel pity for Apex.
The extortionists didn’t even exploit Apex for full profit. Had they known that their latest victim was handling payroll files, they’d have exfiltrated all these customer files then blackmailed every Apex customer. “Pay up, Mr. CEO, or everyone will see that you give your drinking buddies enormous salaries while you skimp on everybody else’s paychecks”. So Apex got off really cheap. This time.
Actually, Francis, it’s quite possible that the crooks DID get all that data — but they may be biding their time before implementing Phase Two of their scheme.
Sadly, many companies think that once they’ve “recovered,” all is well. Maybe, maybe not.
As we’ve seen, many victimized companies find it difficult or even impossible to assess just how much data may have been compromised. Just because they can “recover” the data doesn’t mean that the crooks don’t also have it and will put it to good (read: nefarious) use later.
Because Linux is never vulnerable to ransomware attacks, right?
https://www.bleepingcomputer.com/news/security/b0r0nt0k-ransomware-wants-75-000-ransom-infects-linux-servers/
You can’t trust an operating system – ANY operating system – to do the hard work for you.
I’d bet dollars to donuts that the average Linux desktop user is orders of magnitude less apt to click on some random email attachment when compared to the average Windows user.
It’s not just the base/core technology/vulnerability, and one example for Linux up against the literally thousands in a Windows desktop, doesn’t give us much of a statistical significance.
So you’re also saying it’s the Linux users that make the platform safer?
I agree.
Put the same “click-everything” folks on a Linux box maintained in the same fashion as a Windoze box and you’ll see similar results.
I’ve installed/run/administered/supported Xenix/Unix/Linux for decades. If you don’t know what you’re doing, you can be just as ripe for exploitation as if you were running something from Microsoft.
Once every hundred characters have the backup system write a string into the data. Then write a canary script to run on the backup server that checks incoming data for that particular string. If the string can’t be found in the most recent data written the canary would order the server to shut down immediately. The string would be removed during ordinary data retrieval so it would be invisible to workers or attackers. Yes, some loss of performance.
A good followup story would be about how many users sue Apex for negligence, and what Apex’s insurance company did, or didn’t, require as conditions of coverage.
When we got hit by Ryuk at the end of 2018 all shares that had BUILTIN\Administrators permissions where infected. It seems like some Windows bug that a process can include local group (or something that looks like BUILTIN\Admins) in its security token to be used against network resources. This is (probably) already patched as the infected systems were 6 moths behind on patches.
Nevertheless since this incident we are securing both shares and NTFS permission with domain groups which goes against every security recommendation but this what real-life experience showed to be best. . .
The real lesson here is to hire a accounting staff or a payroll company that can do its work on-site. Do nothing online.
The cloud is a failure for secure and reliable business and government operations. It’s great for fun and educational content, but it’s hopelessly insecure.
No amount of training, preparation, and resources can fix it.
Everything is a security failure, ‘the cloud’ isn’t an exception. Whatever platform you’re on you need to be smart about it. There are plenty of companies with presences in ‘the cloud’ that don’t leave medical records on a public share.
‘The cloud’ is just the base technology, it’s the implementation in this case (and all other cases) that was lacking.
Yes and no. The “cloud” may be neither more nor less secure than a physical server in a data center but the headlong rush to the cloud is creating more opportunities for the criminals.
The cloud is insecure because it relies entirely on the promises of a data center and its employees to be the physical guardians of your data.
Those promises are worthless.
They’re worth even less, when you realize there are no current laws preventing them from obtaining secondary employment with your competitors.
Apex Human Capital:
“Our Story…technology leadership remains at the heart of Apex HCM… Five values… Own It …Know Your Sh**… Do It Right… Push The Limits… Be Successful”-Apex HCM
https://apexhcm.com/about-us/#
Yes! Words to live by…cough… cough.
Bingo!
That is why you need to hire security professions for periodical vulnerability assessments,pen testing and system hardening so it lowers your chances of attacks. Paying the ransom doesn’t help either, but encourage the bad guys.
Is it wise to store client files and data on drop box or other as long was they are encrypted and well backed up?
I wonder if KISS or Keep It Simple has any merit? I remember a rather large British advertising firm some thirty years ago and they had at least three back-ups on storage medium (tape). One was physically offsite, one was in a safe onsite and the other not sure. All of this was Cobol and well before Cloud (mainframe off site). Its almost impossible to stop penetration of a system and depending on the code this can and will attack other parts of the system or cloud storage. Perhaps we need to go back thirty years and KISS and make regular backups that are separate in cannot be violated by code in the cpu? These backups cannot be so called RW medium, they have to be one time write and in reinstalling the system must not have the ability to be affected. The main problem with business is bottom line profits and cutting costs, so sadly these precautions or plans to have a complete backup/failsafe system are not incorporated. Paying ransomware crooks is poor economy and generally data is not recovered as it was. A complete failsafe backup system however would be running very quickly and in the long run far more economical.
mafia never disapere.
90.s we payed to racket/(physical racketering)..now days cyber racketeering.
The 3-2-1 rule is bad advice. We see this time and again. A company follows it only to be hit by malware across its systems, including the offsite backup. It happened in this case and it has happened in many others. Keeping a backup offsite only works if it is also offline. That is what companies like Iron Mountain are for. If your backup is connected to the Internet, or your internal network, it is vulnerable. There have been cases of companies losing multiple offsite backups in multiple locations because they were all connected to the network and were all hit by the same malware simultaneously. The rule should be 4-3-2-1. Four backups, in three formats, two offsite, and one offline. I am not an IT professional, nor a business, but this is the rule I follow for just my personal files. It is not hard to do.
Does anybody here have experience with d7xTech (formerly Foolish IT). I used to test the free version on my windows honeypot, and no malware I ever tested could get past Cryptoprevent; (the name of their product). I haven’t been testing now for over 3 months so I don’t know if the new malware can subvert the present integration of that product. The updates came often enough to lead me to believe they keep up with the newest threats. Seems like it would be MUCH cheaper to pay for their solution than pay a criminal, and also take such libelous losses.
Cryptoprevent places “artificial” group policy objects in the registry and so forth, to block behaviors exhibited by such ransomware. IIRC , it also works with Active Directory to protect the entire system in a Windows based enterprise. You won’t find these settings in the MMC or group policy, but they are assuredly evident, or my tests wouldn’t have worked successfully. I do not work for this outfit, and I’m not advertising for them – I just think we need to discuss actual solutions to these problems. Although I use Malwarebytes at home, I don’t have the confidence in its anti-ransomware capabilities that I did with Cryptoprevent..
It seems to me that a security official at any corporation needs to learn how to do these Group Policy objects on their own, and lock them down to prevent subversion. I would imagine it would take quite a few settings and maybe some snap-ins to the MMC to protect individual machines – I don’t know enough about the Active Directory power to even comment on that.
Thank you Brian, for keeping us up with the latest in these attacks!
Varonis also makes a good anti-ransom ware tool. It will lock out the account executing the encryption of files (you set the threshold and exclusions).
Thanks! That’s interesting! I’ll have to check into that.
Brian,
Would you consider adding a mail/forward feature for articles posted on KrebsonSecurity?
Providing critical information from your site to many readers using this feature will not only spread the word more quickly, but may also increase readership and implementation of preventative security measures you endorse.
Thank you.
Denny Heidenreich
you from switzerland? Heidenreich..or germany?
DR failed at same time, big red flag. I mean ransomware is a VERY common thing, much more common than an earthquake taking out a site. It needs to be taken into account
Trying to find out how it got in is also a bit of red flag. Highly likely someone just clicked it in email.
Speaking as a sr. Linux sysadmin… they had a hot backup, and it got onto *that*? No delay at all? Sorry, explain to me again what the point of a hot backup is, if there isn’t time to find you’ve been blatantly compromised, and rotate to the hot backup?
No. They were running it like a RAID-1, not as an actual backup system.
This is unfortunate and especially since they paid the perpetrators.
Different companies go about things differently but while all would like to be secure they have different limits on what they are willing to pay for that security.
Also there are a lot of “experts” out there that aren’t really hard core experts. They perform basic tests and aren’t anywhere near the level of the hackers in skill.
Some companies also aren’t willing to be “burdened” with properly securing networks and systems.
I have a bunch of certs and I know others with them but if you aren’t doing this kind of work (pen testing, configuring network security, etc.) as a full time job you are going to have issues.
I know personally that while I can detect, find and do a lot of computer security stuff, if I went head to head with a hacker with years of experience, most likely I’d lose because it isn’t my 40+ hr per week task.
Like others, I’d be curious what they actually did try to do to protect their systems and how strongly did they test the security. Many don’t want rigorous testing because they fear down time. Many don’t want to verify backup systems are working because they don’t want to shut things down and do a full restore, etc.
The problem with clouds is that they sometimes rain.
Security problems for shared storage is so old it should be retired. I know how easy it is to make everything shared. everything readable, and everything writable. But that is not a recipe for staying in business.
It is amazing how many people one phish will feed….
Do you know if they were in a commercial CSP of note, mom & pop’s drive thru cloud services, or host-your-own? Not looking for a name but will try to make a point to some co-workers.
“In practice, however, many cybersecurity consulting firms are quietly urging their customers that paying up is the fastest route back to business-as-usual.”
I hope they told their users that in addition to getting them back to business as usual, you have guaranteed that your and companies like you will continue to suffer these attacks. As long as victims continue to monetize ransomware attacks, they aren’t going away.
“A ridiculous number of businesses — particularly healthcare providers — get hit with ransomware because they leave RDP open to the Internet and secured with easy-to-guess passwords.”
Seriously? An eleven year old would tell you that’s a bad idea.
Dream headline: “Website owner offers 1,000,000 bounty for attacker”.
The timing of the ransomware attack was pretty good (or bad). Just as they were mirroring during a DR test they came under attack…
I can understand why people pay the ransom – they know that lots of people are dependent on the receiving the paycheck (in this case) at a precise time and, failure to deliver generally means you lose the contract, get sued, or other bad things so there is big time and money pressure. Unfortunately, as another person stated, ransomware writers rarely are interested in actual restoration of data – and sometimes destruction is their goal – so paying can mean just losing money.
I guess the big thing with Cyber Security is to realistically assess what can realistically go wrong and have plans to deal with those events, and hope those you don’t expect to happen, never happen.
Belt & Suspenders are required for successful B.C. Never rely on a single technology. In-line replication is fine but you need a point-in-time capability too.
please note, the ransom in many cases is much SMALLER than paying for the full security audit and implementation of good protection systems.
Check the prices of some good endpoint protection, and you will be astonished by the numbers.
I would have disagreed with you until I was a Security Analyst and saw some of the prices myself. Some of the tools are insane! Heck, there’s even some GRC tools that are $50k+. I wish I could remember the specific GRC tool off the top of my head, but I was working with some consultants not too long ago and they mentioned they see that GRC tool implemented quite often and it’s $50k+. And you also have to think, some of those tools may include a 1-time fee, but a lot of them tend to be subscription based. I worked for an MSSP in my previous position, and one client alone paid us $40k a month… so I think companies have to ask themselves: Are we going to continue to pay money to an MSSP, or can we get in a couple employees, train them, and have them perform our security monitoring/incident response for cheaper?
With the cost of a security professional rapidly moving into six figures, you aren’t going to get a couple employees cheaper than ANY MSSP platform.
Also, an MSSP is 24/7. To do that in even the smallest environment, you would be hiring 4-5 employees.
An MSSP service would have provided a vulnerability risk assessment, not just incidence monitoring and response. I don’t know what kind of “security audit” they had, but it was obviously lacking by any measure.
If this turns out to be the result of one stupid phishing email, the company network admins really should really be ashamed at the level of resource access that one hacked account was able to reach.
It may be smaller now, but if you continue paying, you encourage more criminals to engage in this behavior. It’s not cheaper at all over any length of time. Not even bringing up brand reputation damage and customers walking out…