Last week cybercriminals deployed ransomware to 1,500 organizations, including many that provide IT security and technical support to other companies. The attackers exploited a vulnerability in software from Kaseya, a Miami-based company whose products help system administrators manage large networks remotely. Now it appears Kaseya’s customer service portal was left vulnerable until last week to a data-leaking security flaw that was first identified in the same software six years ago.
On July 3, the REvil ransomware affiliate program began using a zero-day security hole (CVE-2021-30116) to deploy ransomware to hundreds of IT management companies running Kaseya’s remote management software — known as the Kaseya Virtual System Administrator (VSA).
According to this entry for CVE-2021-30116, the security flaw that powers that Kaseya VSA zero-day was assigned a vulnerability number on April 2, 2021, indicating Kaseya had roughly three months to address the bug before it was exploited in the wild.
Also on July 3, security incident response firm Mandiant notified Kaseya that their billing and customer support site —portal.kaseya.net — was vulnerable to CVE-2015-2862, a “directory traversal” vulnerability in Kaseya VSA that allows remote users to read any files on the server using nothing more than a Web browser.
As its name suggests, CVE-2015-2862 was issued in July 2015. Six years later, Kaseya’s customer portal was still exposed to the data-leaking weakness.
The Kaseya customer support and billing portal. Image: Archive.org.
Mandiant notified Kaseya after hearing about it from Alex Holden, founder and chief technology officer of Milwaukee-based cyber intelligence firm Hold Security. Holden said the 2015 vulnerability was present on Kaseya’s customer portal until Saturday afternoon, allowing him to download the site’s “web.config” file, a server component that often contains sensitive information such as usernames and passwords and the locations of key databases.
“It’s not like they forgot to patch something that Microsoft fixed years ago,” Holden said. “It’s a patch for their own software. And it’s not zero-day. It’s from 2015!”
The official description of CVE-2015-2862 says a would-be attacker would need to be already authenticated to the server for the exploit to work. But Holden said that was not the case with the vulnerability on the Kaseya portal that he reported via Mandiant.
“This is worse because the CVE calls for an authenticated user,” Holden said. “This was not.”
Michael Sanders, executive vice president of account management at Kaseya, confirmed that the customer portal was taken offline in response to a vulnerability report. Sanders said the portal had been retired in 2018 in favor of a more modern customer support and ticketing system, yet somehow the old site was still left available online.
“It was deprecated but left up,” Sanders said.
In a written statement shared with KrebsOnSecurity, Kaseya said that in 2015 CERT reported two vulnerabilities in its VSA product.
“We worked with CERT on responsible disclosure and released patches for VSA versions V7, R8, R9 and R9 along with the public disclosure (CVEs) and notifications to our customers. Portal.kaseya.net was not considered by our team to be part of the VSA shipping product and was not part of the VSA product patch in 2015. It has no access to customer endpoints and has been shut down – and will no longer be enabled or used by Kaseya.”
“At this time, there is no evidence this portal was involved in the VSA product security incident,” the statement continued. “We are continuing to do forensic analysis on the system and investigating what data is actually there.”
The REvil ransomware group said affected organizations could negotiate independently with them for a decryption key, or someone could pay $70 million worth of virtual currency to buy a key that works to decrypt all systems compromised in this attack.
But Sanders said every ransomware expert Kaseya consulted so far has advised against negotiating for one ransom to unlock all victims.
“The problem is that they don’t have our data, they have our customers’ data,” Sanders said. “We’ve been counseled not to do that by every ransomware negotiating company we’ve dealt with. They said with the amount of individual machines hacked and ransomwared, it would be very difficult for all of these systems to be remediated at once.”
In a video posted to Youtube on July 6, Kaseya CEO Fred Voccola said the ransomware attack had “limited impact, with only approximately 50 of the more than 35,000 Kaseya customers being breached.”
“While each and every customer impacted is one too many, the impact of this highly sophisticated attack has proven to be, thankfully, greatly overstated,” Voccola said.
The zero-day vulnerability that led to Kaseya customers (and customers of those customers) getting ransomed was discovered and reported to Kaseya by Wietse Boonstra, a researcher with the Dutch Institute for Vulnerability Disclosure (DIVD).
In a July 4 blog post, DIVD’s Victor Gevers wrote that Kaseya was “very cooperative,” and “asked the right questions.”
“Also, partial patches were shared with us to validate their effectiveness,” Gevers wrote. “During the entire process, Kaseya has shown that they were willing to put in the maximum effort and initiative into this case both to get this issue fixed and their customers patched. They showed a genuine commitment to do the right thing. Unfortunately, we were beaten by REvil in the final sprint, as they could exploit the vulnerabilities before customers could even patch.”
Still, Kaseya has yet to issue an official patch for the flaw Boonstra reported in April. Kaseya told customers on July 7 that it was working “through the night” to push out an update.
Gevers said the Kaseya vulnerability was discovered as part of a larger DIVD effort to look for serious flaws in a wide array of remote network management tools.
“We are focusing on these types of products because we spotted a trend where more and more of the products that are used to keep networks safe and secure are showing structural weaknesses,” he wrote.
Going to be some pretty legitimate lawsuits over this I imagine.
Nope. From the Kaseya license agreement:
“15. Limitation of Liability. NOTWITHSTANDING ANYTHING ELSE IN THIS AGREEMENT OR OTHERWISE, AND EXCEPT FOR BODILY INJURY CAUSED BY GROSS NEGLIGENCE OR WILLFUL MISCONDUCT BY KASEYA’S EMPLOYEES, AND TO THE FULLEST EXTENT PERMITTED UNDER APPLICABLE LAW, KASEYA AND ITS SUPPLIERS AND LICENSORS SHALL NOT BE LIABLE OR OBLIGATED WITH RESPECT TO THE SUBJECT MATTER OF THIS AGREEMENT (INCLUDING WITHOUT LIMITATION INDEMNIFICATION OBLIGATIONS) OR UNDER ANY CONTRACT, NEGLIGENCE, STRICT LIABILITY OR OTHER LEGAL OR EQUITABLE THEORY (I) FOR ANY AMOUNTS IN EXCESS IN THE AGGREGATE OF THE FEES PAID TO IT BY LICENSEE FOR THE SOFTWARE LICENSED HEREUNDER DURING THE SIX MONTH PERIOD PRIOR TO THE CAUSE OF ACTION, (II) FOR ANY COST OF PROCUREMENT OF SUBSTITUTE GOODS, TECHNOLOGY, SERVICES OR RIGHTS, OR (III) FOR ANY INCIDENTAL, INDIRECT, SPECIAL, PUNITIVE OR CONSEQUENTIAL DAMAGES (INCLUDING, WITHOUT LIMITATION, LOSS OF PROFITS, LOSS OF USE OR DATA, DAMAGE TO SYSTEMS OR EQUIPMENT, BUSINESS INTERRUPTION OR COST OF COVER) IN CONNECTION WITH OR ARISING OUT OF THE DELIVERY, PERFORMANCE OR USE OF THE SOFTWARE, DOCUMENTATION, ANY OTHER MATERIALS PROVIDED BY KASEYA OR OTHER SERVICES PERFORMED BY KASEYA, WHETHER ALLEGED AS A BREACH OF CONTRACT OR TORTIOUS CONDUCT, INCLUDING NEGLIGENCE AND STRICT LIABILITY, EVEN IF KASEYA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES). YOU ACKNOWLEDGE AND AGREE THAT KASEYA WOULD NOT ENTER INTO THIS AGREEMENT UNLESS IT COULD RELY ON THE LIMITATIONS DESCRIBED IN THIS PARAGRAPH.”
Pretty much says it all. And the vendors agreed to it.
Luckily shrinkwrap licenses aren’t enforceable when actual laws are broken. When negligence takes place the company doesn’t have immunity due to a clickthrough agreement, although it typically will require losing a criminal case for the floodgates to really open up for civil lawsuits.
Basically, don’t put much faith in shrinkwrap licensing, for the same reason they can’t make murder legal, they also can’t make any other crime (civil or criminal) legal. They’re smokescreens that companies spend a lot of money on to entice investors into believing the illusion of invulnerability so they’ll put more money into the company.
A lifetime ago, my mathematics professor for differential equations would, on occasion, say to the class, “I can recall when America was a great center of technology and manufacturing, with attention to detail.” Being an older student, I would think to myself, “So can I.”
Versus the modus operandus today – like giant Google – that routinely puts out half-baked products and rely on users to loop back complaints and suggestions.
Same with Microsoft and some Open Source. Great Points!
Since Memory is Cheap they moved away from reqs and put everything in ‘String’ Variables which opens up for the Injection Strings
SMH
Yes…yes…we put everything into string variables now. You caught us. Now someone will block the injection strings that inject strings into strings. Then what will happen? A mass change to base 16 variables and an inevitable attack by injection hexes?!?!11!
There was shoddy work in the past too. But there was less pressure. This is all really started in the 80s with fast money culture taking over. Cheap and fast. More 40 year old buildings will be deemed structurally unsound (hopefully before they collapse). The problem with security is that no one really wants to pay for it, and few really understand the threats and risks. And with so many tech companies just positioning for the next handoff most get to duck away before it hits the fan. In my role, I spend on security until someone at higher up wants to own the decision not to implement what is being proposed. And I make sure that is documented.
Priorities has changed. Things are fast moving, new ideas must be implemented yesterday. And just as important, there is a lack of skilled IT staff everywhere in the world.
Personally I do not understand why MSPs are supposed to run servers on the public Internet with that amount of control over so many machines.
One thing is Microsoft doing it, or Kaseya, they should have the resources and supporting tech to do so.
MSP solution should be VPN only IMHO. VPN from small customers LAN, or maybe just a split tunnel VPN if they can’t do anything else. That would make breaching multiple MSPs way more difficult. Lowering the cost/benefit of that target.
Real nice write up , the whole thing makes more sense to me now
Would simple external vulnerability scans have identified this issue at each MSP or end user client, or would the attackers have needed more aggressive “pen test” tools ? I ask because we perform external and internal vulnerability scans for clients quarterly as part of our managed security service plan, and I’m wondering how often you recommend orgs perform 3rd-party vulnerability scans(internal or external) to keep an eye on things ? Network Detective, Nessus, Nexpose, Security Scorecard, and other tools can frequently be used to identify known vulnerabilities without having admin access or even a non-admin user account to start with. All that is needed are public IP addresses and email and website domain(s).
Yes, an external scanner would normally have identified this web portal server.
I have a feeling though, that 3 years after the server was decommissioned, and they went with a completely new type of system… that it was left out of scope for external scanning.
They may have assessed this server as low risk, even with a major vulnerability, since it no longer connected with anything on the back end.
So it slips through the cracks.
Now, THAT video was a production! Literally checked all of the boxes. They spent more on the speech writers and video production than most companies have IT budgets. I guess it’s charged under lawsuit pacification? Seriously, The Whitehouse line was brilliant (give credit – get pass).
How many dental practices were affected? Anyone know?
“…Kaseya, confirmed that the customer portal was taken offline in response to a vulnerability report. Sanders said the portal had been retired in 2018…”
Same year they merged with Unitrends.
“Going to be some pretty legitimate lawsuits over this I imagine.”
SOP for these clowns. They never bothered doing security properly, so it’s just a front for a scam: Build a “security provider” but do it on the cheap with no real effort put into securing anything but it *looks* good and the suckers pay up. When the security inevitably fails and the whole thing comes crashing down:
1. Transfer all the capital offshore to safe havens. Use Hollywood Accounting to hide it.
2. Then declare bankruptcy. Cry poverty, ask for government bailout.
3. Take money, fold the company and move offshore.
4. Lawsuits? There’s nothing to go after.
The customers, staff and everyone except the C-suite and board are screwed, the board and C-suite make out like bandits. Rinse, repeat.
And just like that……those Sec Budgets just jumped to the top of the list.
Almost all CISOs and cyber executives in the US really don’t understand the basics. It all started with IT outsourcing to India around 2 decades ago. What do you expect from these folks in the USA except for bunch of useless cyber certifications and just brainwash their board and CEO with non-sense they never get!
Bryan, I hoping you can clarify something as I am confused, you start out the article by saying “Last week cybercriminals deployed ransomware to 1,500 organizations that provide IT security and technical support to many other companies. The attackers exploited a vulnerability in software from Kaseya…” but in the video and the the reports I have read say that it was around 50 MSP’s that were compromised. I did read a few places that are saying that in turn those 50 MSP’s had a collective ~1,000 customers that were impacted. Just trying to understand how you came up with 1,500 MSP’s (i.e. 1,500 organizations that provide IT security and technical support)
I work for a MSP, we don’t use this product, but have been watching this whole thing closely.
Appreciate all you do, have purchased your book and followed you for years!
Just finished the video posted in your story and he does say in there that the total count of impacted companies are 800-1,500. Those are not the same as “1,500 organizations that provide IT security and technical support,” but rather the end customers of the MSP’s.
The problem with security is that no one really wants to pay for it, and few really understand the threats and risks. And with so many tech companies just positioning for the next handoff most get to duck away before it hits the fan. In my role, I spend on security until someone at higher up wants to own the decision not to implement what is being proposed. And I make sure that is documented.
CVE link goes to generic website. Is there details of vulnerability? https://nvd.nist.gov/vuln/detail/CVE-2021-30116
Typical Kaseya.
We dumped them a few years ago because they didn’t support MacOS Catalina – the first version that was 64-bit only – when it was released which led to all sorts of customer headaches and us having to replace them for Mac customers. Kaseya’s agent was 32-bit only, despite about 30 months notice from Apple that 32-bit support in MacOS was going away. When confronted they said that they “had other development priorities.”
I’m sure that in this case it’s the same thing – despite three months of notice, they “had other development priorities.”
Yeah pretty slow to develop patches for critical vulnerabilities.
They knocked out a few pretty quickly, but ultimately dropped the ball.
–
Here’s the timeline:
–
CVE-2021-30116 – A credentials leak and business logic flaw, to be included in 9.5.7
CVE-2021-30117 – An SQL injection vulnerability, resolved in May 8th patch.
CVE-2021-30118 – A Remote Code Execution vulnerability, resolved in April 10th patch. (v9.5.6)
CVE-2021-30119 – A Cross Site Scripting vulnerability, to be included in 9.5.7
CVE-2021-30120 – 2FA bypass, to be resolved in v9.5.7
CVE-2021-30121 – A Local File Inclusion vulnerability, resolved in May 8th patch.
CVE-2021-30201 – A XML External Entity vulnerability, resolved in May 8th patch.
06 Apr 2021 Kaseya informed.
10 Apr 2021 Vendor starts issuing patches v9.5.5. Resolving CVE-2021-30118.
8 May 2021 Vendor issues another patch v9.5.6. Resolving CVE-2021-30117, CVE-2021-30121, CVE-2021-30201.
04 Jun 2021 DIVD CSIRT hands over a list of identified Kaseya VSA hosts to Kaseya.
26 Jun 2021 9.5.7 on SaaS Resolving CVE-2021-30116 and CVE-2021-30119.
02 Jul 2021 DIVD responds to the ransomware, by scanning for Kaseya VSA instances reachable via the Internet and sends out notifications to network owners
07 Jul 2021 Limited publication (after 3 months).
What did you switch to?
Curious to know the same. We’re sick of Kaseya’s buggy platform.
This is what happens when a major breach occurs. People start looking with microscopes, and going back through everything.
Everyone wants to pile on and be someone who discloses something damning.
–
It looks like this web server wasn’t patched, but rather just disconnected from the back-end system.
This kind of thing happens ALL THE TIME. When the risks are assessed, they take into account the possible impact if exploited. So there was probably no requirement to patch a system that doesn’t contain any sensitive information and cannot be used to infiltrate the network.
In some cases, Security requests that such servers remain online but isolated, as honeypots.
–
At this point, the only risk this server presents… is reputational, as people can point to any unpatched vulnerability as if it paints a narrative of a history of non-compliance.
–
It is good for Mandiant to notify Kaseya that an old server is still online and vulnerable. But we should not make too big of a deal of this. Especially since it isn’t related to the actual breach.
How this be “not involved” in the bigger hack given the amount of information that would have been available.
Discontinued 3 years ago, disconnected from back end, replaced with new system.
A vulnerability on a web front end, that isn’t connected anywhere, probably as useful as a honeypot.
“yet somehow the old site was still left available online.”
Yeah… looks like it fell through the cracks. This, unfortunately is not uncommon.
Large organizations often overlook stuff like this. And when someone points it out… they may or may not care if there is no risk.
Great info, Krebs. Thanks for the details.
Any thoughts on how the vulnerability leaked to the attackers given that it was disclosed by DIVD on April 2 under the responsible disclosure policy only to Kaseya?
Vulnerabilities can be independently discovered, especially vulns in commercially available products.
Anyone hearing anything about why the Toyota Financial Services website is basically down? Customers can’t see their account with payments and the payoff page is basically a blank page all day. They are apologizing but it just feels weird.
Thanks to this article I can learn more. Expand my knowledge and abilities. Actually the article is very real.
I as Kaseyas 3rd MSP customer and thankfully I dropped them because of their aggressive licensing practices. Jumped the shark.