Cloud hosting provider iNSYNQ says it is trying to recover from a ransomware attack that shut down its network and has left customers unable to access their accounting data for the past three days. Unfortunately for iNSYNQ, the company appears to be turning a deaf ear to the increasingly anxious cries from its users for more information about the incident.
A message from iNSYNQ to customers.
Gig Harbor, Wash.-based iNSYNQ specializes in providing cloud-based QuickBooks accounting software and services. In a statement posted to its status page, iNSYNQ said it experienced a ransomware attack on July 16, and took its network offline in a bid to contain the spread of the malware.
“The attack impacted data belonging to certain iNSYNQ clients, rendering such data inaccessible,” the company said. “As soon as iNSYNQ discovered the attack, iNSYNQ took steps to contain it. This included turning off some servers in the iNSYNQ environment.”
iNSYNQ said it has engaged outside cybersecurity assistance and to determine whether any customer data was accessed without authorization, but that so far it has no estimate for when those files might be available again to customers.
Meanwhile, iNSYNQ’s customers — many of them accountants who manage financial data for a number of their own clients — have taken to Twitter to vent their frustration over a lack of updates since that initial message to users.
In response, the company appears to have simply deleted or deactivated its Twitter account (a cached copy from June 2019 is available here). Several customers venting about the outage on Twitter also accused the company of unpublishing negative comments about the incident from its Facebook page.
Some of those customers also said iNSYNQ initially blamed the outage on an alleged problem with U.S.-based nationwide cable ISP giant Comcast. Meanwhile, competing cloud hosting providers have been piling on to the tweetstorms about the iNSYNQ outage by marketing their own services, claiming they would never subject their customers to a three-day outage.
iNSYNQ has not yet responded to requests for comment.
Update, 4:35 p.m. ET: I just heard from iNSYNQ’s CEO Elliot Luchansky, who shared the following:
While we have continually updated our website and have emailed customers once if not twice daily during this malware attack, I acknowledge we’ve had to keep the detail fairly minimal.
Unfortunately, and as I’m sure you’re familiar with, the lack of detailed information we’ve shared has been purposeful and in an effort to protect our customers and their data- we’re in a behind the scenes trench warfare doing everything we possibly can to secure and restore our system and customer data and backups. I understand why our customers are frustrated, and we want more than anything to share every piece of information that we have.
Our customers and their businesses are our number one priority right now. Our team is working around the clock to secure and restore access to all impacted data, and we believe we have an end in sight in the near future.
You know as well as we that no one is 100% impervious to this – businesses large and small, governments and individuals are susceptible. iNSYNQ and our customers were the victims of a malware attack that’s a totally new variant that hadn’t been detected before, confirmed by the experienced and knowledgeable cybersecurity team we’ve employed.
Original story: There is no question that a ransomware infestation at any business — let alone a cloud data provider — can quickly turn into an all-hands-on-deck, hair-on-fire emergency that diverts all attention to fixing the problem as soon as possible.
But that is no excuse for leaving customers in the dark, and for not providing frequent and transparent updates about what the victim organization is doing to remediate the matter. Particularly when the cloud provider in question posts constantly to its blog about how companies can minimize their risk from such incidents by trusting it with their data.
Ransomware victims perhaps in the toughest spot include those providing cloud data hosting and software-as-service offerings, as these businesses are completely unable to serve their customers while a ransomware infestation is active.
The FBI and multiple security firms have advised victims not to pay any ransom demands, as doing so just encourages the attackers and in any case may not result in actually regaining access to encrypted files.
In practice, however, many cybersecurity consulting firms are quietly urging their customers that paying up is the fastest route back to business-as-usual. It’s not hard to see why: Having customer data ransomed or stolen can send many customers scrambling to find new providers. As a result, the temptation to simply pay up may become stronger with each passing day.
That’s exactly what happened in February, when cloud payroll data provider Apex Human Capital Management was knocked offline for three days following a ransomware infestation.
On Christmas Eve 2018, cloud hosting provider Dataresolution.net took its systems offline in response to a ransomware outbreak on its internal networks. The company was adamant that it would not pay the ransom demand, but it ended up taking several weeks for customers to fully regain access to their data.
KrebsOnSecurity will endeavor to update this story as more details become available. Any iNSYNQ affected by the outage is welcome to contact this author via Twitter (my direct messages are open to all) or at krebsonsecurity @ gmail.com.
TODO: Create an Incident Management playbook.
TODO: make offline backups.
TODO: patch vulnerabilities.
TODO: educate users about phishing.
TODO: add email server filtering.
TODO: rotate encryption keys.
TODO: …
Those are all valid non-mission reasons you pay outsourcers to do the ordinary things, like hosting bookkeeping software. It’s not a checklist of things you should expect your hosting provider to ignore.
Unfortunately, that echoes the TO DO list of most companies who have not been compromised… yet. (It also echoes documentation I’m gathering for an internal IT Audit, and Federal regulatory examiners.) Qualified IT professionals already know what TO DO — that’s why all we can relate to that list. The challenge is to convince non-IT-minded bosses to provide the resources required TO DO. It sounds as though iNSYNQ missed that part.
Unblinking, you are right. Work hard, play harder. Thank you.
Thank you Jaded for sharing your knowledge in an open way. We have to work all together to help each other.
From the cached twitter feed:
“Year after year, #security remains at or near the top of the list of IT concerns. Data leaks, hacking, compliance breaches, and other issues push IT departments to find new ways to lock down IT infrastructure and secure data. The cloud can help. How?”
It’s almost like the cloud is just someone elses computer!
You forget “We take security and your privacy seriously….”
The “cloud” is just putting your data in the hands of others. Beancounters love the “cloud” because it costs less than what you can do on your own with decent IT staff. Of course cloud providers like to say they’re doing it by overlapping support resources allowing them to make do with less staff. So far, though, it seems an awful lot like they’re cutting corners all over the place and that’s how they’re doing it for less.
There is a small update on iNSYNQ’s blog. Basically says “probably not today.”
“Unfortunately for iNSYNQ, the company appears to be turning a deaf ear to the increasingly anxious cries from its users for more information about the incident.”
That sounds a little more unfortunate for customers than the company.
Anyone else wonder if there is an analogy to those wonderful volunteer firemen/persons who rush to save the day, and are later found to have committed the arson?
Would be a shame if some of the IT protection racket was involved in starting any of these fires.
My thoughts exactly. With all these consulting firms advising to pay up, I wonder if any, or how many, are engaged in infecting businesses in the first place, and then telling them “oh just pay up, its quicker and easier”
Inexcusable for the businesses and their CPAs to be kept in the dark.
Now would be a great time for the Victims deprived of their data to look very carefully at their Business Interruption insurance coverage and at their Cyber Insurance policies.
Usually a few days would not be that important, unless Quarterly filings are due, or worse, it is April 15th . . .
These “victims” tried to get away with the least expensive data storage solution: cloud servers with minimal local IT skills. They outsourced their data storage, rather than hire an IT professional and buy the requisite hardware.
And they got what they deserve.
This happens over and over and over. Supposedly educated people taking shortcuts and pinching pennies.
It caught them with a bite to the pants.
I’m sure most of them didn’t disclose to their clients that private client data was handled so badly. That’s why they’re in a panic now.
The secret will soon be out: don’t trust an accountant who fails to provide details about their data storage.
“We take your security seriously” always means “we have no idea what happens to the data after it leaves our local network.” Doesn’t anyone learn?
Why is this criminal activity not internationally ajudicated with a sentence of death ? Attacks have hit hospitals etc. that have paid to release their data.
A recent arrest led to a slap on the wrist & no criminal record !!
Defence lawyer claimed no money or data was released & he was a good kid.
Why are these people not deemed “international terriorist” where capture would lead to much harsher sentences.
“These “victims” tried to get away with the least expensive data storage solution: cloud servers with minimal local IT skills”
Isnt that exactly what Infosec pros recommend for a small business that doesnt have the resources to do it themselves?
I’ve never witnessed any IT professional recommend that a small company entrust the live operation of their company’s data to computers stored in another building, either across town or across continents.
I’ve heard it said that such cloud storage can be a useful resource to safely store backup data, as “a backup for your backup.” But again, I’ve never witnessed an IT professional recommend that a small company’s only backup be stored in a cloud.
Cloud nine was ransomwared sometime in 2017 and was offline for nearly two weeks. The MSP I work at migrated the data to a new on premise server that cost less than half of what the annual fee to C9 was and is significantly faster to boot as I employed tiered storage for VMs with solid state drives. Food for thought.
It’s sad how security is never a priority for so many companies … until something happens, then they enlist “security professionals” … maybe you should have done that from the beginning!
Food for thought … trust NOBODY … except me — you can trust me, i promise .. buahaha .. just playin i don’t even trust myself sometimes 😛
You are talking about humility. We miss the point many times. Best regards.
This is an object lesson on why the Cloud isn’t necessarily a good place for businesses to place active data. It’s fine for backups, but a business looses control of operations, not just from ransomware, but just from loss of internet connectivity. My business uses G Suite, and Wednesday during a thunderstorm the ISP went down for 45 .minutes. Everybody wound up having a short day. I did IT for a dispensary, and when the ISP went down, sales stopped. The inhouse servers don’t go down. So I don’t normally recommend cloud services…
What you said. Storing vital business data on “someone else’s computer” is a formula for a business-class Darwin Award.
I’ve been screeching about this re. QuickBooks Cloud for years and it’s largely fallen on deaf ears.
Depending on if/when service is restored, the present attack will cause some number of business bankruptcies. Some of these will be small businesses that trusted their accounting to accountants who in turn trusted someone else’s computer.
There will also likely be loss of businesses’ customer information including financial information, leading to attacks on their customers. So: you bought a widget from ABCco, paid via debit card, your debit card is in their cloud file: uh-oh, bye-bye to your checking account.
It’s time to bring the hammer down in the form of the one thing that the Boy Kings of Silly-Con Valley hate most of all: regulation.
A few things to start out:
– Cloud-based services should be required to include local machine applications and local machine backup, sufficient to conduct essential operations in the event of cloud service failure or internet outage.
– Cloud service providers with more than a threshold number of clients should be required to conduct regular security audits and publish the results in accessible and transparent form, in a manner analogous to the on-time performance stats of airlines. Market forces will deal with the poor performers after that.
– “No warranty” clauses in EULAs for mission-critical apps should be forbidden, period. Arbitration clauses should be forbidden from including negligence, such as negligent security practices.
As for the sources of these attacks, I’m a progressive Democrat but I endorse GW Bush’s “harboring” doctrine: states that fail to take action against subnational groups or nonstate actors that conduct attacks across national borders, should be held responsible for the attacks.
International legal mechanisms are needed to hold “harboring” countries accountable for the financial and other costs of cyberattacks based in those countries. If they keep harboring and won’t pay the judgements, cut them the hell off from the internet: they can try getting dialup modems to work on international PSTN trunks (good luck to that).
Enough was enough long ago.
I’m not normally a big government advocate either, but I totally agree with you – especially with enough is enough. I also like your idea of walling off countries who don’t cooperate from the internet; we need to slap a “Great Fire Wall of China” on them, so to speak.
I believe is the other way around. Opening the Internet as much as posible to work all together to build a better World. Best regard
I hope this was a joke/troll/sarcastic comment. I don’t think anyone would be naive enough to believe that “opening up the internet” would somehow make everyone sing campfire songs and hug trees while holding each other’s hands. That’s not going to just magically make bad people stop being bad people.
What’s clearly NOT working is what we’re doing now. If ransomware effects hospitals and delays healthcare to its patients, this could be deadly–literally. We need stronger laws against this kind of crime instead of a slap on the wrist, or worse, making the internet even more open.
1) doesn’t Google drive have offline capabilities? 2) if your business depends on internet, where is the second backup wan link whether it’s another isp or cellular? Or heck even wifi as wan to some other available network (with VPN of course)?
1) Yes, but collaborative work is pretty much done, and any other tasks that require actual connection to a website, such as placing orders, paying accounts receivables, etc.
2) I’ve been barking up that tree for over a year to no avail. They have been warned to get a backup ISP, and it’s on them if they don’t. The backup right now is personal cell phone hotspots, which kind of flies like a Lead Zeppelin, and doesn’t work on workstations because they don’t use WIFI.
It is unthinkable and completely irresponsible for a QB hosting company to store their backups on the same network as their server farms. If InSynq were truly security conscious (as they claim to be) they would have at least had backups to offer their customers. We asked repeatedly for the latest backup and were given excuses such as ” your request has been sent to another team for action” or that we had been put “on a list” of customers who would be getting backups. We kept pushing and were finally told that they did not have access to the backups either.
Oh the curtness of their Chat Personnel …
Chat Log: https://ibb.co/Cm0pNtG
Even when presented with media personality, like Brian Krebs, they basically sent me to marking.
Let’s hope this ends well!
“The Cloud Is Just Someone Else’s Computer”
Becky seems to be every bit of a bot as Justin.
Nope, looked her up, a real person. She has been working there over 7 years and according to a video on YouTube, she’s the head of sales.
Pretty dismissive, I may add to anyone NOT a current customer!
Regarding the “Justin” bot, pretty funny canned responses!
TBH, No verifiable credentials were presented and i doubt webchat is the preferred method of decimating authoritative information.
I am guessing that was why you were dismissed. less so because the company doesn’t care about their image.
And IIRC most press operations are handled out of the marketing department anyway, so it looks like you were given solid advice.
All they have to do is restore their VM machine snapshots and their pre-ransomware offsite backups..
Wait-wut?
Ransomware-resistant backup techniques are difficult to get right, especially when it is necessary to save transactions right up to the point of going offline.
Yup.
That’s what I did last year to recover a Windows server that was hit. Worked like a charm. Snapshots are your best friend – assuming your hosting provider doesn’t fall victim and only your vps is affected.
Trust cloud they said…Whats the worst to happen they said… Offline is far more easier to secure for small business. Just my opinion
CloudJumper had all their servers ransomware and inaccessible for 3 days a few months ago. Similar story, client was out with no communication. Never host with a private cloud, you never know how they’ve setup your security stack.
Separate your tenants, reduce the risk of spread. Multi tenant is the worst idea ever.
Gad, I can just imagine the next press release from iNSNK:
“We are pleased to announce a new service for our customers. It is called Ransomed iNSNK assurance. For only 1 Bitcoin per 100 megabytes of data lost from our cloud datacenter, we will negotiate on your behalf with the ransomed thieves to reduce the amount of money demanded for the return of your data. We usually pay 90 percent of the demanded ransom amount on your behalf plus our normal fees. Take the hassles out data security and send us your money now!”
Anyone who thinks that they are immune to this kind of attack is delusional. The person that comes up with the magic bullet to protect against these kinds of attacks will be very successful in the market. From my experience, the best protection is just good, standard IT best practices. Not sexy, but it reduces the risk of it happening, and gives a path to recovery if it does.
The “magic bullet” isn’t really so magic. It consists of skill, discipline and money, equally weighted. With these resources properly, consistently applied and aligned intrusions *can* be practically eliminated.
Getting companies to understand and commit to this is the real challenge.
In depth defense at minimum, besides backups(which are suspect of course). Management console and Active Directory settings to set permissions on encryption activity, perhaps a good tool that has a good reputation to do that for you, and an anti-malware that guards against folder and other similar activity that malware exhibit. And of course best practices in email, and maybe even email restrictions to keep social engineering to a minimum.
To this day – in depth defense is still the standard to be judged by.
It was “NSYNC” that had that famous hit “Bye Bye Bye” after all… How ironic, since that’s what this iNSYNQ is being told by the market and its customers following this incident, all because they didn’t handle the basics of business continuity planning, testing and validation. I bet Justin Timberlake and friends had backup tour gear and buses back in the day! 😉
Another reason more companies should STOP relying on software and firewalls to protect their data and transition to WORM storage. This storage locks data at the hardware level, rendering data deletion impossible. It’s not a coincidence Gov’t agencies, stock exchanges and other criminal enforcement agencies all use this product. Stop risking your data, wake up! Greentec-USA.com
No web reputation, no BBB rating – that would be taking a risk.
I agree with you. Best regards
If It’s Raining Outside, Should One Ask The Weatherman To Confirm?
Ok folks. I’m not a highly educated person and do not have the ability to express myself as eloquently and succinctly as most you. I know virtually zero about the technical workings of computers and the related peripherals, however, I don’t need a weatherman to tell me it’s raining. WT*, ELLIOTT????? Lay off the subterfuge and obfuscation…the pooch has been screwed!!. Y’all did not have industry standard back ups nor an industry standard disaster recovery plan in place. Y’all bet on the come that this situation wouldn’t happen to you. And why did you place that misguided bet?
F**king greed mi amigo. It would have cost you some pretty pesos to do that. Now I’m scrambling to make payroll, I have financial statements and 2018 tax return to complete for a much needed loan and here I am in this situation because Insync. Now here’s the deal, it ain’t like I’m God and you’re Abraham and I’m asking you to go murder your child so you can prove how much I mean to you. No, no mi hermano, I’m easy. I just want you to come clean and tell all of us the truth. “Your data is gone FOREVER!!! It’s gone. There is nothing to recover. ZERO, ZILCH, NADA.” The truth will set you free. Moreover, you need to put that shovel down…you’re hole is just getting deeper.
I have noticed you don’t monetize krebsonsecurity.com, don’t waste your traffic, you can earn additional cash every
month with new monetization method. This is the best adsense alternative for
any type of website (they approve all sites), for more details simply search in gooogle:
murgrabia’s tools
Ransonware bill payed by bitcoin? Or other currency accepted too??
Crooks got btc guess what they will dump that……?
Return they get Fiat currency!
Did these people have no vetting process? Before relying on any 3rd party for critical infrastructure they should be audited and assessed for risk.
Almost every breach has been the result of vendor failure, why isn’t vendor risk management a primary part of computer security planning?
Most likely been mentioned before but the name iNSYNQ made me laugh compared with the 90’s heart throbs Nsync
thank you for sharing this blog with us keep sharing more such blogs .it was really very informative for us Technology Write for Us
The restore problem becomes infinitely harder and more time consuming when it was a domain admin credential s breech (for an unknown period of time), followed by a crypto event. Air-gap your backups!!!!!!
Sure, the Equifax breach of 2017 made us all pretty mad. But what is the purpose of continually lambasting Equifax year after year? Often, the true testament of an organization’s integrity and trustworthiness is how they respond to a disaster like this.
A little research shows that Equifax is doing everything in their power to course-correct by implementing its 2020 technology plan. In 2018 they hired over 1,000 people and spent $307M to drive this plan forward as they actively promote a “security first” culture. They plan to spend the same in 2019 and continue the improvement process while working toward being a leader in data security and building a customer-centric service model.
No, I’m not an employee of Equifax and was never what you’d call a huge fan—until now because of how the new leadership is changing their entire culture. It’s impressive.
Let’s start letting them fail forward. They’re making the effort, and the cybersecurity community should embrace it. Some organizations don’t bounce back from a major breach. Smart organizations jump in and expend resources improving their security posture and are safer as a result. The rest of us should be learning from those guys. More power to them!