Email provider VFEmail has suffered what the company is calling “catastrophic destruction” at the hands of an as-yet unknown intruder who trashed all of the company’s primary and backup data in the United States. The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.
Founded in 2001 and based in Milwaukee, Wisc., VFEmail provides email service to businesses and end users. The first signs of the attack came on the morning of Feb. 11, when the company’s Twitter account started fielding reports from users who said they were no longer receiving messages. VFEmail’s Twitter account responded that “external facing systems, of differing OS’s and remote authentication, in multiple data centers are down.”
Two hours later, VFEmail tweeted that it had caught a hacker in the act of formatting one of the company’s mail servers in The Netherlands.
“nl101 is up, but no incoming email,” read a tweet shortly thereafter. “I fear all US based data my be lost.”
“At this time, the attacker has formatted all the disks on every server,” wrote VFEmail. “Every VM [virtual machine] is lost. Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”
In an update posted to the company’s Web site, VFEmail owner Rick Romero wrote that new email was being delivered and that efforts were being made to recover what user data could be salvaged.
“At this time I am unsure of the status of existing mail for US users,” Romero wrote. “If you have your own email client, DO NOT TRY TO MAKE IT WORK. If you reconnect your client to your new mailbox, all your local mail will be lost.”
Reached by KrebsOnSecurity on Tuesday morning, Romero said he was able to recover a backup drive hosted in The Netherlands, but that he fears all of the mail for U.S. users may be irreparably lost.
“I don’t have very high expectations of getting any U.S. data back,” Romero said in an online chat.
John Senchak, a longtime VFEmail user from Florida who also has been a loyal reader and commenter at this blog, told KrebsOnSecurity that the attack completely wiped out his inbox at the company — some 60,000 emails sent and received over more than a decade.
“I have a account with that site, all the email in my account was deleted,” Senchak said.
Asked if he had any clues about the attackers or how they may have broken in, Romero said the intruder appeared to be doing his dirty work from a server based in Bulgaria (94.155.49[9], username “aktv.”)
“I haven’t done much digging yet on the actors,” he said. “It looked like the IP was a Bulgarian hosting company. So I’m assuming it was just a virtual machine they were using to launch the attack from. There definitely was something that somebody didn’t want found. Or, I really pissed someone off. That’s always possible.”
This isn’t the first time criminals have targeted VFEmail. I wrote about the company in 2015 after it suffered a debilitating distributed denial-of-service (DDoS) attack after Romero declined to pay a ransom demand from an online extortion group. Another series of DDoS attacks in 2017 forced VFEmail to find a new hosting provider.
In December 2018, Romero tweeted that service had been disrupted by a DDoS attack that he attributed to “script kiddies,” a derisive reference to low-skilled online hooligans.
“After 17 years if I was planning to shut it down, it’d be shut down by me – not script kiddies,” Romero wrote on Dec. 8.
Attacks that seek to completely destroy data and servers without any warning or extortion demand are not as common as, say, ransomware infestations, but when they do occur they can be devastating (the 2014 Sony Pictures hack and the still-unsolved 2016 assault on U.S.-based ISP Staminus come to mind).
It’s not clear how or whether VFEmail will recover from this latest attack, but such actions are an unsettling reminder that although most cybercriminals have some kind of short- or long-term profit motive in mind, an intruder with privileged access to a network can just as well virtually destroy everything within reach as they can plant malware or extortion threats like ransomware.
I am…was a VFEmail user, but it’s my habit not to trust my data storage to anyone. Rick wasn’t running a public service, his customers included businesses. The points made above re DR and offline storage are valid especially when talking about an IT company nearly two decades old. His surly attitude pissed me off on a couple of occasions, but I think you’d have to be super pissed off to want to catch so many others in the fallout. So long, Rick, enjoy your retirement.
dont they have DR far remote site ?
this must be a joke , today secondary copy of back and DR is must for each company. i guess there security was weak as well … what a shame !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I have nothing to add for this topic.
Question, though: if I wanted to create a simple, single text file (not a database file or proprietary formated file) containing every email I’ve ever sent/received via a major free email provider, how would you folks suggest I do that?
OfflineIMAP can do this. Will store your email in mbox format, which is essentially just plaintext emails with the headers
Essentially Thunderbird does this though – I have a copy of Thunderbird Portable that I back up to an external USB drive each weekend; as a side bonus, I can pull emails from the remote server to local folders by simple drag-drop to clear the space on the remote server but keep a local copy….
Evidently Thunderbird Portable is only available for Windows – I’ve been on linux exclusively for over 2 decades. But the standard Thunderbird seems to be perfectly portable across linux distros (just copy the .thunderbird directory). Windows is more difficult.
Interesting question
to avoid completely using anything of a proprietary nature I would be interested in converting the eMail to .pdf format. Remember that .pdf has been released into the public domain and is usable in MSFT, AAPL, and Linux environments.
I would recommend using FileManager — just as you would for .jpeg images: set up a /Correspondence directory with sub-directories by topic and correspondent. Such a file ought to be serviceable on any of the 3 major platforms (as above).
getting eMail into .pdf format isn’t all that easy, unfortunately. I like to save the message in .html format using Thunderbird, and then open it in LibreOffice/Writer; from there the message can be saved as .pdf
a conversion tool would be nice; perhaps we’ll discover one in this thread.
I’ve read the current three replies.
Thanks for the ideas….. the Imap/thunderbird idea sounds easiest.
Maybe I’ll give a try this weekend.
Thunderbird uses the mbox format that isn’t proprietary.
Simple quick thought on true DR situation.
1) monthly complete tape off site back ups.
2) weekly system images
3) daily or real time redundancy to off site.
4) proper password rotation !!
I am sure you meant to say offline, not offsite. Or perhaps both. Because having your backups online, even if they are in a different physical location is a recipe for disaster. Sadly, even large companies confuse the two. NotPetya wiped out hundreds of backups simultaneously causing havoc for some major corporations. Anything connected to the Internet is vulnerable.
I thought that was SOP for anyone in the business!
What worries me most about Windows 10 (and what NO ONE seems to be discussing) is the fact that support for Windows 7 ends at the end of the year and we’re all expected to change over, and yet Windows 10 is NOT HIPAA compliant (even if you disable Cortana) which is the KISS OF DEATH in the world of healthcare. Yes, there are measures you can take in Enterprise version to turn off telemetry as well as Cortana (and hope that an update doesn’t undo them), but small labs can’t afford to fill desks with Enterprise licenses and, unfortunately, our legacy software can’t run on other operating systems so Macs are out. I can’t get a single lawyer to commit to an answer that we will be safe with any upgrade or any of my programmers for that matter (most act like they have no clue there is an issue) and, it may only be February, but when you are switching an entire building over to new computers (as well as a new operating system), there is never enough time. I wish Brian would address this issue!
If the HIPAA issue isn’t resolved, many health care providers will be forced to pay for Windows 7 updates. This will be a welcome revenue stream for the miscreants in Redmond. The healthcare industry (for whom I have little love) could, perhaps, engage in a public shaming campaign to force Microsoft into providing continuing free support for Windows 7 until the issues with W10 are resolved. That on the other hand, will righteously piss off lots of folks who made the switch to W10 who would much rather be using W7.
Are there any health care organizations, say medium to large, that use Linux programs? Are there any small organizations doing so? Because they don’t have large development budgets, Linux OS creators rely on users to work through the bugs with the creators. But Linux is, for the moment, much safer than Windows, and it does eliminate reliance on Redmond.
My apologies to BK & readers for the above comment. It was ill-informed. Never should have posted a reply in the first place.
No Internet-connected computer, of any vintage, is hipaa compliant, no matter what claims and assurances you may have received.
They can all be accessed remotely and have medical records exfiltrated. Every healthcare malware and ransomware attack is proof of this.
Security is a Process, not a Product.
This is likely just part of general collapse (Dr’s Joe Tainter and J. Diamond, etc.) in the infosphere. The point is how to deal with these stressors. It will be harder in high-tech arenas such as complex, large cities. To my mind, these hackers represent the barbarians at the borders of the techno-/info-spheres, to use the rome example… (Tainter [2015])
It seems like the ‘net gods and the OS gods need to be tackling this as the dominant issue (security and hardness) for the remainder of this century. I remember the 70’s, when nobody foresaw the massive hacking of our system – easy inter-connectivity dominated our motives and design. It can even be claimed that HTML and the WEB are instantiations of this desire.
Just sayin’….
Motive?
I’ll bet that this is actually a nation state covering their tracks (Russia?) perhaps eliminating evidence related to the U.S. election?
I’m curious what virtualization platform they were running: KVM, vmware, hyper-v, etc. And what version they were using.
It reads to me like the bad actor(s) had root access to the host OS and didn’t need access to the guests. As their intent was to destroy.
That was my first thought too. VM Escape. But with the amount of data they were harboring, there had to be more than one host, which means that the hosts themselves were not secure. Probably Escaped into the first host and then had immediate admin credentials to traverse the network. Maybe we’ll hear about a patch to one of the VM providers in the upcoming weeks.
There a lot of people with that much experience who prefer to run on colocated servers, especially if they are one man bands. So my guess would be – no Virtualization
@Harris S. Newman
I agree, this may be any criminal organization trying to eliminate evidence of a crime(s). At least that was my first guess due to the fact they have no other motive listed
Two comments:
1. Best way to deal with this type of phenomenon is impose hefty penalties and serious prison time for the calprits. A 20 year sentence for criminal damage will make potential IT thugs consider their indecent actions twice.
2. If you are an email service provider you OUGHT to maintain offsite offline regular doomsday scenario backups tapes or what have you. It is reckless not to with no viable excuse.
…and another comment, if you are a user of any public email service provider, any problems haunting the provider will be filtered back to you so backups at the user end are essential even though noone can blame the users in this case for lack of foresight.
Those who preach that the cloud comes without a price should perhaps learn from this incident. The more publicly available your email server is, the more prone to attacks it becomes, irrespective whether it is supported by 800 or so security experts like the big boys in the hood. Sometimes the odd email server running linux stashed in a cellar somewhere is more secure by obscurity alone.
Strange that the hacker got so high privileged access to the machine even when they had different passwords. Or he/she hacked it or used an exploit/vulnerability
I’m wondering if MFA was setup for the admin accounts or alternatively a pwd vault with dual control or a shared secret to authenticate to the machines – for starters
Possibly a spearphishing attack with a RAT tool?
VFEmail, as a result of this unfortunate hack, will likely not return according to the owner’s words on Twitter. They stated, “Yes, @VFEmail is effectively gone. It will likely not return. I never thought anyone would care about my labor of love so much that they’d want to completely and thoroughly destroy it.”
There is actually a lot of good supplemental information found on the BleepingComputer’s article for anyone interested: https://www.bleepingcomputer.com/news/security/hackers-wipe-vfemail-servers-may-shut-down-after-catastrophic-data-loss/
What’s so sad is the fact that, completely regardless of if the owner “pissed someone off”, or any other situation for that matter, someone completely and utterly annihilated a service that many people used. This is almost unfathomable to me, as I find it hard to believe that someone (even nefarious in nature) would go these lengths “just to do it”. Sure, there’s DDoS attacks that happen all too frequently… but these die off given time and the business recovers usually quickly after.
There is some unseen motive that we have yet to uncover as a result of this attack. Was it an insider threat? Maybe. Was it due to insurance fraud? Doubt it; I really hope not. Was it just some random hacker doing it just because they can? Maybe. There’s too many questions with not enough answers this early on. I feel terrible for the owner in having to deal with this. I can’t imagine spending blood, sweat, and tears into your own business and then have it be removed from you effectively overnight. I really hope the truth comes out and whoever did this suffers for their actions.
I am sorry, but I no sympathy for this guy. There is no way this should of happened but, He should of had an off site back up some how of something at least for the past 10 years, or how ever long they have been open. Sorry to hear this, but I bet you just taught a bunch of others to get off their ass and make sure they have better security, updates, and more back ups.
It is clear there wasn’t any offline/tape backup. I would guess a mirrored SAN setup and would guess they got root access to the SAN and the mirror had the same creds.I would guess they didn’t want to pay for a large tape backup solution. With the size of data stores doing tape backup becauses less and less practical as it can take months to do a single backup and with 18 years of emails from so many customers those numbers add up.
Would also say this seems to be a small operation and with as good as you maybe without other people checking your work and thinking of things you didn’t think of stuff gets missed. IT is just too big and complicated for one person to do it all right all the time. When there are paying customers involved it is just irresponsible. I would guess there wasn’t much customer facing documentation on what backup strategies they had in place to ensure their data was safe.
Anyone know what the pricing was on this solution? People paying bottom dollar you get what you pay for…
You can fit 30TB (compressed) on a single LTO-8 cartridge (if you can find one at the moment). A decent tape library costs less than ten thousand dollars, and any decent backup program (Veeam, for instance) can do forever forward incremental backups and synthetic full backups.
Datastore size is no excuse for not having proper backup.
Lots of people talking about offsite backups and offline backups and being highly critical of the data protection arrangements for this company. I don’t think this is entirely fair.
For more than a decade every new entrant into the backup market has been solely online disk based, at least initially and come in with a message that tape is dead. You can’t expect non-data protection experts to fully understand the implications of online vs nearline vs offline, especially if every product that’s being targeted at them is telling them that online disk is best. A large majority of the data protection industry is at least in part to blame for this kind of disaster. We need to look at our messaging and reign in marketing departments who insist on simple, easy generalisations to get the sale.
Simply put, if you can issue an “echo /dev/random > /dev/sda1” or a “format d:” so can a hacker. It doesn’t matter if that is on a Windows or Linux backup VM or on a large appliance (which will typically be running Linux or BSD and absolutely not “immutable” as is often claimed).
Tape on a shelf (an offsite shelf) is the only way you can even approaching 100% assured restore, but it’s relatively slow to the first data being recovered, it’s also expensive if you use small volumes of data, so some sort of online disk is still valuable. You can have a better speed to first restore in a disaster using some sort of cloud hosted object storage *with object retention enforcement*, but it has to be cloud hosted with the cloud provider having sole and total control of the OS of the servers/VMs and the customer only having access to the object API. Both offsite tape and cloud object storage are still vulnerable to a dedicated attack – the hosting providers can be attacked the offsite vault can be attacked, though these are much higher bars to reach.
Then you have to retain some method for keeping access to the software you need to perform the restore, the installs, any enabler keys, passwords, encryption keys, etc – after all that, you’re gong to have to recover the backup server and then it’s catalogue database.
This stuff gets very complicated, very quickly and much of the industry has spent a lot of time trying to convince their customers that it’s easy. It’s not.
Disclosure: I work for one of the data protection vendors who still have a tape offering. I am not going to mention who we are, I’m also not going to call out specific competitors, I’m not trying to sell, just inform.
“Tape on a shelf (an offsite shelf) is the only way you can even approaching 100% assured restore,”
Are there not online solutions that support WORM?
I thought I’d replied to this, it appears not:
WORM is an archive technology, while there are backup products that support WORM tape, it’s not really intended for operational recovery, because you can’t re-use it.
LTO-8 WORM media currently costs around $150+ per cart, for 12TB native (30TB compressed). That’s a really expensive proposition for backup, if you just have to bin it, once it’s finished with.
It does indeed sound very difficult to grasp and quite suspicious with the alleged wrongdoer leaving nothing to be retrieved and having access to different firewall door accesses. It may just be a “convenient” explanation for something more sinister.
He is so legit I have confirmed it, CYBER DON will help you fix any kind of credit issues. My old client introduce me to CYBER DON and I get in touch with him, He help me increase my credit score to 810 excellent plus and He help me remove eviction from my credit report within 72 hours. He also help me clear off my three credit card debts within few weeks, He’s real, fast and affordable price. Contact him
(CYBERDON212 at GMAIL dot COM)
It seems like the ‘net gods and the OS gods need to be tackling this as the dominant issue (security and hardness) for the remainder of this century.
I have been battling to pay off my credit card but a friend of mine introduced me to this hacker guru called CYBER DON. He help me erase my driving records with few negative items on my credit report and he also increased my credit score to 800 excellent plus within 72 hours. He also pay off my credit card debts within few weeks, all thanks to him. His service is fast with a substantial service charge. Contact him now and be free. Contact details:
(CYBERDON212@GMAIL.COM)
I have been battling to pay off my credit card but a friend of mine introduced me to this hacker guru called CYBER DON. He help me erase my driving records with few negative items on my credit report and he also increased my credit score to 800 excellent plus within 72 hours. He also pay off my credit card debts within few weeks, all thanks to him. His service is fast with a substantial service charge. Contact him now and be free. Contact details:
(CYBERDON212 at GMAIL dot COM)
If you are have issues regarding boosting your credit score, don’t hesitate to get in touch with TROJAN VIRUS the credit guru. He help me delete all the negatives items on my credit report and help me increase my credit score to 795 which is a golden score within 72 hours. His service are fast, secured and reliable without stress, Contact him:
(TROJANVIRUS212 at GMAIL dot COM)
Too bad they didn’t have a data backup to film. Disaster-proof. Very high-density of storage with DIY possibility of data retrieval if the companies that do it go out of business.
Piql and CPCLONDON offer such a thing. There’s probably more.
Unlike tape, film is unaffected by magnets, seismic shocks rattling hard drives, hacking, and third-party mix-ups. All cloud storage is susceptible to screw-ups and mechanical failure.
Tape degrades after 5 to 10 years and has to be re-copied. What happens when your tape reader busts and the company that encrypted it goes belly up?
Film easily lasts 50 years with minimal care, as evidenced by everyone’s parents’ old home movies and negatives. With care, archivists have films going back at least 100 years.
Wonderful Information, I did fix my credit and got rid of my debts with the help of THECREDITWIZARD at WRITEME dot COM and he did a near perfect job on my profile increasing my credit about 284 points in a few days. My score’s amazing at its high 780’s now and all debts and inquiries are gone & I’ve never been so happy like this. I refer this Credit Wizard to those who have a fixed date to fix their profile. He will surely deliver if you maintain your side of the bargain. I’m Pamela Green.
I had credit scores of 350 (Transunion) 440 (Equifax) in January. I applied for a surgery loan in February and was not approved because of my low FICO scores. I contacted my loan officer and she told me that I need a score of 750-805 to get qualified (considered by FICO as Good) and I was desperately in need to get my score boost because I had to do the surgery at that time. I had seen some wonderful recommendations of a good Credit Agency on the internet called TROVIANCREDITREPAIR So I contacted them and they fixed my FICO scores to 805+ in less seven working days across all 3 credit bureaus and I was approved within a week. I still have my scores at 790 because it is a permanent hack. I am so happy I found this Credit Agency and I will recommend them to anyone out there who is desperately in need of their services as well, you can contact them on now TROVIANCREDITREPAIR@HOTMAIL.COM