Email provider VFEmail has suffered what the company is calling “catastrophic destruction” at the hands of an as-yet unknown intruder who trashed all of the company’s primary and backup data in the United States. The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.
Founded in 2001 and based in Milwaukee, Wisc., VFEmail provides email service to businesses and end users. The first signs of the attack came on the morning of Feb. 11, when the company’s Twitter account started fielding reports from users who said they were no longer receiving messages. VFEmail’s Twitter account responded that “external facing systems, of differing OS’s and remote authentication, in multiple data centers are down.”
Two hours later, VFEmail tweeted that it had caught a hacker in the act of formatting one of the company’s mail servers in The Netherlands.
“nl101 is up, but no incoming email,” read a tweet shortly thereafter. “I fear all US based data my be lost.”
“At this time, the attacker has formatted all the disks on every server,” wrote VFEmail. “Every VM [virtual machine] is lost. Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”
In an update posted to the company’s Web site, VFEmail owner Rick Romero wrote that new email was being delivered and that efforts were being made to recover what user data could be salvaged.
“At this time I am unsure of the status of existing mail for US users,” Romero wrote. “If you have your own email client, DO NOT TRY TO MAKE IT WORK. If you reconnect your client to your new mailbox, all your local mail will be lost.”
Reached by KrebsOnSecurity on Tuesday morning, Romero said he was able to recover a backup drive hosted in The Netherlands, but that he fears all of the mail for U.S. users may be irreparably lost.
“I don’t have very high expectations of getting any U.S. data back,” Romero said in an online chat.
John Senchak, a longtime VFEmail user from Florida who also has been a loyal reader and commenter at this blog, told KrebsOnSecurity that the attack completely wiped out his inbox at the company — some 60,000 emails sent and received over more than a decade.
“I have a account with that site, all the email in my account was deleted,” Senchak said.
Asked if he had any clues about the attackers or how they may have broken in, Romero said the intruder appeared to be doing his dirty work from a server based in Bulgaria (94.155.49[9], username “aktv.”)
“I haven’t done much digging yet on the actors,” he said. “It looked like the IP was a Bulgarian hosting company. So I’m assuming it was just a virtual machine they were using to launch the attack from. There definitely was something that somebody didn’t want found. Or, I really pissed someone off. That’s always possible.”
This isn’t the first time criminals have targeted VFEmail. I wrote about the company in 2015 after it suffered a debilitating distributed denial-of-service (DDoS) attack after Romero declined to pay a ransom demand from an online extortion group. Another series of DDoS attacks in 2017 forced VFEmail to find a new hosting provider.
In December 2018, Romero tweeted that service had been disrupted by a DDoS attack that he attributed to “script kiddies,” a derisive reference to low-skilled online hooligans.
“After 17 years if I was planning to shut it down, it’d be shut down by me – not script kiddies,” Romero wrote on Dec. 8.
Attacks that seek to completely destroy data and servers without any warning or extortion demand are not as common as, say, ransomware infestations, but when they do occur they can be devastating (the 2014 Sony Pictures hack and the still-unsolved 2016 assault on U.S.-based ISP Staminus come to mind).
It’s not clear how or whether VFEmail will recover from this latest attack, but such actions are an unsettling reminder that although most cybercriminals have some kind of short- or long-term profit motive in mind, an intruder with privileged access to a network can just as well virtually destroy everything within reach as they can plant malware or extortion threats like ransomware.
Seems like a poor DR planning to me.. (or none…)
These are my thoughts exactly. One of the first few sentences of the article mentioned that: “The firm’s founder says he now fears some 18 years’ worth of customer email may be gone forever.” With this in mind, I initially felt terrible as that is the worst case scenario. However, my next thought was, “well where is your DR plan?”
They’ve had to have some form of backups or redundancy somewhere. In any case, not only is it scary, but it’s also sad that people will go these lengths to ruin a business. I think this highlights the extreme importance of having solid BCP and DRP policies in place.
This. So much this. There’s so much that’s necessary about proper business practices, not the least of which is proper disaster recovery and business continuity, and it’s not always on the forefront of the beginning infosec professional’s mind. But it’ll be there quick once those beginning infosec pros actually get going in their jobs.
A lot of companies use virtual tapes for backups. If a malevolent actor gained access to an administrative account he could also wipe out the backups stored on the SAN. Unless the DR site uses separate authentication and authorization from the rest of the systems the DR site is vulnerable, too.
Unfortunately a lot of folks want single sign on so they keep credentials in Active Directory. Break into one DC and, with a good password cracker, you have the keys to everywhere. If a villain can remain hidden until he has all the password he needs, complete destruction is easy.
Another issue is that programmers sometimes (often?) embed login credentials in a program used to automate access.
Couple of words come to mind, offsite/offline backups, should always have some rotated backups that are not reachable online or this could happen, bummer. 🙁
You can still retrieve data from formatted hard drives. Kinda makes you wonder if something else actually happened, like a flood or something. Not sure if you can recover from virtual drives though. Is that why they think this?
If on shared hosting (AWS, Azure, GCP, etc.), chances of restoring via HDD recovery would be almost nil. 🙁
Have you ever used a data recovery service? Their success rate is far from 100%.
Not from flash drives or from RAID drives.
Disk recovery works OK-ish for stuff like grabbing some documents off of a non-RAID disk that was formatted. For a blown-away RAID volume where you have to retrieve a functional database? I wouldn’t hold my breath.
Never used a recovery service, I just do it myself.
That makes more sense. I didn’t think about the RAID hard drives. I’m so used to recovering data from formatted drives from my customers, but none of them have been businesses with RAID drives. Just regular configured drives in laptops or desktops that are configured for regular consumers.
One of the programs I use to recover data on formatted drives, makes it very easy to recover almost all of their data. But it’s all disorganized, that part kind of sucks.
Just glad I don’t have too many of the kind where you have to remove the headers of the hard drive. Those I don’t try to do, because I don’t have a clean room available.
But yea, I can see now with RAID drives; that could be a problem.
Yes, you can recover from formatting but there are variables to consider. How many times were the drives formatted? If the bad guy(s) did so, how many overwrites were used to destroy all data on the drives? In the 90’s I read a Peter Norton how to on what I considered a way to conceal/destroy data from all prying eyes. He stated that data could usually be recovered easily from overwrites. Going 8 layers of data deep to get at old data might be possible. So if one were to do 10 overwrites on the complete drive, data should not be recoverable, even by the Feds or State actors!
Now, how much have things changed from then, I don’t know. Drives are much improved, so is the technology to access them and the Feds have the latest and greatest toys to do so but I’m sure they can be defeated by someone who really knows how to destroy data. Solid-State Drives are another story depending on how they are setup, tape backups not considered but who knows.
I would never trust someone with my digital data. It would have at least two other backups at different sites, that rules out a fire destroying everything at one site. It also prevents hackers from getting at all three sites, at least not easily, especially if one is offline and requires a personal visit to access it. But this hack, if meant to only hurt one person, hurt a lot of people. Doubt they’ll ever catch the culprit. Another reason why I “never” store anything online, trust no one.
Not using a backup solution that users immutable backups? Shame.
I second Tom Bishop’s comment. Some orgs seem to trust “another spinning disk or another SSD, run by some other server” as a sufficient backup. Anyone who has given it the least thought knows that removable storage is needed, for exactly the case shown in this article. Taking tapes, or tape cartridges like the old IBM 3480, or big thumbdrives, or even huge boxes of punched cards, and storing them somewhere away from the data center building, gives a method of recovery from a fire, a flood, a tornado, a military battle, or whatever else might damage one location. Distributing copies in other servers’ online storage reduces the risk for a problem at one location, but not for an online intrusion that could penetrate all locations.
You’re certainly right about offline backup. But an offline backup that doesn’t use reliable media is a waste of effort. Speaking from experience.
Also, you kind of want media that uses *current* off-the-shelf tech. Good luck finding a punched card reader!
The nice thing about puch card is that they can be read by eye. Not very fast, but possible. an autofeed scanner can be programmed to read read them.
I recall a backup system that printed QR Code style data on 8 1/2×11 paper which could be scanneed and converted back to data. No specialized hardware needed.
Boston computer museum
Yep. I think we are all pretty much in agreement on this. Why is there no offline backup? When the RansomWare attacks starting going around years back encrypting drive contents, and then a newer version started encrypting all connected/network/backup drives and restore points, it kind of became mandatory I thought.
Your backups must be tested
So you know they work as expected
Offline is best
So you can rest
When hackers strike unexpected
Your limerick, sir, is a winner!
Clearly, you are no beginner
At writing such verse.
But me, I am the worst
Torturing rhymes like a habitual sinner.
From the user point of view, this is why it pays to use a true e-mail client (i.e. not rely only on the web interface) and keep your own copy of all your mail at home.
I use an offline client (Thunderbird) for all mail, including my gmail account. Your suggestion only works if you use POP. With IMAP some mail might be downloaded, but most will be lost.
You can configure Thunderbird to make an offline copy of your IMAP folders. Handy for offline searches. But counting on local copies of your cloud storage is really silly. The extra yak-shaving is not friendly to reliability.
The idea that data is more secure and unlosable just cuz its on physical media you happen to control directly is as naive as assuming a SaaS provider has a proper disaster plan without checking.
It’s a bit convoluted, but running something like Offlineimap (https://www.offlineimap.org) makes local backups quite simple. It can be configured to never delete local copies, or combined with a regular backup solution to keep some versions and give time to stop syncing and restore if a provider goes down.
It also makes migrating really simple if that’s necessary.
What is with the tweeting?
I have worked with system administration in IT departments serving thousands of users for 18 years, including a military contractor and never once seen or been involved in DR planning or testing apart from simple backup restore tests. It might seem like a natural part of any IT service but in my experience it simply is not.
I was at a Federal agency a few years back which had a solid failover/dr plan. However, the contractor never tested it for fear of disrupting users who needed high availability. Then one day there was a power glitch. The backup generator didn’t start, the failover switch failed, and a new contractor was in place the next week. And every month or so a warning notice would go out about the upcoming tests of the failover/dr plans.
Heh… you remind me of an instance I recall: During a failover test we had a failure of our generator (at that time, there was only the one). And that failure managed to lock the operations personnel out of the data center. Hilarity ensued, then panic when services went down as individual UPSs gave out.
That problem was resolved, pronto: During failover tests, one staff member must remain in the DC and in a position to push back to the utility provider. And to let the other person doing the external generator check back in if necessary. 🙂
Thank God that was a test. And that is **why** you test.
Where was the generator located, and the failover power switch?
Inside the data center?
Just curious, back in the late 1990s a nonprofit I volunteered with bought an old unoccupied building to move into. It had been a movie house in the 1940s and 1950s. In the basement was a generator so they could keep power to the building, it looked to be late 1950s or early 1960s equipment and the building might have been repurposed as a fallout shelter after the movie business left the building. There was water damage corrosion on the generator so at one time the building basement got flooded.
What is VFEmail and/or Romero doing to attract so many attacks? He says he may have “really pissed someone off”, but he doesn’t elaborate. Seems like a bad idea to draw attention to your service if you don’t have a plan to mitigate attacks.
It’s not clear. But you don’t really have to do anything untoward to attract DDoS attacks these days. Could just be someone hosted on your server or using your services that they don’t like.
Their is always vulnerabilities in email CMS programs like Hoarde and Roundcube , you can’t put much blame on Rick for this issue.
Conspiracy theory here: attackers want to delete a possible evidence for one of the customers?
Heh, reminds me of Breaking Bad, using the huge magnet in a truck to destroy Gus Fring’s laptop of surveillance records… certainly could be.
POP3 client anyone?
The practically drew the hackers a map and dared them to come in on the about page.
This is a perfect example of why I use POP and not IMAP and why I have setting to not delete local mail that was deleted on the server. Always keep a local copy of your mail.
a “perfect” example. lol. tick the box that saves your IMAP mail and stop blaming the protocol for your failings.
At some point, doesn’t this smell like an inside job? Or at least some level of collusion or at best, gross negligence?
Yep, inside job/ex-employee would explain the catastrophic aspect and no ransom.
My immediate thought, exactly. At best, this was a disgruntled former employee. At worst, their network topology didn’t segment backups from live systems and the hacker got access to everything with a single attack vector. Either way, my guess is that we’ll find out (if they care to publish) there is a systematic lack of quality IT policy at this place.
I think Rick was a one man show running the website, but I could be wrong here.
A few years back I accidentally deleted quite a few emails – all my own issue. Ever since then I keep my own backups of important emails and do not rely on the servers, backups or redundancy implentations of others. I’m sorry to read of this situation. Thanks for the report Brian.
I feel real sorry for that Senchak guy, he lost all his emails :–(
I would want more details before condemning for having no backup plan. If an attacker is patient, (and knowledgeable about the system, — whether through long-term surveillance or inside information) it seems quite possible to corrupt most backup systems as well as the live systems.
If your backup system is _solely_ an online system, you’re doing it wrong. Always, always, always have offline backups offsite. There’s no excuse in 2019 for a professional IT organization to not know how to properly backup data. HA is NOT backup. Rookie mistake.
Seriously, for an organization that old to do something so egregious borders on negligence.
Agreed.
Storage is cheap enough to keep off-site backups that are not connected to the Internet. I personally make off-site (cold storage) backups every year (in addition to regular (weekly) local, not connected, backups). Worst case you lose only the most recent data. For larger commercial organizations, services like Iron Mountain are designed for this.
I wasn’t impressed by Iron Mountain when we spoke with them recently. They had some basic misconfigurations in their public presence. The bright side was that when it was pointed out to them, their technical person acknowledged them and committed to getting them fixed. I think they’ve since fixed the issues.
I appreciate the general “you should always have a truly offline copy of your data”, but I have a hard time picturing how people really use such a thing in the real world.
Distinct limited privileged backup credentials and append only data for backups (with a different hosting provider and distinct key material) seem to be a basic first step for modern systems.
I would really love to say I am amazed by this, but I am not. Many companies are failing to meet they baseline requirements on keeping backups secure and not easily accessible by calls or random nature.
This is a perfect example to hopefully cause some eye opening for business leaders for change.
An email service provider running a biz for 18 years as if it were a mom and pop store. Really?
This sounds like the type of email service a certain presidential candidate would have favored.
Podesta had gmail and was phished. As far as I know, the story about a certain presidential candidate with a private email server being hacked is fake news. But plenty of advisors on both sides have had their emails compromised. Most hacks are simple behavioral engineering attacks. Use 2FA folks.
Including his sons and various operatives?
Bob Mueller sure keeps finding a lot of incriminating emails.
I’m no IT expert by any stretch but I’ve never heard of VFEmail. Why would one use a service like this vs. Gmail, Outlook, or any other commercially/well-known email service??
Gmail, yahoo, outlook, aol, etc. are harvesting your data, reading your emails, and spamming you constantly. Paid email providers often base their business model on having a clean interface, protecting from spam, and not harvesting your data. I could recommend a few email providers, but I don’t want to come across as a shill. You can search Privacytools.io or That One Privacy Site for some comparisons.
Some people like to claim that privacy and security are two completely different things and to not confuse the two, but in my mind, the more you reveal about yourself, the more easily you can be phished or hacked or have your identity stolen. Just make sure that whatever email you use, you have two factor enabled (preferably a U2F hardware key).
Been reading Mr. Krebs for years. One thing bothers me about some of the comments. I have two email sources, one from Spectrum and one from Gmail. In both cases I switched from POP to IMAP years ago.
Every single email I have received going back 17 years is stored on my PC. I dont understand why some of the commenters are indicating POP is superior to IMAP.
Please elaborate.
I believe it’s because, by default, most clients only fetch headers when IMAP is set up. Full messages are downloaded when opened, of course, but I think a lot of clients don’t even retain these for long afterward (again, by default). Assuming you change these settings to fetch full messages as they come in, there shouldn’t be much to worry about in that regard.
I think there may also be some concern with IMAP being more of a sync type system – thus the warning not to connect client software at this time… The client would see an empty mailbox on the other end and think ‘ah yes, I too should be an empty mailbox.’
POP – your email client owns the messages. As soon as she gets them from the server, she usually tells the server to delete them.
IMAP – the server owns the messages, your email program logs in and asks for some to show you. The benefit? Trying to sync up your laptop, your phone and your desktop mail with POP is kind of ugly. (who got mail last, who should delete what?) With IMAP its simple. The server keeps track, and she holds all the messages. That is, until she goes down.
Thankfully many clients will store most/all of a persons messages if asked nicely. So if the users don’t sync up. They will still have the messages on their computer’s at least.
A friend has IMAP. She has a private .PST on her PC. Mail comes to the IMAP folder. She then moves it to the private .PST. She now has a permanent copy and IMAP deletes it from the server and her phone.
I run POP3 across PC, iPad and phone. iPad and phone are set to not delete from server. Only the PC deletes. Yes, I have to delete from iPad and phone but that is my cross to bear.
IF You live door open your house door unlocked.
then After You complain to insurance company You Got robbed….
life is simple, Same thing in cyber world shut the door and That’s it!!
suppid… idiotss!!
Simple!
punch of kids grow Up!
keep the doors closed
In 2011, Google made headlines when it had to use tape to restore the email boxes of thousands users.
https://searchdatabackup.techtarget.com/opinion/Gmail-outage-illustrates-the-importance-of-tape-backups
… tape continues to be the ideal solution for long-term and off-line data storage
You can bet that Google’s tape was library hosted though. Not offline, but nearline. You can still have a malicious attack against a tape library. I’ve seen it done accidentally – a careless backup admin issuing a tape label command and accidentally applying it to all 5000 carts in the library, rather than the ten or so new ones they’d just added. It does take a while to run though…
Many companies think they’re safe with a dual datacentre operation where tape libraries maintain replicas of the data stored in each. It should be evident that this is not the case against a dedicated attack.
I have never heard of the service but think it’s a shame it will give up 17 years
but the admins & management are happy
Previously I’ve never heard of this e-mail/webmail hosting company (did they host for the government or why are they interesting at all these days when free e-mail comes in most packages along something else?), but I guess they at least heard of Iron Mountain and that the whole e-mail content they were hosting could fit compressed on one tape and shipped off with forever retention.
The guy is a hero imho. Running practically a public service for so many years barely covering his costs, I speculate.
The least I could do is backup my own email box or.. go to Micro$oft, Google and the rest of the main stream surveillance state personal data gulpers with shiny tape backups…
Brian, why do you think nobody has been charged for the Staminus hack? It’s pretty clear who did it.
Not a surprise that its 2019 and people still don’t use proper offsite backups. Tapes are still great and so are services like Iron Mountain. If the guy didn’t have servers overseas he could have stored his monthly/annual backups in his basement.
Sucks for the people who lost their email, but still today IT shops large and small don’t implement proper DR procedures.
Offline backups to tape are pretty simple if you only have one server or one office to backup up. Anything larger than that especially anything that needs off site load balancing gets much more complicated to handle. AWS doesn’t offer a tape vault with an S3 like interface.
You have a choice, avoid single server failures with easy to restore backups and load balancing or have a system that can recover from all of the servers being formatted at once. Which more likely a single server being unavailable or someone wiping out all of your servers.
Yes, a lot of companies can’t handle one server deleting its own data, but the complexity of implementation and thus cost jumps significantly between systems that have to handle high availability for normal glitches versus recovering from every possible disaster.