The Minnesota-based Internet provider U.S. Internet Corp. has a business unit called Securence, which specializes in providing filtered, secure email services to businesses, educational institutions and government agencies worldwide. But until it was notified last week, U.S. Internet was publishing more than a decade’s worth of its internal email — and that of thousands of Securence clients — in plain text out on the Internet and just a click away for anyone with a Web browser.
Headquartered in Minnetonka, Minn., U.S. Internet is a regional ISP that provides fiber and wireless Internet service. The ISP’s Securence division bills itself “a leading provider of email filtering and management software that includes email protection and security services for small business, enterprise, educational and government institutions worldwide.”
U.S. Internet/Securence says your email is secure. Nothing could be further from the truth.
Roughly a week ago, KrebsOnSecurity was contacted by Hold Security, a Milwaukee-based cybersecurity firm. Hold Security founder Alex Holden said his researchers had unearthed a public link to a U.S. Internet email server listing more than 6,500 domain names, each with its own clickable link.
A tiny portion of the more than 6,500 customers who trusted U.S. Internet with their email.
Drilling down into those individual domain links revealed inboxes for each employee or user of these exposed host names. Some of the emails dated back to 2008; others were as recent as the present day.
Securence counts among its customers dozens of state and local governments, including: nc.gov — the official website of North Carolina; stillwatermn.gov, the website for the city of Stillwater, Minn.; and cityoffrederickmd.gov, the website for the government of Frederick, Md.
Incredibly, included in this giant index of U.S. Internet customer emails were the internal messages for every current and former employee of U.S. Internet and its subsidiary USI Wireless. Since that index also included the messages of U.S. Internet’s CEO Travis Carter, KrebsOnSecurity forwarded one of Mr. Carter’s own recent emails to him, along with a request to understand how exactly the company managed to screw things up so spectacularly.
Individual inboxes of U.S. Wireless employees were published in clear text on the Internet.
Within minutes of that notification, U.S. Internet pulled all of the published inboxes offline. Mr. Carter responded and said his team was investigating how it happened. In the same breath, the CEO asked if KrebsOnSecurity does security consulting for hire (I do not).
[Author’s note: Perhaps Mr. Carter was frantically casting about for any expertise he could find in a tough moment. But I found the request personally offensive, because I couldn’t shake the notion that maybe the company was hoping it could buy my silence.]
Earlier this week, Mr. Carter replied with a highly technical explanation that ultimately did little to explain why or how so many internal and customer inboxes were published in plain text on the Internet.
“The feedback from my team was a issue with the Ansible playbook that controls the Nginx configuration for our IMAP servers,” Carter said, noting that this incorrect configuration was put in place by a former employee and never caught. U.S. Internet has not shared how long these messages were exposed.
“The rest of the platform and other backend services are being audited to verify the Ansible playbooks are correct,” Carter said.
Holden said he also discovered that hackers have been abusing a Securence link scrubbing and anti-spam service called Url-Shield to create links that look benign but instead redirect visitors to hacked and malicious websites.
“The bad guys modify the malicious link reporting into redirects to their own malicious sites,” Holden said. “That’s how the bad guys drive traffic to their sites and increase search engine rankings.”
For example, clicking the Securence link shown in the screenshot directly above leads one to a website that tries to trick visitors into allowing site notifications by couching the request as a CAPTCHA request designed to separate humans from bots. After approving the deceptive CAPTCHA/notification request, the link forwards the visitor to a Russian internationalized domain name (рпроаг[.]рф).
The link to this malicious and deceptive website was created using Securence’s link-scrubbing service. Notification pop-ups were blocked when this site tried to disguise a prompt for accepting notifications as a form of CAPTCHA.
U.S. Internet has not responded to questions about how long it has been exposing all of its internal and customer emails, or when the errant configuration changes were made. The company also still has not disclosed the incident on its website. The last press release on the site dates back to March 2020.
KrebsOnSecurity has been writing about data breaches for nearly two decades, but this one easily takes the cake in terms of the level of incompetence needed to make such a huge mistake unnoticed. I’m not sure what the proper response from authorities or regulators should be to this incident, but it’s clear that U.S. Internet should not be allowed to manage anyone’s email unless and until it can demonstrate more transparency, and prove that it has radically revamped its security.
As a USInternet ISP and email customer for 28+ years, I have to say I have NOT Once had any inappropriate email activity with them or with the card I use to pay them. I will restate that they are justified in being rated as far and away the best ISP in the Twin Cities area, always fast and responsive with a huge up-time percentage. Their fiber was easy to have installed, consistently 1Gb+, and and very affordable ( relatively ). Don’t paint them with a broad brush for an obscure technical failing.
Sounds naive, and even like you’re try to cover for them. It’s a bad breach, and that can undo many years of good service by wrecking peoples lives for years to come by the theft of their private communications.
Unfortunately, it’s not clear that any other service will be better. Remember, security is very very very hard, and attackers only have to find 1 breach but defenders have to plug all of the branches. It’s hard.
In this case, it looks like the company probably did not do any security testing or security code reviews (which are very different from regular code reviews). Unfortunately, this is very common.
If you really want security, you are going to have to change a lot. Here are the changes we will need:
1) People are going to have to accept using simpler software which changes slowly and has far fewer features.
2) People are going to have to find a way to evaluate software’s security. No one has figured this out. The security certifications are not very useful because even certified software has lots of serious security bugs.
3) Managers and executives are going to have to find a way to evaluate technical staff (software engineers, IT, operations, etc.) and determine which technical people are producing more secure software and which do not care. Unfortunately, the majority of our technical workforce pays lip service to security.
4) Organizations will have to spend a lot more effort on testing security and a lot less effort on new features. Customers are also going to have to accept that the cost of security is smaller less functional software with fewer features. It will also cost more.
I doubt that these changes will happen. Instead, I suspect we will continue on our current path because it is good enough (i.e. people prefer cheaper software which is somewhat security over perfect software which costs a lot more and has a lot fewer features).
> security is very very very hard
I work with Ansible playbooks, Nginx and IMAP and know what all those words mean. I also manage security. If they just followed standard security practices like those outlined in ISO 27001, they would have several chances to notice something was wrong, including:
1. Developer spots own error in Ansible playbook before it’s run
2. Peer-reviewer catches error.
3. Post-launch reality-check catches error.
4. Standard monthly vulnerability scans catch that IMAP server is exposing unexpected HTTP port.
5. Standard log review notices unexpected activity on IMAP server’s web server.
6. Standard third-party quarterly penetration tests uncovers mis-configuration after the fact.
7. Standard firewall rules block port web access to IMAP server (by only allowing IMAP ports).
This wasn’t a single screw up. Several different pieces of their security program have to be sub-standard for this to happen.
“Obscure technical failing”???
Read the article.
They made public email from hundreds of clients. If YOU are ok with that, that’s fine. Nobody else is. Full stop.
Stop trying to justify your friends’ failure. Nobody cars if “it’s ok in the ‘Twin Cities’ for someone to publish all their emails.” You ‘Twin Cities” guys feel free to have your own ‘standards.’ The rest of the world wants out data secured by those we entrust them to.
I think you don’t comprehend the issue at hand if you think this is “obscure”. What we have is unencrypted, plain-text copies of emails from vast quantities of accounts of users from all walks of life (government, private, personal) that use their service. Not just recent emails either, these are the emails going back months – …
If bad actors had gotten this treasure trove it would be a much, much worse scenario. This data in the wrong hands can wreck havoc… it’s bad. I like my money, my identity, and my stuff. Do you?
Did BK question their pricing, fiber install service, uptime etc? No.
He said he wondered if they were slyly trying to buy his silence,
which given the circumstances would be the #1 corporate move.
It’s a serious incident, plenty of “obscure” technical fails are that,
but who is inferring anything about their pricing, service, etc?
It’s missing the point of a security incident investigation/blog…
Nobody is questioning their customer service either way AFAIK.
But just because trains run on time doesn’t mean all is well…
As a current Securence customer, I’m disappointed that we have not received any kind of notification of the system being compromised by a bad setting or misconfiguration, etc. So the question now is what to do moving forward. What else was exposed?
Assume everything.
US Internet has quality and very affordable fiber internet with minimal overhead. I was very sad when we moved out of their coverage area: I now pay much more for much less for cable-delivered internet.
That said, this incident reflects several gaps in process, automation, basic network security, and audit. The company must demonstrate both the failures, and the mitigations transparently if they want to regain the trust of their customers.
This needs more attention. As Chip reports above, who pays for the notification of breach? I imagine a lazy employee was using this to access and snoop around, perhaps after they left the org. This kind of breach is awful. Imagine the intelligence day traders could be using from watching email exchanges. Imagine if this is happening elsewhere that isn’t even public yet.
Anyone sending anything confidential in an email that is not encryption the message is at fault of that data begin breached. All Email communication should be assumed not secure. So who in the right mind would be sending anything related to day trading in email? SMTP is the worst way possible to communicate securely.
First off all, it doesn’t matter if the information is confidential or not. Dont be an apollogist.
Secong, E2E encryption is beyond most users technical capabilities.
Third, interoperability is a huge issue
Fourth, it isn’t really feasible at scale with today’s technology.
This is victim blaming at its finest. In order to intercept SMTP, you have to compromise the network it’s traveling on. Client to server communications are ordinarily encrypted. It’s true that E2E encryption is more secure, but there is no excuse for such an egregious fiduciary failure as this.
The basic fact is that data is compromised and believing what USi individuals are now saying is also questionable. This is not only a breach of data but also a breach of confidentiality and trust. I for one will be assessing legal implications
I would agree. Legal action should be take
Thank you Aaron.
I’m confused why you responded to yourself, as yourself.
Probably because the respondent is not the original poster Jim.
I just realized that I read an entire article that didn’t try to sell me anything. The article was full of the information that I need and didn’t contain any junk that I didn’t need. That is an impressive feat. Well done.
Chris, I really miss being able to sleep at night, secure in the knowledge that you were minding the the government information store for all of us. Thank you for everything that you did under trying circumstances.
Unreal.
Hilarious.
The worst part is, many of the shmoos whose emails were compromised won’t even care that much. Remember, “privacy” is so bourgeouise. In the future, you’ll own nothing, have no privacy, and be happy.
Well, this isn’t good.
Shucks with Hockey Pucks!
Is that PG enough? … ugh!
What’s that song by Whitesnake? “Here we go Again”?
Time to move to Belize and make bird houses out of popsicle sticks perhaps.
I had concerns about Securence email security years ago and dumped them for another provider. Now we’ve been notified that the messages they were supposed to have purged long ago are probably in the hands of malicious actors. This level of incompetence is appalling. It’s indefensible. This is gross negligence on a scale rarely if ever seen. Management needs to be held fully accountable, up to and including criminal charges. I will be forwarding this to our legal team for possible action.
Di*k move making the domain list publicly available. That made this much worse.
wanna know how long it’s been open in clear text? archive,org anyone?
wanna know how long it’s been open in clear text? archive,org anyone?