The Minnesota-based Internet provider U.S. Internet Corp. has a business unit called Securence, which specializes in providing filtered, secure email services to businesses, educational institutions and government agencies worldwide. But until it was notified last week, U.S. Internet was publishing more than a decade’s worth of its internal email — and that of thousands of Securence clients — in plain text out on the Internet and just a click away for anyone with a Web browser.
Headquartered in Minnetonka, Minn., U.S. Internet is a regional ISP that provides fiber and wireless Internet service. The ISP’s Securence division bills itself “a leading provider of email filtering and management software that includes email protection and security services for small business, enterprise, educational and government institutions worldwide.”
U.S. Internet/Securence says your email is secure. Nothing could be further from the truth.
Roughly a week ago, KrebsOnSecurity was contacted by Hold Security, a Milwaukee-based cybersecurity firm. Hold Security founder Alex Holden said his researchers had unearthed a public link to a U.S. Internet email server listing more than 6,500 domain names, each with its own clickable link.
A tiny portion of the more than 6,500 customers who trusted U.S. Internet with their email.
Drilling down into those individual domain links revealed inboxes for each employee or user of these exposed host names. Some of the emails dated back to 2008; others were as recent as the present day.
Securence counts among its customers dozens of state and local governments, including: nc.gov — the official website of North Carolina; stillwatermn.gov, the website for the city of Stillwater, Minn.; and cityoffrederickmd.gov, the website for the government of Frederick, Md.
Incredibly, included in this giant index of U.S. Internet customer emails were the internal messages for every current and former employee of U.S. Internet and its subsidiary USI Wireless. Since that index also included the messages of U.S. Internet’s CEO Travis Carter, KrebsOnSecurity forwarded one of Mr. Carter’s own recent emails to him, along with a request to understand how exactly the company managed to screw things up so spectacularly.
Individual inboxes of U.S. Wireless employees were published in clear text on the Internet.
Within minutes of that notification, U.S. Internet pulled all of the published inboxes offline. Mr. Carter responded and said his team was investigating how it happened. In the same breath, the CEO asked if KrebsOnSecurity does security consulting for hire (I do not).
[Author’s note: Perhaps Mr. Carter was frantically casting about for any expertise he could find in a tough moment. But I found the request personally offensive, because I couldn’t shake the notion that maybe the company was hoping it could buy my silence.]
Earlier this week, Mr. Carter replied with a highly technical explanation that ultimately did little to explain why or how so many internal and customer inboxes were published in plain text on the Internet.
“The feedback from my team was a issue with the Ansible playbook that controls the Nginx configuration for our IMAP servers,” Carter said, noting that this incorrect configuration was put in place by a former employee and never caught. U.S. Internet has not shared how long these messages were exposed.
“The rest of the platform and other backend services are being audited to verify the Ansible playbooks are correct,” Carter said.
Holden said he also discovered that hackers have been abusing a Securence link scrubbing and anti-spam service called Url-Shield to create links that look benign but instead redirect visitors to hacked and malicious websites.
“The bad guys modify the malicious link reporting into redirects to their own malicious sites,” Holden said. “That’s how the bad guys drive traffic to their sites and increase search engine rankings.”
For example, clicking the Securence link shown in the screenshot directly above leads one to a website that tries to trick visitors into allowing site notifications by couching the request as a CAPTCHA request designed to separate humans from bots. After approving the deceptive CAPTCHA/notification request, the link forwards the visitor to a Russian internationalized domain name (рпроаг[.]рф).
The link to this malicious and deceptive website was created using Securence’s link-scrubbing service. Notification pop-ups were blocked when this site tried to disguise a prompt for accepting notifications as a form of CAPTCHA.
U.S. Internet has not responded to questions about how long it has been exposing all of its internal and customer emails, or when the errant configuration changes were made. The company also still has not disclosed the incident on its website. The last press release on the site dates back to March 2020.
KrebsOnSecurity has been writing about data breaches for nearly two decades, but this one easily takes the cake in terms of the level of incompetence needed to make such a huge mistake unnoticed. I’m not sure what the proper response from authorities or regulators should be to this incident, but it’s clear that U.S. Internet should not be allowed to manage anyone’s email unless and until it can demonstrate more transparency, and prove that it has radically revamped its security.
Wow!
‘You never cease to amaze me’
John to Sherlock
So much for Change Management and Internal Audit doing their jobs. The CIO, CISO (if they have one), all senior IT staff, IT Security, and IT Audit should be fired.
I worked for them back in 09 (my emails were probably there)… The stories I could tell. They take shortcuts all the time
We worked together, but I was a lowly level 0 so I don’t recall many shortcuts.
It looks like usiwireless.com doesn’t separate employee and user emails. For example, “travis@” is the CEO, but “tonysigcafe@” sounds like a business, and “thetaylors@” sounds like a family.
That’s a security red flag by itself, because it makes phishing much easier. (Sign up for an account with an official-sounding name, then contact users from that email, which will pass automated and visual checks because the domain is the same.)
You write that “this incorrect configuration was put in place by a former employee and never caught.” Is it possible that the former employee was disgruntled and left a logic bomb along with his resignation? Or are the company’s vulnerability scans just checkbox items that are never really used to find vulnerabilities?
Fair point on change management controls failing, but remember that Internal Audit is at best a detective control and is not tasked with live monitoring of all processes.
Wow, I’ve seen some wild stuff in my nearly 25 years in IT, but this is pretty bad. Sounds like the staff at US Internet may not be all that experienced and likely have minimal security. The scary part is I have personally worked at many small to medium-large sized companies (wide range roughly 30 to 3000 employees) where the technical staff were basically winging it and they did not have separate security, compliance, or audit teams/staff.
Using Ansible is usually a good thing in regard to automation and standardization, but absolutely dangerous when you hand the keys over to inadequate staff who leave default, or test configurations in place combined with little to no oversight and poor company technical & security practices. Hopefully this is a wake up call and they get their stuff together.
If they’re even halfway serious about hiring BK for consulting off the cuff, that says a lot!
This is what most small businesses look like in terms of security focus : Ex post facto.
Whether Ockham’s incompetence initially or malice, a postmortem should eventually inform
but if actors abused this level of access (decades?) merely to click-inflate I’d be surprised.
Huge mistake
Re “The timestamps listed do not appear to be accurate somehow”
Is it that the timestamps are showing the time the folder was created, not the time of the contents.
I don’t think so because in a lot of cases where you drill down into the actual inboxes below and you find much more recent timestamps. But perhaps you’re right for the main index page. Honestly, their systems were so screwed up it’s hard to explain a lot about this.
I can confirm that the date is the time created. As a former employee of a company that maintained the Securence partnership, the date matches when we added the managed domain to Securence.
I disagree. I am an MSP and many of my customers and former customers are in this list. The date does not line up with when these customers were put on the platform.
Folders can be recreated for a number of reasons. I wouldn’t exclude the possibility this is still the date created for the folder
I would agree to this. This is the date the folder was created. The emails and subfolders would have the date they were created. So some of these URLs date back to 2008. Simply wow is all I got.
Every single time I read about one of these “massive internet security fails” any longer, that is, maybe over the last ten years or more, I am amazed these supposedly solid professionals seem incapable of learning just how much effort the bad guys are expending to ruin their security precautions.
Ignorance, Incompetence, Arrogance, Parsimony or just plain Stupidity?
What about notification to CISA or similar? https://www.cisa.gov/report
So many companies are failing at their core competency, experion, Boeing… are we so complicated that this is what we should expect?
Or could they be so focused on their stock performance that they forget what is important.
Speaking of stock performance: I did some limited checking and could not determine if they are publicly held. If they are it will be interesting to read about it in their SEC disclosure.
The entire world is compromised. I don’t care. Let everyone know what everyone else is doing.
That’s just simplistic.
USI is a relatively small, privately held company.
Speaking of stock performance: I did some limited checking and could not determine if they are publicly held. If they are it will be interesting to read about it in their SEC disclosure.
Did it have the actual emails of their customers or just the US Internet emails?
Both
Does this sound like it just affected their email/IMAP customers? They also have Business Continuity and Shadowing in their Securence spam filter. So that technically holds email. Is that list still available somewhere to check for compromised domains?
Here is a list of the customers I found with the original link Holden shared. However, this list should not be considered exhaustive or complete by any means. Changing the URL slightly by incrementing or decrementing a number in the URL caused a slightly different set of customers to be listed. I didn’t have a chance to go through it all before they took it down (note to self: pillage BEFORE burning).
https://docs.google.com/spreadsheets/d/1wgKe1VrfNF8Afav1aJtMrZDeNuqBtSFku58fWlCBp6Q/edit?usp=sharing
I know of a few of the domains on that list and they are customers of their “secure” email gateway service
There are a number of law firms on your list there that will not be happy. First one: https://www.2027law.com/ “BOUTIQUE LAW FIRM Specializing in Commercial Litigation, Business Law, and Real Estate”.
…and medical. Who else has read the emails from minneapolisplasticsurgery.com ?
Thank you!!! I was able to confirm that at least two of my clients were in there. It doesn’t look like my most critical ones were though.
Gosh I wonder what could possibly be found in cleanup-fridays3.sh…
This sets up a fascinating reporting issue. If you have had email interactions with law firms, medical etc. on this list that contains PII, PHI, do you have reporting requirement to the clients? It was not your breach, but it would indicate loss of the data.
Only if you know about it. Don’t read the list if you’re afraid of what you’ll find…
This is very disappointing for me personally, as USI was my very first ISP back in like 1995, and I currently use their fiber to the home service. At least I didn’t use their email services.
Great job Brian, and anyone aiming to try to essentially bribe you for nondisclosure must really have NO IDEA of your already stellar record of instantiating top journalistic ethics, as well as having to deal with some of the “worst of the worst” over the years, that’s for sure!
All of which they could discover at least something about just by “googling Brian Krebs on Security” –
ironically enough.
Maybe they should leverage AI, ask copilot “how can ansible playbooks become a security vulnerability?”
How much of their staff was outsourced or H1B labor? I’m guessing most of it. You get what you pay for, folks.
Some people might find your comment racist, or some other “-ist”, but I’ve worked in IT since 1999 and I have to agree. “You pay peanuts, you get monkeys”. The whole point of H1B is to pay peanuts. And as a bonus, you get people who can’t easily quit. It’s not a recipe for getting well qualified, hard-working, security-minded IT staff.
An exemplary case of F*** UP BIG, Summa cum Laude. Thanks for the great reporting, Brian, and for the laugh on reading C Suite bovine excreta-speak. Your gut feeling is probably correct, IMHO…
Yow, really bad. I’m a customer of USI and I’ve gotta say they’re generally one of the greatest companies I’ve ever dealt with. Really bad to get this much egg on their face.
As for asking about your consulting – I don’t think that’s necessarily a red flag, they may just be trying to figure out who is reporting this vulnerability. I highly highly HIGHLY doubt they were trying to pay you off.
Yeah, I never come up to bat for corporations typically, but USI has been by FAR the best ISP I have ever dealt with. On top of that, at least in my experience, their CEO has always been transparent and responsive to customers. Sucks to see this incompetence, but I am certain that he was not acting in bad faith with that query.
With basic usernames like that. These comments sound like AI generated replies to make the company seem more reputable than it is.
US Internet is the most beloved ISP in Minneapolis. Living in their service area is a dream. Travis Carter replies to questions on reddit. This of course is bad and they should probably publicly respond, but USI is probably the only ISP I will ever like.
Doubtful. USI offers a great FTTH product and their customers are extremely loyal.
Yep. These are all AI generated.
US Internet is so great that I was able to download the hack list in a giffy. It does not get any better than this!
test
I can confirm. I live in Timbuktu and US Internet’s service is absolutely flawless. They are simply eggless.
“The feedback from my team was a issue with the Ansible playbook that controls the Nginx configuration for our IMAP servers”
No. Just, no.
Just reading through the Nginx docs I can confirm that Nginx as a proxy for an IMAP service wouldn’t show an index unless you just left the default server enabled. ‘
But the root directory for the default server is usually either a basic example web page in a mostly empty directory like /var/www/html or it is an empty directory.
So this might only happen if your ansible playbook was also configuring the default server’s root directory. In which case, why would it do that? And more importantly, in any case why the hell is the root directory set to a directory where email boxes are stored?
It’s likely their email system uses Maildir storage (like Qmail and CourierMTA), and the Maildir root got published to the webserver (I managed Qmail and Courier mail servers for a couple of decades).
The Maildirs are never accessed directly by users, but through IMAP, POP, Webmail, or a local MUA. The only reason to access the maildirs directly are usually for troubleshooting or discovery (I may have used grep against a Maildir a few times on a records request) and no reason to ever publish the Maildirs to be user accessible over web, NFS, SMB or anything else. I’m guessing this was done as a troubleshooting or dev tool and wasn’t meant to be world visible. It would be interesting to find out how long it was visible.
I haven’t looked at USI email servers to see if they force TLS/SSL, but I’m going to guess most email was transported in plan text SMTP, so USI could claim it’s not a secure email service.
Ha.. Maybe some “smart” person at the company used this as a quick way to debug customer issues with their emails.
It is using port :81 after all (not standard 80 or 443), and we all know that makes it secret 😉
As TS mentioned, you would have to purposely configure nginx to show that list.
Yes. This is also what I was thinking. A troubleshooting hack that was never removed.
How difficult should it have been for this company to find this on their own? Assuming they actually wanted to discover and fix it, what would it take to do so? Would discovering this misconfiguration (assuming that’s what it was), have been easy if standard security audits were conducted or would it have required a highly detailed, involved, and expensive red-team program?
Trivial, if they’d run any basic network audit. Find open ports (especially those visible to the internet), trace them back to the services running them, determine what they’re doing, what they’re used for, whether they should be running, secured or unsecured.
Doubtful. Doesn’t take somebody well versed in infosec to run netstat or lsof to find what’s listening. It’d just take an admin running one of those, noticing Nginx listening on a non-standard port, and going from there. Like, if I saw that on one of my customer’s servers, I’d immediately be asking questions about its purpose. This is also why firewalls are important. If this was set up for troubleshooting, as others have suggested, there’s no reason why this port should be publicly open. Create an allow rule for the IP doing the troubleshooting, and then trash it all once finished.
Interesting find, but I’m somewhat surprised and disappointed you published a screenshot of the actual email addresses of victims. Just seems wrong for someone who I thought was concerned with privacy and security.
The email addresses BK showed are for the ISP itself, not their customers…
Looking at some of the email addresses listed, not all look like employee email addresses. E.g., what employee would be using “the___family@” for an email account?
Some look like email addresses for customers who were using the USIWireless WISP service.
An email address is not private information.
I agree, very disappointed that you published the list, it was not necessary in your reporting of their failure.
Do you have deeper screenshots? Is this like an exposed Maildir store?
What happens when you don’t understand how much you really don’t know.
Usinternet.com/privacy-policy/ is a joke.
Email CEO and let him know how you feel. TRAVIS@USIWIRELESS.COM
Those douchebags need their balls nailed to a wall, in an isolation cell in hell! We NEED A CLASS ACTION LAWSUIT! These are the morons who for YEAAARS HAVE CONTROLLED ALL FORMS OF OUTSIDE COMMUNICATIONS FOR ALMOST EVERY SINGLE JAIL AND PRISON IN MINNESOTA! OF COURSE THEY WILL GET AWAY WITH IT, SMH,
Brian,
Do you happen to know if Hold Security 100% confirmed that these unsecured links were already in the hands of criminals? (So, an example of that would be Hold Security spotting these links on the dark web, or something like that.) – OR – did they instead find this during their own testing of USInternet’s services, where it then MIGHT not have gotten into the hands of criminals? (yes, I know, it’s prudent to assume that criminals did find this).
Well, this is disappointing to learn as a current customer of their Securence email security gateway and continuity service.
I think more disappointing is that it’s now the 15th and I have yet to see any email from USI or Securence acknowledging this security breach has occurred. Wondering if I’ll see any mention from them before next week.
Brian, it looks like some of those “employee” email addresses do not appear to be so. Some look like they could be email addresses for subscribers to USI’s wireless ISP service. It might be worth reviewing the published image out of an abundance of caution to not further exacerbate the fallout for victims of this security breach/lapse.
How can you find out if your company’s emails were leaked?
We are one of their customers for SPAM filtering, and we got a canned response to our inquiry.
“Below is the official statement from USI on the issue.
On February 8th, 2024, we discovered a vulnerability in our hosted “pop/imap/continuity” environment. Core Securence, Archive, and Exchange were not affected. As a result of a configuration error, some email addresses, domain names, and emails were publicly accessible.
Immediately upon discovering the vulnerability, we resolved the issue and initiated a comprehensive investigation to determine the cause and extent of the unauthorized access. We have since confirmed that the misconfiguration was due to human error. To address the issue, we have taken the following steps:
– Implemented additional security protocols and controls to validate all port configurations and prevent unintended access
– Reviewed our security practices and policies to identify and address any vulnerabilities
– We are reviewing all log data and informing the affected individuals
USI is committed to safeguarding our customers’ data with the highest level of security measures, including firewalls, intrusion detection systems, and encryption technologies. Our security team continuously monitors our systems for suspicious activity and implements new security measures to stay ahead of evolving threats.
We understand that this incident may have caused concern, and we deeply regret any inconvenience or worry it may have caused. We assure you we are doing everything possible to protect your privacy and keep your data safe. We value the trust you place in us and are committed to earning and maintaining that trust daily. If you have any questions or concerns, please do not hesitate to contact our customer support team. We are here to help and support you in any way we can.”
I’m disappointed that you had to reach out to them to get that response. I’m now wondering if I’ll receive any communication from them regarding this matter if I don’t proactively reach out to them myself.
If nothing else they should get an earful for not proactively reporting this to customers. Having to find out from a media outlet is bad form, IMO.
Don’t forget the FBI gave me back all my files including like 800,000 Social Security numbers I found on Google that no one will cover
Discovering this is disheartening as someone who currently uses their Securence email security gateway and continuity service.
This sets up a fascinating reporting issue. If you have had email interactions with law firms, medical etc. on this list that contains PII, PHI, do you have reporting requirement to the clients? It was not your breach, but it would indicate loss of the data.
Also note that USI runs the City of Minneapolis wireless network, and collects credit card numbers for people to use it..
https://www.minneapolismn.gov/government/programs-initiatives/digital-inclusion/#d.en.143832
https://www.minneapolismn.gov/government/programs-initiatives/digital-inclusion/internet-services/