Email service provider Sendgrid is grappling with an unusually large number of customer accounts whose passwords have been cracked, sold to spammers, and abused for sending phishing and email malware attacks. Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers, but that solution may not come fast enough for organizations having trouble dealing with the fallout in the meantime.
Image: Wikipedia
Many companies use Sendgrid to communicate with their customers via email, or else pay marketing firms to do that on their behalf using Sendgrid’s systems. Sendgrid takes steps to validate that new customers are legitimate businesses, and that emails sent through its platform carry the proper digital signatures that other companies can use to validate that the messages have been authorized by its customers.
But this also means when a Sendgrid customer account gets hacked and used to send malware or phishing scams, the threat is particularly acute because a large number of organizations allow email from Sendgrid’s systems to sail through their spam-filtering systems.
To make matters worse, links included in emails sent through Sendgrid are obfuscated (mainly for tracking deliverability and other metrics), so it is not immediately clear to recipients where on the Internet they will be taken when they click.
Dealing with compromised customer accounts is a constant challenge for any organization doing business online today, and certainly Sendgrid is not the only email marketing platform dealing with this problem. But according to multiple emails from readers, recent threads on several anti-spam discussion lists, and interviews with people in the anti-spam community, over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.
Rob McEwen is CEO of Invaluement.com, an anti-spam firm whose data on junk email trends are used to improve the spam-blocking technologies deployed by several Fortune 100 companies. McEwen said no other email service provider has come close to generating the volume of spam that’s been emanating from Sendgrid accounts lately.
“As far as the nasty criminal phishes and viruses, I think there’s not even a close second in terms of how bad it’s been with Sendgrid over the past few months,” he said.
Trying to filter out bad emails coming from a major email provider that so many legitimate companies rely upon to reach their customers can be a dicey business. If you filter the emails too aggressively you end up with an unacceptable number of “false positives,” i.e., benign or even desirable emails that get flagged as spam and sent to the junk folder or blocked altogether.
But McEwen said the incidence of malicious spam coming from Sendgrid has gotten so bad that he recently launched a new anti-spam block list specifically to filter out email from Sendgrid accounts that have been known to be blasting large volumes of junk or malicious email.
“Before I implemented this in my own filtering system a week ago, I was getting three to four phone calls or stern emails a week from angry customers wondering why these malicious emails were getting through to their inboxes,” McEwen said. “And I just am not seeing anything this egregious in terms of viruses and spams from the other email service providers.”
In an interview with KrebsOnSecurity, Sendgrid parent firm Twilio acknowledged the company had recently seen an increase in compromised customer accounts being abused for spam. While Sendgrid does allow customers to use multi-factor authentication (also known as two-factor authentication or 2FA), this protection is not mandatory.
But Twilio Chief Security Officer Steve Pugh said the company is working on changes that would require customers to use some form of 2FA in addition to usernames and passwords.
“Twilio believes that requiring 2FA for customer accounts is the right thing to do, and we’re working towards that end,” Pugh said. “2FA has proven to be a powerful tool in securing communications channels. This is part of the reason we acquired Authy and created a line of account security products and services. Twilio, like other platforms, is forming a plan on how to better secure our customers’ accounts through native technologies such as Authy and additional account level controls to mitigate known attack vectors.”
Requiring customers to use some form of 2FA would go a long way toward neutralizing the underground market for compromised Sendgrid accounts, which are sold by a variety of cybercriminals who specialize in gaining access to accounts by targeting users who re-use the same passwords across multiple websites.
One such individual, who goes by the handle “Kromatix” on several forums, is currently selling access to more than 400 compromised Sendgrid user accounts. The pricing attached to each account is based on volume of email it can send in a given month. Accounts that can send up to 40,000 emails a month go for $15, whereas those capable of blasting 10 million missives a month sell for $400.
“I have a large supply of cracked Sendgrid accounts that can be used to generate an API key which you can then plug into your mailer of choice and send massive amounts of emails with ensured delivery,” Kromatix wrote in an Aug. 23 sales thread. “Sendgrid servers maintain a very good reputation with [email service providers] so your content becomes much more likely to get into the inbox so long as your setup is correct.”
Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.
“Single-factor authentication for a company like this in 2020 is just ludicrous given the potential damage and malicious content we’re seeing,” Schwartzman said.
“I understand that it’s a task to invoke 2FA, and given the volume of customers Sendgrid has that’s something to consider because there’s going to be a lot of customer overhead involved,” he continued. “But it’s not like your bank, social media account, email and plenty of other places online don’t already insist on it.”
Schwartzman said if Twilio doesn’t act quickly enough to fix the problem on its end, the major email providers of the world (think Google, Microsoft and Apple) — and their various machine-learning anti-spam algorithms — may do it for them.
“There is a tipping point after which receiving firms start to lose patience and start to more aggressively filter this stuff,” he said. “If seeing a Sendgrid email according to machine learning becomes a sign of abuse, trust me the machines will make the decisions even if the people don’t.”
SendGrid was already the source of spam, happily authenticated via DMARC.
Many IT people would disagree with “Sendgrid takes steps to validate that new customers are legitimate businesses”
I completely agree with you and disagree with the statement that Sendgrid cares if it’s a real company or not. They are greedy and only care about money.
I agree with you also. Sendgrid has been the worst single source of spam for at least 1 year.
I have sent them logs, headers, I have called and complained to Twilio.
On their web pages they don’t offer a way to report them.
We gave them notice we would be blacklisting their IP ranges – which we have several of them, the entire subnet. We told our vendors that use them to use another source to contact us.
Sendgrid, it’s to little to late, and you are only addressing it due to your paying customers, with no regard to the recipients… You are Junk mail.
Right. Sendgrid is and always has been a spam factory.
Agreed. I’ve had to migrate 2 clients away from sendgrid recently because sendgrid had several of their IP addresses on RBL/SBL blacklists and email campaigns were being blocked by recipient antispam. Their response was “oh, you should just white list our IP addresses”. How about no.
I blocked SendGrid’s spam cannons years ago – Zero complaints.
If SendGrid was serious about security, it would offer some form of SSO too. Its the only b2b app I know that doesn’t support Saml (most products charge more but at least offer it); so companies (even massive ones) have to use username password. Security teams have no ability to control these passwords nor can they require 2fa.
I blocked sendgrid 2 years ago, but was forced to unblock because so many companies use it. This has been an issue much longer than just a few months. At least 80% of phishing email against us comes from sendgrid servers.
It’s easy to permit certain lists. Look for the u-number in the from/ehlo etc.
Block or spam folder the rest.
Just got done trying to remediate an issue with one of our vendors who uses Sendgrid. That vendor recently implemented a “new security” feature that requires users to respond to an email validation that is sent by – you guessed it – Sendgrid. Our email filtering was rejecting those emails, so the “new security” process of that vendor could never successfully complete. End result – users could no longer login to that vendor’s website. Ugghh! The workaround was to whitelist an IP address at Sendgrid. Now, it looks like we need to back that out whitelist entry out. No idea where to go from here.
You could suggest to the vendor that they dump Sendgrid.
Does your filtering system have enough control to whitelist based on the static part(s) of the subject for those login email messages?
There’s a lot less margin for phishing activity in messages with a subject of “Your authentication code for websitexyz”
Take a look at the email headers. There is a header that includes a customer ID, so you can whitelist just that send grid customer, as long as your solution allows it(you need a wildcard in my experience).
Thank you for this write up! We see a targeted phishing email to executives across our organization on pretty much a daily basis. Half of my reports to abuse@sendgrid.com actually generate a Twillio Support case, the other presumably go into the bitbucket.
Like others, we can’t wholesale block SendGrid.net due to other business partner use.
We continually receive phishing attacks from SendGrid senders. It’s the single largest source and a perpetual pain because there are also legitimate vendors utilizing its services. I’ll stop short of calling out those vendors here in a comment but, needless to say I am pleading with them to ‘see the light’ and reverse their plans to onboard with them.
We noticed these things starting up back in February but there was no recourse of action against sendgrid and vendors that used them didn’t care to push it.
As an aside, Sendgrid allows unverified, free accounts to send alot of email. And since it uses the same servers as legit email, this becomes an issue.
As a small business, we use a paid SendGrid account that only offers shared IPs for sending. Our emails to clients are being blocked because the IPs are finding their way into Spamhaus and others.
When I complained, I got a canned response that was nothing more than a pitch to upgrade to an account with a dedicated IP.
We noticed this happening over the past few weeks. Massive amounts of inbound fake Voicemail messages / fake Office 365 password reset messages. At first, didn’t notice the common thread between all of them but soon realized they were all coming in with sendgrid.net header and links in the messages. We are now hard blocking anything with sendgrid.net in the email headers as well as hard blocking any URL coming through our firewall with sendgrid.net as the domain. We are not messing around with this stuff anymore and it’s inexcusable for a company of their size to have let this happen.
We blocked Sendgrid at the edge a couple of years back. As others have said, no complaints from staff. We have had very few manual releases of real email from Sendgrid email servers during that time frame. From our perspective, Sendgrid is a spam-sending monster.
Sendgrid’s parent company, Twilio, is also the largest source of SMS spam, and again, it is harder to filter because they also have legitimate users. However, if you receive an unsolicited text message pushing diet pills, payday loans, free gift cards, or magic skin cream, chances are it came from Twilio.
As a company they are running on startup economics. They are losing tens of millions of dollars a quarter, so the only thing they have to show the investors is rapid growth, and the only way they can get that is by turning a blind eye to spammers.
(Views expressed are my own and not those of my clients.)
Yeah, I was wondering what was going on. SendGrid sounded like a very dense company to deal with. For instance, about a week ago I got a phish with a shortened url coming from SendGrid servers. So I forwarded it to abuse@sendgrid.com and it bounced back to me as “rejected”.
I was laughing hard at it.
This is what basically happens when you get greedy and sell too many accounts to too many unscrupulous customers. I don’t think it’s just credential stuffing like they claim.
The bottom line, if you need to do a legit mailing list, stay away from SendGrid. We personally went with Amazon SES, it requires a bit more setup, but is way better and reputable and is less costly at the end.
Is this a case of a company looking the other way during the pandemic in order to make money?
Judging by some of the comments that may not be true in this case since it seems like it has been going on for too long. Companies like this need to have their upper management fined and imprisoned in order to teach them a lesson otherwise the financial incentive is too good and ethics too poor.
I have noticed some companies have lowered standards in order to try and make some money during the lean advertising times since March.
I’ve gotten several spam SMS messages from yahoo email addresses (maybe spoofed) and I hadn’t seen that in a long time. I’m assuming someone I gave my number to recently has been either selling the phone numbers or more likely have had their system compromised.
At a previous company, we outsourced our email because we didn’t want to deal with all the headaches, and the VP wanted to cut costs to fluff his bonus. They got hacked and we ended up on all the RBLs. I think we had to sue them to get them to implement better security.
often something goes via email spam .. so at office 365 .. even Mircosoft itself warned a long time ago
Our researchers also noticed attacks impersonating Sendgrid login pages.
Here what we found .
https://ironscales.com/blog/Fake-Login-Attack-Spoof-Two-Leading-Email-Delivery-Service-Providers/
“Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers.”
If my calendar is correct, it’s the end of August 2020. And they are now just WORKING on a plan for multi-factor authentication? That speaks volumes right there.
I’m no fan of blaming the victim, but maybe just in this case?
I don’t know if this is somehow connected to this security breach, but out of the blue, Amazon suddenly demanded that I change my password after using the same password for the past 15+ years (I know I know I deserve to have my laptop and internet connection taken away from me for being so stupid).
Not only did Amazon demand that I reset my password, but the system insisted on sending me both a text message and an email with a temporary security code before my new password was accepted. Weird.
I’m using SendGrid for most of my customers for WP sites. What alternative would you suggest to non-advanced user? Mailgun?
Yeah Mailgun is good, and they have a decent free tier.
I’m a very light user but Postmark (postmarkapp.com) has worked very well for me. I’m still using the free trial credits provided.
Sendgrid has become an increasing source of abuse for me over the last year, but I do a lot of contact management (e.g., disposable email addresses), so I’m not at all surprised that other people have had problems with them for many years.
I’m also suspicious about the self-reported “hacked” angle. How many times have we heard that same story when someone does something awful online and tries to shirk responsibility? I find it far more likely that someone inside Sendgrid (or their client companies) is trying to make some extra cash on the side. If it were legitimate cracking going on, I’d expect to hear more about what Sendgrid has actually done to tighten up their security, not this “Oh, maybe we’ll get around to requiring 2FA some day” nonsense.
Honestly, at this point, I don’t see any mass senders as legitimate unless they are offering a cash bounty for anyone who reports spam coming from their systems. I gave up on reporting abuse to the abusers a long time ago because, at best, all I got back was a “we can’t respond to all complaints, but we’ll take action that *we* think is appropriate” boilerplate. Decades of anti-spam rhetoric like that hasn’t accomplished anything. The day I can collect even $1 in compensation for attacks on my servers is the day I’ll finally believe that being abusive isn’t part of the business model for these companies.
I’m sendgrid’s customer, using few services per month.
Last month my account have been hacked, cause a lot of charge to my credit card.
I’ve contacted sendgrid staff but I have no satisfied answer or responsible so far.
So, any alternatives to SendGrid? We use it, with MFA enabled, without any issue. But I foresee the day where I’ll need to switch to another service.
Particularly, one that works within the Microsoft Azure ecosphere.
Depends on your features but Postmark is rock solid for me. Also, Mailjet
/agree Postmark is the best for transactional mail.
Yup. Same here. A few months ago. Sendgrid account hacked, someone set up a campaign and sent a huge amount of spam mail. Big bill on my credit card. Contacted Sendgrid, they blamed it on me. Cancelled my account, never going back. Screw them.
My companys Sendgrid account got hacked yesterday or the day before, and sending out fake paypal emails. No replies from Sendgrid support. Enabled 2-factor login and deleted the API-key and created a new one.
But I still get no-replys from Sendgrind with the fakeemails being sent back to me about emails not being delivered, full on fake paypal ones in my name.
Shouldnt 2-factor login and new API-key stop these mails? If anyone has any other idea what to do, please let me know.
Lets help these guys do better or go out of business Money (customers) talk. I’ve seen these as alternatives:
mailjet
postmark
mailgun
what others or which of those has the best deliverability reputation with gmail in particular. For small quantity occasional mailouts (<3k), non marketing, subscribed, clean maillist.
Postmark only very recently started allowing any bulk mail.
Sendgrid needs to:
1. Require senders domains to have TXT records to authenticate them, and
2. Parse email blasts to see if third party domains are linked within the email messages.
I’ve reviewed about 10 phishing emails sent by Sendgrid. They have impersonated website hosting companies and domain registrars to hijack domain names using obfuscated links.
Sendgrid has been the largest source of very targeted phishing for well over a year.
Messages sent to their abuse department have never been answered.
I block their systems at every email platform that I can.
Most Ivy league alumni associations unfortunately use Sendgrid.
These people are the problem. They’re not victims, they’re a wolf in sheep’s clothing.
Important to note is that many legitimate companies using Sendgrid are affected massively without their accounts being compromised. With Spamhaus and others listing Sendgrid IP addresses (from 2020-08-17 onwards), our email delivery dropped to 50% due to spam filters. This will force many to switch to a different email provider.
Determined attackers can see from their target dns records that they use SG, and their admins probably set their spam filter to whitelist the SG network entirely, so almost any phish\malware from a compromised SG account will get through.