Email service provider Sendgrid is grappling with an unusually large number of customer accounts whose passwords have been cracked, sold to spammers, and abused for sending phishing and email malware attacks. Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers, but that solution may not come fast enough for organizations having trouble dealing with the fallout in the meantime.
Image: Wikipedia
Many companies use Sendgrid to communicate with their customers via email, or else pay marketing firms to do that on their behalf using Sendgrid’s systems. Sendgrid takes steps to validate that new customers are legitimate businesses, and that emails sent through its platform carry the proper digital signatures that other companies can use to validate that the messages have been authorized by its customers.
But this also means when a Sendgrid customer account gets hacked and used to send malware or phishing scams, the threat is particularly acute because a large number of organizations allow email from Sendgrid’s systems to sail through their spam-filtering systems.
To make matters worse, links included in emails sent through Sendgrid are obfuscated (mainly for tracking deliverability and other metrics), so it is not immediately clear to recipients where on the Internet they will be taken when they click.
Dealing with compromised customer accounts is a constant challenge for any organization doing business online today, and certainly Sendgrid is not the only email marketing platform dealing with this problem. But according to multiple emails from readers, recent threads on several anti-spam discussion lists, and interviews with people in the anti-spam community, over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.
Rob McEwen is CEO of Invaluement.com, an anti-spam firm whose data on junk email trends are used to improve the spam-blocking technologies deployed by several Fortune 100 companies. McEwen said no other email service provider has come close to generating the volume of spam that’s been emanating from Sendgrid accounts lately.
“As far as the nasty criminal phishes and viruses, I think there’s not even a close second in terms of how bad it’s been with Sendgrid over the past few months,” he said.
Trying to filter out bad emails coming from a major email provider that so many legitimate companies rely upon to reach their customers can be a dicey business. If you filter the emails too aggressively you end up with an unacceptable number of “false positives,” i.e., benign or even desirable emails that get flagged as spam and sent to the junk folder or blocked altogether.
But McEwen said the incidence of malicious spam coming from Sendgrid has gotten so bad that he recently launched a new anti-spam block list specifically to filter out email from Sendgrid accounts that have been known to be blasting large volumes of junk or malicious email.
“Before I implemented this in my own filtering system a week ago, I was getting three to four phone calls or stern emails a week from angry customers wondering why these malicious emails were getting through to their inboxes,” McEwen said. “And I just am not seeing anything this egregious in terms of viruses and spams from the other email service providers.”
In an interview with KrebsOnSecurity, Sendgrid parent firm Twilio acknowledged the company had recently seen an increase in compromised customer accounts being abused for spam. While Sendgrid does allow customers to use multi-factor authentication (also known as two-factor authentication or 2FA), this protection is not mandatory.
But Twilio Chief Security Officer Steve Pugh said the company is working on changes that would require customers to use some form of 2FA in addition to usernames and passwords.
“Twilio believes that requiring 2FA for customer accounts is the right thing to do, and we’re working towards that end,” Pugh said. “2FA has proven to be a powerful tool in securing communications channels. This is part of the reason we acquired Authy and created a line of account security products and services. Twilio, like other platforms, is forming a plan on how to better secure our customers’ accounts through native technologies such as Authy and additional account level controls to mitigate known attack vectors.”
Requiring customers to use some form of 2FA would go a long way toward neutralizing the underground market for compromised Sendgrid accounts, which are sold by a variety of cybercriminals who specialize in gaining access to accounts by targeting users who re-use the same passwords across multiple websites.
One such individual, who goes by the handle “Kromatix” on several forums, is currently selling access to more than 400 compromised Sendgrid user accounts. The pricing attached to each account is based on volume of email it can send in a given month. Accounts that can send up to 40,000 emails a month go for $15, whereas those capable of blasting 10 million missives a month sell for $400.
“I have a large supply of cracked Sendgrid accounts that can be used to generate an API key which you can then plug into your mailer of choice and send massive amounts of emails with ensured delivery,” Kromatix wrote in an Aug. 23 sales thread. “Sendgrid servers maintain a very good reputation with [email service providers] so your content becomes much more likely to get into the inbox so long as your setup is correct.”
Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.
“Single-factor authentication for a company like this in 2020 is just ludicrous given the potential damage and malicious content we’re seeing,” Schwartzman said.
“I understand that it’s a task to invoke 2FA, and given the volume of customers Sendgrid has that’s something to consider because there’s going to be a lot of customer overhead involved,” he continued. “But it’s not like your bank, social media account, email and plenty of other places online don’t already insist on it.”
Schwartzman said if Twilio doesn’t act quickly enough to fix the problem on its end, the major email providers of the world (think Google, Microsoft and Apple) — and their various machine-learning anti-spam algorithms — may do it for them.
“There is a tipping point after which receiving firms start to lose patience and start to more aggressively filter this stuff,” he said. “If seeing a Sendgrid email according to machine learning becomes a sign of abuse, trust me the machines will make the decisions even if the people don’t.”
It’s easy for Sendgrid to blame hackers. The real guilty here is themselves not filtering at all those mails. Most of those spam/phishes and even virus would be rather easy to block before it even leaves the servers. But hey, I guess they are too afraid to loose one or two legitimate customer by not letting an e-mail wave out.
SendGrid allowed their “users” to send our company massive amounts of criminal spam and attack emails. I reached out to SendGrid and they refused to do anything about it on several occasions. It took a criminal complaint against SendGrid for them to block their “users” from emailing our domain. They never did anything about the fraud, spam, or other crimes their service is being used for. Some of the emails were sent to fake honeypot addresses used to detect this sort of malicious activity. Some emails claimed to be Wells Fargo, Amazon, Microsoft, etc… claiming there is an issue with the account or the password is expiring and needed entered to prevent expiration. Most imitated large companies and many that we have no account with.
Also, SendGrid did not just start these criminal email attacks or spamming. SendGrid has taken part in this activity for well over a year. I have email records and a criminal complaint that can prove they have been “allowing” spoofed attacks from places like banks, email companies, Amazon, Microsoft, etc. Another company that takes part in this criminal behavior is ConstantContact. Our company had to do the same with them to get them to stop. Both companies refused to look into the criminal activity being conducted on their servers and it took criminal complaints to get them to stop spamming our company. I honesty think they are conducting these crimes and then using the “it was out users, not us… we’re a paid email service” excuse to get away with it. And the US gov is just going to be like “okay” and let them continue it… as usual. The FTC doesn’t even allow this type of crime to be reported to them anymore. And the stuff the FTC lets you report, they do not investigate. Like the criminals running Paper Street Media out of Florida.
I just found out that there’s a Sendgrid account using my email address. I requested a password change, logged in, and there’s no activity. I can’t tell when the account was created or what its been used for. This is super creepy.
I also noticed this trend a couple months ago. Attempts to report the problem to abuse@sendgrid.net didn’t work.
Try abuse@sendgrid.com. It is apparently read by more people who can actually do something about reproted spam, especially malware or phish. (No, I have no idea why Sendgrid doesn’t simply make abuse@sendgrid.com the same as abuse@sendgrid.net, but I don’t work there.)
I blocked their entire IP space as I only received spam from Sendgrid. There are several companies which use Sendgrid to Spam, and Sendgrid never does anything about it.
I modified my client_info file to reject all email connections with the message
“Twilo/SendGrid: You have been blocked for network abuse: Spamming Contact Twilo/SendGrid Technical support at (855) 910-8712 to resolve issue”
Let Sendgrid’s spamming customers and Sendgrid technical support waste each other’s time hashing out the cause of the message instead of mine in dealing with problems they created.
^^^^ This is the ideal response.
SendGrid did not adopt 2FA because it cost them money. SendGrid does not addressed abused accounts because it cost other people money. Until SendGrid experiences a clear, direct effect on their own cost structure they will not change.
The funny thing is that it won’t cost any notable amount of money if they were eager to do it. You don’t need to acquire Authy etc to implement 2FA
Someone should tell them about Google Authenticator ))) it’s literally few hours of coding, updating etc to implement 2FA. I wonder what these guys are smoking LOL
Sendgrid DOES NOT CARE about using it. They make their money providing spam support services as far as I am concerned. If it weren’t for the fact that I would be the one getting penalized, I would have long ago shut down Sendgrid.
I agree. I try to report spammer and many operators take care of the problem, but after a few complaint and spam cycles and abuse complaints going nowhere, it become time to shift the cost to the offending ISP. When an ISP supports spammers, block I the entire ip space with messages telling the sender to contact the ISP tech support line to resolve issue. My goal is to generate as many expensive tech support calls as possible in order to shift the cost of spam back to where it belongs. I am thinking about making a peer to peer blacklist in order to shift more of the costs to spammers.
Hi. This is very interesting and a little disheartening. Anyone else use systems like Atlassian (https://confluence.atlassian.com/cloud/atlassian-cloud-ip-ranges-and-domains-744721662.html) or others that require you to add SendGrid IP addresses to the allow list so their notification messages do come through? We have had 3 [reported] cases of spam/spoofing coming into our organisation because of quite a few IPs added to our incoming allow list, opening the gate for the bad guys like in this case … Thoughts?
Was it spearphishing or was it spam though?
I’m a SendGrid customer–as of yesterday my emails don’t get delivered to Hotmail any more. I use 2FA, so I presume my account didn’t get compromised.
I think I need to move on and migrate to a more credible email service provider. Any suggestions? I’m done with this SendGrid cluster.
We switched to Postmark. So much better than Sendgrid in every possible way.
I also have an email account from the early days of GMail (when an invite was required to get the account). Mine is not “OG”, but i have had a similar experience with other people being apparently able to sign up for an account with the same name – but adding a period or some other character in between words.
GMail documentation states that this is not possible, but perhaps there is something about those early accounts?
I’ve also received an enormous amount of sensitive information from these people. I’ve been very responsible with that information, and always try to inform the sender of their error.
My big fear is that some nefarious action by someone else, using my email as identity, is then attributed to me.
this was intended for https://krebsonsecurity.com/2020/09/the-joys-of-owning-an-og-email-account/#more-45506
Haven’t accepted an email from sendgrid on my servers in 8 years … Can’t believe that suddenly it’s news that they’ve been spamming…..
RIGHT?
My company has been upstream blocking email traffic from sendgrid and internally blocked DNS resolutions of sendgrid domains since before last christmas. That was the only way to deal with all of the phishing the issue after trying to contact them. I went to pretty extraordinary lengths to contact their abuse team in December. With a bit of social engineering I got the Twilio sales team to put me in contact with Sendgrid Abuse team who promptly pretended to do be interested in addressing the issue and then ultimately did nothing.
I would be pretty ashamed to work on the abuse team there. Kind of makes you curious about the turnover rate on the abuse team.
SendGrid does not care. The number on Spamhaus Block List keeps increasing every day. See twitter thread …
https://twitter.com/SamSubramanian/status/1304828079957774336
Our account was hacked in August, sendgrid support is useless. Rather move away to an alternative company. Sendgrid doesn’t care about helping their customers, support takes 2 – 3 weeks to even answer an email, then another 2 weeks to even get back to your replies.
Hope they lose a lot of business because they are useless.
Sendgrid would have more credibility if their website wasn’t designed to prevent any human contact. There is no choice but to block their domain.
Hey, i guess it is true. But there are some problem. Sendgrid is trusted by many customer but it is not expected to give the email to the spammers or hacked.
This is a scandal.
SendGrid could give a toss, really.
Their suggested solution? Upgrade to the 90 USD plan. They are doing little to nothing to solve the issue, and are even downgrading support tickets from customers asking for an update. This is akin to RACKETEERING in our books.
Twitter is popping up with irate customers, baffled with SendGrid’s nonchalant approach to this issue. Just look at the #SendGrid and #SendGridFiasco hashtags.
This whole issue shows the dark side of cloud services, where customers can be at the mercy of subpar support and poor resolve coming from service providers.
SendGrid has ignore these issues for years. They are lucky that this article is being so kind. Every client should push on the Twilio Trust team and Compliance to get details about the ATO problem. SendGrid is in the top 5 for Spamhaus listings. They are not incompentent but evil.