Google is eating our mail

25.04.2019 20:06

I've been running a small SMTP and IMAP mail server for many years, hosting a handful of individual mailboxes. It's hard to say when exactly I started. whois says I registered the tablix.org domain in 2005 and I remember hosting a mailing list for my colleagues at the university a bit before that, so I think it's safe to say it's been around 15 years.

Although I don't jump right away on every email-related novelty, I've tried to keep the server up-to-date with well accepted standards over the years. Some of these came for free with Debian updates. Others needed some manual work. For example, I have SPF records and DKIM message signing setup on the domains I use. The server is hosted on commercial static IP space (with the very same IP it first went on-line) and I've made sure with the ISP that correct reverse DNS records are in place.

Homing pigeon

Image by Andreas Trepte CC BY-SA 2.5

From the beginning I've been worrying that my server would be used for sending spam. So I always made sure I did not have an open relay and put in place throughput restrictions and monitoring that would alert me about unusual traffic. In any case, the amount of outgoing mail has stayed pretty minimal over the years. Since I'm hosting just a few personal accounts these days, there have been less than 1000 messages sent to remote servers over SMTP in the last 12 months. I've given up on hosting mailing lists many years ago.

All of this effort paid off and, as far as I'm aware, my server was never listed on any of the public spam black lists.

So why am I writing all of this? Unfortunately, email is starting to become synonymous with Google's mail, and Google's machines have decided that mail from my server is simply not worth receiving. Being a good administrator and a well-behaved player on the network is no longer enough:

550-5.7.1 [...] Our system has detected that this
550-5.7.1 message is likely unsolicited mail. To reduce the amount of spam sent
550-5.7.1 to Gmail, this message has been blocked. Please visit
550-5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError
550 5.7.1  for more information. ... - gsmtp

Since mid-December last year, I'm regularly seeing SMTP errors like these. Sometimes the same message re-sent right away will not bounce again. Sometimes rephrasing the subject will fix it. Sometimes all mail from all accounts gets blocked for weeks on end until some lucky bit flips somewhere and mail mysteriously gets through again. Since many organizations use Gmail for mail hosting this doesn't happen just for ...@gmail.com addresses. Now every time I write a mail I wonder whether Google's AI will let it through or not. Only when something like this happens you realize just how impossible it is to talk to someone on the modern internet without having Google somewhere in the middle.

Of course, the 550 SMTP error helpfully links to a wholly unhelpful troubleshooting page. It vaguely refers to suspicious looking text and IP history. It points to Bulk Sender Guidelines, but I have trouble seeing myself as a bulk sender with 10 messages sent last week in total. It points to the Postmaster Tools which, after letting me jump through some hoops to authenticate, tells me I'm too small a fish and has no actual data to show.

Screenshot of Google Postmaster Tools.

So far Google has blocked personal messages to friends and family in multiple languages, as well as business mail. I stopped guessing what text their algorithms deem suspicious. What kind of intelligence sees a reply, with the original message referenced in the In-Reply-To header and part quoted, and considers it unsolicited? I don't discount the possibility that there is something misconfigured at my end, but since Google gives no hint and various third-party tools I've tried don't report anything suspicious I've ran out of ideas where else to look.

My server isn't alone with this problem. At work we use Google's mail hosting and I've seen this trigger happy filter from the other end. Just recently I've overlooked an important mail because it ended up in the spam folder. I guess it was pure luck it didn't get rejected at the SMTP layer. With my work email address I'm subscribed to several mailing lists of open source software projects and regularly Google will decide to block this traffic. I know since Mailman sends me a notification that my address caused excessive bounces. What system decides, after months of watching me read these messages and not once seeing me mark one as spam, that I suddenly don't want to receive them ever again?

Screenshot of the mailing list probe message.

I wonder. Google as a company is famously focused on machine learning through automated analytics and bare minimum of human contact. What kind of a signal can they possibly use to train these SMTP rejects? Mail gets rejected at the SMTP level without user's knowledge. There is no way for a recipient to mark it as not-spam since they don't know the message ever existed. In contrast to merely classifying mail into spam/non-spam folders, it's impossible for an unprivileged human to tell the machine it has made a mistake. Only the sender knows the mail got rejected and they don't have any way to report it either. One half of the feedback loop appears to be missing.

I'm sure there is no malicious intent behind this and that there are some very smart people working on spam prevention at Google. However for a metric driven company where a majority of messages are only passed with-in the walled garden, I can see how there's little motivation to work well with mail coming from outside. If all training data is people marking external mail as spam and there's much less data about false positives, I guess it's easy to arrive to a prior that all external mail is spam even with best intentions.

This is my second rant about Google in a short while. I'm mostly indifferent to their search index policies, however this mail problem is much more frustrating. I can switch search engines, but I can't tell other people to go off Gmail. Email used to work, from its 7-bit days onward. It was one standard thing that you could rely on in the ever changing mess of messaging web apps and proprietary lock-ins. And now it's increasingly broken. I hope people realize that if they don't get a reply, perhaps it's because some machine somewhere decided for them that they don't need to know about it.

Posted by Tomaž | Categories: Life

Comments

Jok za 7 bita email me spominja na 100-letno ohranjanje Morse telegrafije. Google search dela še hujše prekrške z kratko zgodovino. Pozdrav iz Srbije, dežele mitov.

Depressing. Problems like yours were the reason why I switched from "self-hosted" to using Fastmail. I am sure it does not prevent errors, but having a bigger ape behind me helps reduce problems with this gorilla.

Personally I kind of prefer SMTP level rejection to ending in SPAM folder. At least then I know that my mail has not been received. The most frustrating experiences for me where when I would not receive a reply and would have no idea why. In my experience most people do not check their junk folder (I rarely do) except maybe when they are expecting an email.

The same is happening to me. At first it was only with mails from the self hosted mail server with a dynamic IP. But more and more it happens with externally hosted email providers as well. If I am in the receivers contact list, it is mostly fine.
But when I was recently organizing our summer holiday road trip, I tried contacting hotels along the route and holiday house owners at the destination. Often my mails would not be delivered. At times I got this response from Google that it didn't deliver, and sometimes it just vanished. Sometimes I had to remove all URLs in the message, even though these would have been important. It was quite comic that I would reply to an email by way of filling in the contact form on the hotels website.
Google has transformed from "do no evil" into a cyber bully in a short time.

I also run my own mail server and this is now getting too much.
Who, how and where can we get someone to stop this madness.
With this type of over-arching control the prospects of an open internet with free communication are quickly becoming a thing of the past.

Posted by Shaun

We run small hosting company, if any domain blocks our clients mails we block theirs and problem gets solved magically.

On the other hand, I cannot stop receiving infinite emails from rich princes in Arabia, Apple fake invoices and porn offers (sometimes 300+/month). Why these can't be blocked at SMTP level?

What about some sort of union of independent email servers in one big conglomerate that google (and others) can't ignore?
Couple of questions here:
1. Is it technically possible? Every company from the union will want obviously to use its own domain. Is it possible to have anything in common for all these servers, so that list of domains shouldn't be always updated. Other simple option is just keeping a list of domains.
2. How to enforce big players to play nicely with small ones. I think once it becomes big enough, big players won't have a choice.

Sasha

Posted by Sasha

When this happened to me, it was because I was lacking DMARC records. Setting those up magically let me send mail to Gmail addresses again. I already had SPF and DKIM from previous times that Google had pulled this stunt.

In all cases, the mail server that Google was blocking had good reputation in the various spam systems, and had been sending mail to Gmail for over a decade without issue, so Google knew it was in good standing.

*sigh*

Posted by UghGoogle
Posted by RobertX

RobertX - That page isn't a good way of testing non-GSuite mail domains. That site is for testing mail domains that use GSuite, hence the error about SPF not being correct (i.e., not set up for GSuite). Additionally, the DKIM cannot be tested without knowing the selector.

Google now drops emails sent from within my domain to users within my domain as a g suite user. Product is garbage.

Posted by Ryan

I think at the end of the day, Google just doesn’t care about your use case, nor do they need to to be successful. it’s servers like yours that you can easily use to forge sender data and spoof emails. Most people don’t use their own SMTP server to send email, so most people are safer from these hypersensitive filters.

Posted by Bridge

I just assume that anyone with a gmail address is indifferent about receiving e-mail. It's up to that person to use a sensible service; it's not up to me to accommodate gmail.

Posted by Alex

I've been hosting a small mail-server for around 20 years and have to deal with this kind of problem too, setting up DMARC has done the trick for sending to gmail.

Check mxtoolbox.com, they can check about 100 block lists to see if your IP is clean, and give you info on how to get off if you're found on any. And also mail-tester.com is a great resource, you send a message to it and it checks that everything is looking good from the receiver's perspective and gives you a list of improvements etc.

Unfortunately I'm seeing more and more email servers that are just not bothering to test DKIM, SPF or DMARC, they just assume that if you're not one of the big corporate mail servers you must be a spammer :-(

Posted by Aran

Ha this sounds a lot like Apple's iMessage problem... You can't guarantee text message delivery or receipt unless you have an iPhone. Group "text" messages are far from reliable for the iPhone user.

Posted by John

Maybe the problem is caused by the low volume of mail.

Perhaps google code is written to presume a certain minimum volume of mail coming from any given legitimate SMTP server.

Maybe test this by having a bot that sends snippets of Shakespeare plays to a set of gmail accounts on a random schedule?

Posted by Pat

@Pat good idea! But, if you have to keep up a steady stream of email why not up the ante. I'm sure Tomaz has friends who run or work at companies that would like to let users know about their useful products. Maybe the emails can reference them, or even include a promotional coupon.

Posted by igor

Just a mention that I was receiving the same error from gmail until I changed my DNS SPF record into a TXT record (with the SPF details within).

Posted by Gordon

$ host tablix.org
tablix.org has address 193.164.138.132
tablix.org mail is handled by 10 mail.tablix.org.

$ host mail.tablix.org
mail.tablix.org has address 193.95.199.109
mail.tablix.org has IPv6 address 2a00:ee2:100:3a01::1

$ host 193.95.199.109
109.199.95.193.in-addr.arpa domain name pointer chandra.tablix.org.

-> The PTR of the mail server IP address is not the same as its actual name.

Posted by Dropple

I sympathize. Google's indifference to this class of problem borders on nefarious. Reading your first paragraph sounded exactly like my trajectory except, like another commenter, I actually handed of relaying of mail to a third party (DNSExit). But I refuse to put my private mail in Google's hands.

Don't give up.

#metoo. I have a small business and will now have to put a message on my website:. Google mail AI is broken and you will not receive our emails because of that.

Posted by Marcus

I've been seeing the same problem when sending to gmail for a few months now. The IP address of my host lies in a completely clean company network and only I send the occasional e-mail or two from the machine in question. SPF is proper, DKIM is proper, yet this bullcrap happens regularly.

Posted by David

^ And what's funnier still is that this sh*t happens even with e-mails that are -replies- to people who contacted me from their gmail accounts, where my replies return the various unique and unguessable headers that gmail produced in the initial e-mail sent to me. Fix your garbage, Google.

Posted by David

My openpgp encrypted emails tend to fly thru gmail filters from my own server. Maybe the trick is to not allow Google AI to read the message.

I don't blame Google. I ran my own mail servers for years and gave up due to spammers. As people said above, you most likely don't have one of the many mail features like DMARC setup. Also there is S/MIME which is being added now.
Now I just route all my mail through Zoho as they are more affordable than Google and I like their interface better. I spent too much managing my own server. Really isn't worth the time and headaches.

Good luck, keep up the good fight. I think g is destroying the free web and people like you are keeping it alive.

Posted by James

It's not just Google, Microsoft is just as bad. I'm surprised you get messaged in your SMTP logs. I need to start checking mine. This isn't a new problem. I wrote about it a few years back:

https://penguindreams.org/blog/how-google-and-microsoft-made-email-unreliable/

..and if you check my sources, there's an older post about the 'Hostile e-mail landscape' (it's gone now, but you can find a copy on archive sites).

Anecdotal evidence here, but I've been operating an SMTP relay for ~two-dozen users for about 10 years now. Never had any issues to speak of with delivery to Gmail; and I have been blacklisted by several others. That being said, I do feel for you--if you're doing everything properly (I'm assuming you're doing DMARC?), and you're still getting fucked by the algorithm, there's not much you can do about it..

PS @Dropple it's 2019, that's to be expected. Run the same queries on smtp.gmail.com; that's really not the problem here.

Posted by captnjlp

I wonder if you're being joe-jobbed.

Posted by Emil Aich

Our company emails have been getting this kind of false spam tag by gmail for months now. Sometimes social media links trigger it and sometimes a text with a hyphen triggers it. I've gone thru the procedures you followed multiple times. We have SPF, DKIM and DMARC but of course gmail dosesn't send DMARC reports. It is so frustrating to see how gmail has made something as reliable as an email as their AI experimental toy.

Posted by Tim

Same shebang, same issues. I like the idea about unionizing. If google unfairly blocks a domain owned by a union member, the union as a whole block them until google fixes their system.

Posted by Tired Admin

I also deal with this very often, even though I don't run my own mail server. I use my own domain name and a small internet company provides me with SMTP services. I've noticed recently that without reason, email no longer gets reliably delivered to gmail addresses. It always seemed completely absurd to host my email with a private company I have no control over, which is why I've also used a domain name and web hosting I'm paying for. The day I have no choice but to register a .gmail address to send email is probably the day I will stop using email altogether.

The end of the free web is near on so many levels

Oh, interesting. This explains so much. I have a gmail account and I'm always amazed at how little "spam" it gets. Now I know why, they reject everything. Easy way to make an email service look super good at rejecting Spam when you just reject everything. Kind of like cheating don't you think? So I'm glad I kept all my other non-gmail accounts.

I've also had extensive issues self-hosting email and getting stuff rejected by all manner or large mail providers.

I did everything I could to make myself as acceptable as possible to them (SPF, DMARC, DKIM, you name it) but since I was sending from a domain with a non-standard TLD (.link)

The basic premise that well-behaving email servers get denied randomly when sending normal mail remains.

Same here. Deliver in over IPv4 looks like works every time. But over IPv6 I got most the reject message even all settings are the same as for ipv4.
Problem here is, i can setup the transport for gmail, but not so easy for all gsuite domains. This leads me to the conclusion, IPv6 is broken on google side.
On the other side, i see recently a lot of spam sent over google, from gmail and also over gsuite accounts.

Posted by Ingo

I have also been running my own postfix mailserver for years and not had much issues. The key is to have the reverse of the mailserver set to equal the mail system host name, which will get rid of most trust-issues. SPF is also a must today, thanks to Google.

I have mitigated spam by having postfix require other servers follow protocol and also to require that the reverse is equal to the remote mail system host name. If the remote server follows protocol, the next step is adding heavier checks like greylisting, which makes my personal spam mail amount go down to one or two a year because spambots fire and forget instead of trying again like real mail servers.

No magic, just following protocol.

Posted by d00bianista

Fix your PTR record for 193.95.199.109 - should point at mail.tablix.org, *not* chandra.tablix.org.

That will improve matters tremendously.

Had similar situation... Disable communicating to google's smtp via ipv6 and use only ipv4. This will probably solve your problem.

A decade ago I was hoping that SPF would die any day now... Instead, companies like Google perpetuate this broken idea. http://david.woodhou.se/why-not-spf.html was written ~15years ago.

When Google tech works, it's great. When it doesn't, it's like a bulldozer on autopilot going through your town. It's unstoppable, and for practical purposes - unmanned.

BTW, if you want to be sure your SMTP relay has not landed on any blacklists over long stretches of time, I'd recommend some automated monitoring service like rblmon.com or rbltracker.com - both have free plans.

Posted by spfdead

> Fix your PTR record for 193.95.199.109 - should point at mail.tablix.org, *not* chandra.tablix.org.

That's just plain wrong. Guys, don't give DNS advice if you're not familiar with the topic.

His server's hostname seems to be chandra.tablix.org and it's A record points to 193.95.199.109.

And the other way around, PTR for 193.95.199.109 is chandra.tablix.org.

This is absolutely correct. PTR record is correctly set.

Posted by Ales

Duh, "its A record", not "it's A record". Man I hate to do this.

Posted by Ales

@Ales, either one has to configure consistently or then configure SPF properly. I'd recommend using a single name instead of for example chandra and mail. This is why my servers hostname is mx as it's most critical task is mail exchange and SPF says that only mx is allowed to send mail for my domains. And it's a must that the reverse matches the hostname that the mailserver is telling other servers that allegedly is.

Posted by d00bianista

As a couple of other people in the comments have suggested I think something like a cooperative of independent mailers is a great idea.
It could be as simple as an MTA configured to relay mail for members.

Posted by Charles

I used to get this regularly from Gmail. Once I got my server listed on DNSWL, never again

I believe contributing factors are forwarding genuine spam (as would happen when forwarding all mail to an address, no filter is perfect) and sending lots of zip attachments

Posted by Luke

There is no DKIM or DMARC Record for your domain, a DKIM record along with a SPF record should help solve this issue!

The same happened to Firefox developers... from time to time Firefox failed on working with Google services...

Posted by Iso9660

Umm... if the PTR must match the hostname, that's completely new and has never been documented by everyone.

tablix.org has an MX record mail.tablix.org
mail.tablix.org has an A record 193.95.199.109
109.199.95.193.in-addr.arpa has a PTR record chandra.tablix.org
chandra.tablix.org has an A record 193.95.199.109

There's nothing wrong there (in this case it's considered polite to have the SMTP banner say "mail" instead of "chandra", though I've never heard of any software caring).

All that matters is that the A of the PTR resolves back to the IP address, which demonstrates you control that IP address; there's no reason a mail exchanger can't have multiple A records.

Posted by Weldon

As the rest of the folks mentioned here, disable IPV6 and use only IPV4.

DKIM actually has a very marginal impact on sender reputation these days. It only confirms if an email is spoofed. Have you ever heard of a top email provider that rejects non-DKIM emails or checks if DKIM emails are signed?

SPF is generally still more than enough unless you send out enough emails to justify developing a DKIM reputation. DKIM requires email servers to actually enforce the standard, which they generally don't.

Posted by A4

I let a seemingly competent hosting company handle my SMTP shenanigans under my own domain name, so I'm not the only one using the mailserver I have. However, lately, I was getting the impression that some people don't seem to hear from me anymore, as I didn't get any replies. At first I thought they were just ignorant of me...

I can't 100% confirm this happened/happens to me, as I have no logs/time to look into them right now, but I strongly doubt it's not that, because the time frames fit perfectly.

I like the unionization idea of Sasha. Discussion does never help. Only action does.

Posted by Khan

Thank you everyone for the many comments and useful suggestions on how to improve my mail setup. Thanks to the tool RobertX linked above, I've fixed a minor issue with my DKIM record (the recommended "v=DKIM1" record was missing).

My plan is to setup DMARC next, since I haven't yet taken time to look into that spec properly. If DMARC doesn't improve things, I'll try disabling IPv6 for SMTP connections to Gmail. My IPv6 setup is much more recent than my server, so it makes sense that my IPv6 address would have a lower reputation than the IPv4 address.

Posted by Tomaž

There is a high probability of misconfiguration. I have been hosting personal SMTP for years and I stubled upon these ar the beginning:
- missing or wrong PTR record
- dynamic IPs or IPs from previously-abused network blocks
- missing SPF, DKIM
- bad domain/IP reputation (sending spam or sending email on behalf of someone else - like forwarding contact form TO: me@gmail.com FROM: someone@notMyDomain.com)
- self-signed TLS certificates

That being said, Google should definitely give more troubleshooting insight. When it starts dropping your email you don't immediately see why and being a big company, there is almost impossible to write to them.

Posted by Roman

Tomaz -- make sure your HELO banner from the SMTP server is chandra.tablix.org. The commenters above are correct that the A/PTR records must match, but they must match what's in the HELO banner, not the MX record.

Posted by Aaron

I've been self-hosting email for a little while too now. I originally followed the Ars Technica "taking email back" series, but I've since made some customisations to it.

I had an issue a few months ago where I emailed a family member who has a gmail account a zipped-up binary file. Next thing I know, the email never arrived and my whole server was blocked for at least a month! No warnings. When I tried to contact them, I got stonewalled and received no reply.

It's all deepening my distrust of Google. 1 company should never have this much power.

So add me to the list of those not able to email users with gmail accounts from my business email. I sent emails to 10 potential clients Friday and NOT one of them received it until I then called them and they found it in their spam folder. I am so frustrated as I'm loosing business. Of course today I also saw that an email to an AOL user didn't go through either.

I in part wonder if this is gmails way of getting users to sign up for their monthly payment of using gsuite for email rather then using ones provided by their hosting company.

Posted by cgema

Similar thing happens for me since few weeks (previously there was no problem), however, my e-mails aren't rejected by Google's servers, but simply land right away into Spam folder of the recipients, with the explanation that the message is "similar to ones already caught as spam", which is absurd, because each one of these messages is different.

What's most strange, this doesn't happen to people on Gmail with whom I already have some correspondence history - they continue to receive my e-mails normally. If someone on Gmail writes to me first, and I reply, my reply also *usually* (but not always) is correctly delivered to recipient's Inbox and not to Spam.

However, if I'm writing to someone on Gmail for the first time, Google treats my mail as spam.
To test this, I have created a fresh account on Gmail and sent e-mail to it from myself. The mail went straight to Spam. SPF, DKIM and DMARC tests were shown as "pass", but the e-mail still went to spam. Clicking "This is not spam" helps only for that particular account. If I repeat this with another freshly created account (I did this three times), mail again goes to spam.

Of course there is no way to contact Google, no contact form for such cases etc. - I think that having no such possibility to contact a company should be forbidden by law! All this makes me get really mad at Google...

I have lost touch with many personal and professional contacts over the years, using my ISP email client system as I was not aware of this problem until recently. I set up my own gmail account and found emails I sent to my own gmail address were either not being delivered at all or ending up in the spam folder. Gmail servers have blocked this for years and any complaints to my ISP are met with "we are working with Google to solve the problem" - they haven't!

To many people using gmail, their spam folder would be as good as no email as they're actually blissfully unaware.

To me, this is tantamount to criminal negligence as it undermines the integrity of a supposed reliable comms system.

As stated a number of times above, Google simply does not care because of the sheer numbers of their tech unsavvy customers who don't know and/or care.

In this way they have hijacked a good, fast and cheap comms system and turned it to their self-interest.

Time for some sort of intervention by some sort of regulator!?

Posted by Bobb09

Google is becoming to powerful in technology in general, and definitely in the email space. Although I have not had any deliverability problems with Gmail from my personal SMTP server, I am a firm believer that Google should not be able to singlehandedly dictate what email is allowed to be sent like this.

Enough people are using the Gmail service that if you don't follow their rules, your email will not be delivered. They are also abusing thier powers in the web space, making decisions on the whole web with their high market share Chromium, but that's a topic for another day.

This is happening to me, too.
Especially the cases where emails just get dropped, without a notification to the sender, is super frustrating.

No idea how y'all can stay so calm, I'm fucking angry right now. It is causing me a lot of trouble.

Google, fix your stuff.

Posted by mearon

Google is suffering enormous internal attrition of it's oldest and most talented workers, who were primarily responsible for core search and gmail. This is due to it's evil HR practices, it's evil relationships (under duress?) with the US/NATO intelligence community, and it's evil attempts to cozy up to China. Important executives have been bailing from the company since ~2015. Sergei Brin himself has washed his hands with it and walked away before Chrismas.

It's highly likely Gmail is a black box built out of millions of lines of interdependent spaghetti code and no one remaining on staff knows how it works.

What about the Union idea? There was some progress in that regard?
We have to do something to do something...

Posted by JL

As others have pointed out, you definitely want your forward and reverse mail entries to match your MX declaration.
I've run SMTP daemons since the early 1990s. Over time I've had to add SPF, DKIM, DMARC to keep certain hosts happy.
Overall, things have worked well, but during the past couple years I've begun to see problems where emails sent to large email hosts (hotmail, google) have nor been received by the destined users.

I've also shockingly observed (when sending email from hotmail to gmail) a rejection when warning a friend about the FTX crypto fraud when it was first announced. Removing any references to FTX and crypto finally allowed the mail to be accepted.

Alas, it feels like we're quickly reaching the point where many have decided to surrender their freedom of speech in email communication. Consequently, the value of email services for me has been exponentially decreasing with every day that passes. :(

Posted by Adam

Add a new comment


(No HTML tags allowed. Separate paragraphs with a blank line.)