Mail server setup

“Why is legitimate email sent from my server rejected or flagged as spam?”

In the last eight years working as mail- and spam filter administrator I’ve had to answer this question many times. If you google it on the web, you’re often getting answers like “because you don’t have SPF/DKIM/DMARC”. This might be the reason in some cases, but most of the time the problem lies in the basic server hostname and DNS setup.


Many spam filters first try to find out, if the sending device was meant to send mail or if its just yet another hacked VPS / home PC  out there. So your job is to clearly tell them "yes, the administrator of this device and the ISP agree that this system should send mail".  And you tell them this by setting a distinctive SMTP HELO (which makes clear that the SMTP engine was set up by a mail administrator), DNS A-record (which defines that IP address is the one meant send mail) and reverse DNS (which confirms that the owner of the IP adddress  agrees). If you get these three things right, chances are already pretty high that your legitimate mail will be accepted even without SPF and DKIM.  

System hostname / SMTP HELO

Most mail servers automatically use the system hostname as SMTP HELO - so this is the first thing that should be set correctly.

• Configure a fully qualified domain name on your mailserver (something like ‘mail.example.com’, 'exchange.example.com', 'mta.example.com' )
• DO NOT actually use “example.com” - we will use this domain as an example here, but you should not on your server. Whenever you read ‘mail.example.com’ below, replace it with the fully qualified hostname you’ve chosen in a domain under your control. (yes, I’ve seen people literally configure ‘example.com’ on their servers. It’s a bad idea.)
• DO NOT use hostname in a domain of your ISP/VPS Provider. Use your OWN domain.
• DO NOT choose a hostname that looks auto-generated (based your IP address etc). 
• DO NOT try to be cute and invent your own top level domain ("myserver.home", "exchange.lan") - your hostname should be publicly resolvable

IP

• Send mail from a static IP address. If you don’t have one, use a smarthost that does
• If your IP address is listed on the Spamhaus PBL it is not meant to be used to send mail directly. Use a smarthost or ask your ISP for an IP in a different range.
• if you have multiple static IP addresses available, configure a dedicated IP for your mail server which is not used as gateway by any other devices in your network. This reduces the risk of an infected device causing your mail IP to get blacklisted

Reverse DNS

• Set the reverse DNS (PTR) entry to the your server’s hostname (x.x.x.x.in-addr.arpa PTR mail.example.com)
• DO NOT set multiple PTRs. Your sending IP should have exactly one PTR. It does not matter if this server is hosting multiple domains for web / mail. The PTR is used to identify the sending server, not the domains it is hosting
• DO NOT use generic PTRs from your upstream/IP/vps provider. (like x.x.x.x-static.reverse.softlayer.com). Set this to a domain under your control.

A-record

• Make sure there is an A record that maps the hostname used in your PTR back to your sending IP address (see FcRDNS)
• if you used a different FQDN in your smtp HELO for any reason, add an A record for this  as well . Only extremely agressive spam filters will actually check if your helo is resolvable, but RFC 5321 states "Only resolvable, fully-qualified domain names (FQDNs) are permitted  when domain names are used in SMTP.", so better be safe than sorry.