Best Practices for running a mail server

Don’t hide behind anonymity.

Mail server domain whois should have an identifiable registrant organization, there should be a point of contact for any technical and abuse problems related to the mail server. If your registry hides registrant data, it might be a good idea to have a web site with the same name that’s not just showing a welcome message from an uninitialized CMS or hosting package. Mails sent to the abuse address must be read and acted upon, except for blatant spam of course.

Naturally, don’t send spam

Have all your users understand that sending unsolicited bulk/commercial mail is not acceptable and will lead to termination.

Have proper DNS setup

MX record for the domain pointing to the mail server.
FCrDNS records for all you public-facing IPs that match the SMTP HELO and EHLO
Stable IP address (not 5-minute TTL for dynamic DNS updates, no long-lasting outages)

DMARC

This is recomended to reduce spoofing.

TLS

Use TLS for both incoming and outgoing traffic whenever offered by the other side.
Use a proper certificate with a proper chain. Don’t use something self-signed, expired, etc
You can use CheckTLS to check your server is setup correctly

Split off email submission

Use a separate submission port for authenticated and encrypted mail submissions from your users. Add authentication information in mail headers to make identifying hacked mail accounts possible.

Restrict addresses users can use

If possible, restrict the use of foreign From: addresses to trusted users and automatic software. Don’t let just anybody send mails from president@whitehouse.gov…

Avoid creating backscatter

Either reject mails in the SMTP dialog or accept them. If you use spam detection software after SMTP acceptance, it should flag messages but still deliver them. There are cases such as autoresponders for vacations and mailing list software which will need to automatically send responses to sender addresses, but these should be monitored closely to detect abuse early.

SPF and Forwarding

SPF rules means that you must usually rewrite the from address if you forward email to another provider (eg run a mailing list)
You should also filter out Spam so destinations don’t see your site as a spam source
Some information on the Best ways to forward mail to Gmail
Having SPF for your own domains is usually considered a weak signal as is filtering on them

Large Providers (gmail, Microsoft, etc)

If you want to send mail to recipients who have accounts at big email providers, be aware that all of the above cannot guarantee that these providers won’t reject your mail, put it straight into recipient’s spam folder or just silently discard it - they just impose their own rules on anyone and you virtually can’t do anything about it.