Set Up Your Own Mail Server on a VPS: Postfix + Dovecot

Most of the mail guides we publish are about fixing deliverability on an existing mailbox — SPF records, PTR entries, port 25 relays. This one's different. If you want to run your own SMTP and IMAP stack on a VPS instead of using cPanel's built-in Exim/Dovecot setup or an outside provider, here's how to do it with Postfix and Dovecot, and what to watch out for once it's live.

Why run your own mail server

Reasons vary. Some people want mail on a bare Ubuntu or AlmaLinux VPS with no cPanel license. Some are running a SaaS app that needs to send transactional mail from its own domain and don't want to depend on a third-party API. Others just want to understand the plumbing instead of trusting a black box. Whatever the reason, the two pieces you need are the same: Postfix to handle SMTP (sending and receiving), and Dovecot to handle IMAP (so mail clients can actually read the mailboxes).

Before you start, be honest about one thing: self-hosted mail is unforgiving. A misconfigured DNS record or an open relay will get your IP blacklisted fast, and getting off a blacklist takes days, not minutes. If you just need reliable outbound mail for a couple of addresses, cPanel's built-in mail or a transactional service is usually less work. If you're doing this anyway, here's the real setup.

What you need before you begin

• A VPS with a static public IP and root/sudo access (Ubuntu 22.04/24.04 or AlmaLinux 9 in this guide)
• A domain you control, with access to its DNS zone
• A PTR (reverse DNS) record already pointed from your VPS IP to your mail hostname — ask your provider's support panel or ticket system if you can't set this yourself
• Port 25 open outbound (some VPS providers block it by default on new accounts — check before you go further)

Pick a hostname for the mail server itself, separate from your website — something like mail.yourdomain.com. That's the name Postfix will identify itself with, and it's what your PTR record should resolve to.

Step 1: Install Postfix and Dovecot

On Ubuntu/Debian:

sudo apt update
sudo apt install postfix dovecot-core dovecot-imapd mailutils -y

During the Postfix install prompt, choose Internet Site and enter your mail hostname (mail.yourdomain.com) as the system mail name.

On AlmaLinux/RHEL:

sudo dnf install postfix dovecot -y
sudo systemctl enable --now postfix dovecot

Step 2: Configure Postfix

Edit /etc/postfix/main.cf and set these values (adjust the domain to yours):

myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = may
smtp_tls_security_level = may

The home_mailbox = Maildir/ line matters — it tells Postfix to deliver into the Maildir format Dovecot expects, instead of the old single-file mbox format.

Restart Postfix to apply:

sudo systemctl restart postfix

Step 3: Get an SSL certificate for the mail hostname

Mail clients will refuse to connect cleanly, or throw certificate warnings, without valid TLS. Use Certbot the same way you would for a website:

sudo apt install certbot -y
sudo certbot certonly --standalone -d mail.yourdomain.com

Make sure port 80 is free when you run this (stop Nginx/Apache briefly, or use the webroot method if the box is already serving a site). The certificate paths above in main.cf already point to where Certbot drops the files.

Step 4: Configure Dovecot for IMAP

Edit /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:~/Maildir

Edit /etc/dovecot/conf.d/10-auth.conf and make sure plaintext auth is disabled unless the connection is encrypted:

disable_plaintext_auth = yes
auth_mechanisms = plain login

Point Dovecot at the same certificate in /etc/dovecot/conf.d/10-ssl.conf:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

Restart Dovecot:

sudo systemctl restart dovecot

Step 5: Create a mailbox and test it

The simplest approach is one Linux system user per mailbox:

sudo adduser sales
sudo passwd sales

Send a test message from another account and check delivery:

echo 'test body' | mail -s 'test subject' sales@yourdomain.com
sudo su - sales
ls Maildir/new/

If a file shows up in Maildir/new/, delivery is working locally. Next, configure a mail client (Thunderbird, Outlook, or a phone mail app) with IMAP on port 993 (SSL) and SMTP on port 587 (STARTTLS), using the system username and password.

Step 6: DNS records — the part that actually determines deliverability

Postfix and Dovecot handle sending and reading mail, but whether the rest of the world accepts your mail comes down to DNS. Set these in your domain's zone:

Skip DKIM and most receiving servers will still accept your mail today, but it'll get flagged as suspicious sooner rather than later — it's worth setting up opendkim from day one rather than retrofitting it after your first blacklist scare.

Prevention: keeping a self-hosted mail server healthy

• Never leave it an open relay. Double-check mynetworks in main.cf only includes your own trusted IPs, not 0.0.0.0/0.
• Install fail2ban with the Postfix and Dovecot jails enabled — brute-force login attempts against port 587/993 start within hours of the server going live.
• Set up automatic Let's Encrypt renewal (certbot renew via cron) so the cert doesn't silently expire and break TLS for every client.
• Monitor your mail queue with postqueue -p periodically — a growing queue almost always means a deliverability problem you need to catch early, not after 500 messages pile up.
• Check blacklist status on a site like mxtoolbox occasionally, especially in the first few weeks while your IP is still building sending reputation.

If keeping up with all of that sounds like more ongoing work than you want to take on, that's a fair conclusion — it's exactly why most of our customers run mail through cPanel's built-in stack or a managed transactional provider instead of rolling their own. But if you need full control over the mail stack itself, Postfix and Dovecot on a VPS get you there.

Frequently Asked Questions

Do I need cPanel to run Postfix and Dovecot?

No. This setup is meant for a bare VPS without cPanel/WHM. If you already have cPanel, it ships with Exim and Dovecot preconfigured, and you're better off using the Email Accounts interface there instead of installing a parallel mail stack.

Why is my outbound mail landing in spam even after this setup?

Almost always missing or incorrect SPF, DKIM, or PTR records, or a brand-new IP with no sending history. Run your domain through a mail-tester tool after sending a test message and fix whatever it flags before assuming the server itself is broken.

Can I host multiple domains' mail on one Postfix instance?

Yes, using virtual domains and virtual mailbox maps instead of one system user per mailbox. That's a more involved config than covered here — worth doing once you're comfortable with the single-domain setup above.

My VPS provider blocks port 25 — what now?

You'll need to either request port 25 be unblocked (most providers will do this after identity verification) or route outbound mail through a relay like SendGrid or Amazon SES configured as Postfix's relayhost. Local delivery and IMAP still work fine either way.

How do I know if my mailbox actually received a message versus it silently failing?

Check /var/log/mail.log (Ubuntu/Debian) or /var/log/maillog (AlmaLinux/RHEL). Every accepted, bounced, or deferred message shows up there with a reason, which is the fastest way to tell delivery failures from a misconfigured mail client.