Production Server Security: Lessons from Pentesting My Own Infrastructure

I wear two hats — I build applications, and I try to break into them. Having done both for years, I’ve seen the same security gaps repeatedly. This post covers the practical steps I take on every server I manage, not theoretical best practices you’ll never implement.

Initial Server Setup

The first 10 minutes on a new server matter the most. Here’s my checklist:

1. Create a Non-Root User

Never run services as root. First thing after provisioning:

# Create user with sudo access
adduser deploy
usermod -aG sudo deploy

# Switch to new user for everything else
su - deploy

2. SSH Hardening

Edit /etc/ssh/sshd_config:

# Disable root login
PermitRootLogin no

# Disable password authentication
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (pick something above 1024)
Port 2222

# Limit login attempts
MaxAuthTries 3
LoginGraceTime 30

# Only allow specific users
AllowUsers deploy

# Disable empty passwords
PermitEmptyPasswords no

# Disable X11 forwarding (you don't need it on a server)
X11Forwarding no

Then restart SSH:

sudo systemctl restart sshd

Important: Always test SSH in a new terminal before closing your current session. I’ve locked myself out of servers before.

3. SSH Key Setup

On your local machine:

# Generate a strong key
ssh-keygen -t ed25519 -C "deploy@myserver"

# Copy to server
ssh-copy-id -p 2222 deploy@your-server-ip

Ed25519 keys are shorter, faster, and more secure than RSA. There’s no reason to use RSA anymore.

Firewall Configuration

UFW (Simple)

# Reset everything
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH on custom port
sudo ufw allow 2222/tcp

# Allow web traffic
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enable
sudo ufw enable

# Verify
sudo ufw status verbose

iptables (When You Need More Control)

# Flush existing rules
sudo iptables -F

# Default policies
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT ACCEPT

# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH
sudo iptables -A INPUT -p tcp --dport 2222 -j ACCEPT

# Allow HTTP/HTTPS
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Rate limit SSH attempts
sudo iptables -A INPUT -p tcp --dport 2222 -m recent --set --name ssh
sudo iptables -A INPUT -p tcp --dport 2222 -m recent --update --seconds 60 --hitcount 4 --name ssh -j DROP

# Save rules
sudo iptables-save > /etc/iptables.rules

Fail2Ban: Automated Intrusion Prevention

Fail2ban monitors log files and bans IPs that show malicious behavior:

sudo apt install fail2ban

Create /etc/fail2ban/jail.local:

[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
banaction = ufw

[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 86400

[nginx-http-auth]
enabled = true
filter = nginx-http-auth
logpath = /var/log/nginx/error.log
maxretry = 5

[nginx-limit-req]
enabled = true
filter = nginx-limit-req
logpath = /var/log/nginx/error.log
maxretry = 10
findtime = 120
bantime = 600

Check banned IPs:

sudo fail2ban-client status sshd

Nginx Security Headers

Every response from your server should include security headers. Add this to your Nginx server block:

server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:;" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;

    # Hide server version
    server_tokens off;

    # Limit request body size
    client_max_body_size 10M;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://localhost:3000;
    }
}

Automated Security Updates

Unpatched servers are the #1 attack vector. Enable unattended upgrades:

sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Edit /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";

Monitoring and Alerting

Security isn’t a one-time setup. I run these on every server:

# Install logwatch for daily log summaries
sudo apt install logwatch

# Install rkhunter for rootkit detection
sudo apt install rkhunter
sudo rkhunter --check

# Check for open ports regularly
sudo ss -tulpn

# Monitor file integrity
sudo apt install aide
sudo aideinit

Set up a cron job to email you daily reports:

# /etc/cron.daily/security-check
#!/bin/bash
logwatch --output mail --mailto [email protected] --detail high
rkhunter --check --skip-keypress --report-warnings-only | mail -s "rkhunter report" [email protected]

What I Find During Pentests

The most common issues I find when testing other people’s servers:

1. Default SSH port with password auth enabled — bots find it in minutes
2. No firewall — every port exposed to the internet
3. Outdated software — WordPress plugins from 2019, PHP 7.2 still running
4. Database ports exposed — MySQL/PostgreSQL accessible from the internet
5. No rate limiting — APIs that accept unlimited requests
6. Verbose error messages — stack traces leaking to production responses
7. Missing HTTPS — or HTTPS with HTTP still accessible

Every single one of these is fixable in under an hour. There’s no excuse.