This is an honest account of how this server is hardened and why. Not a marketing checklist. Not a certification claim. A record of decisions made, trade-offs weighed, and failures caught before they shipped.
Security is treated as a posture — not a checklist, but a disposition that shapes every decision. The architecture is designed to be unforgiving: the same hardening that stops an attacker stops a careless operator. Every component follows the principle of least privilege.
We're publishing this because security by obscurity isn't security. If our approach has gaps, we want to know. If it's useful to someone building something similar, they should have it.
Stack Overview
The production environment runs two layers that are independently hardened.
| Layer | Technology | Role |
|---|---|---|
| Edge | Cloudflare | TLS termination, DDoS, CDN |
| Application | Node.js / Express on Debian VPS | IMAP sessions, business logic |
The Cloudflare Worker (Google OAuth track) runs in an isolated execution environment — no persistent server, no SSH surface. That track's hardening lives in the application code. This document covers both, with emphasis on the VPS layer where the attack surface is real and the decisions are ours.
1. Network Layer
- Firewall (UFW): deny-all default. Three ports explicitly allowed: HTTP, HTTPS, SSH. Everything else is dropped.
- SSH port: moved off the standard port. Invisible to default scanners. The choice is arbitrary — the point is to eliminate noise, not achieve obscurity.
- Brute force protection (fail2ban): active on SSH. 10 failed attempts within 1 hour triggers a 1-hour ban. Bans expire automatically — no permanent lockouts. This is a secondary measure, not a primary one: key-only auth means password brute force is already impossible. fail2ban catches bad key attempts and scanner noise. The threshold is set at 10 because 3 punishes bad memory, not attackers — a script will trip the limit in under a second regardless.
2. SSH Hardening
- Key-only authentication. Password auth disabled entirely in
sshd_config. No exceptions. - Root login disabled.
PermitRootLogin no. All admin work is done as a named sudo user. - Max 3 authentication attempts per connection.
- 20-second login grace timeout.
- Restricted to a single named user.
- Non-standard port. Confirmed working before port 22 was closed. Application login was tested before root was disabled. You don't cut off your backstop before you've confirmed the new path.
3. Kernel & OS
- SYN cookies: enabled — protects against SYN flood attacks.
- ICMP redirect ignore: prevents man-in-the-middle via route manipulation.
- Martian logging: flags impossible traffic routes.
- Unattended upgrades: automatic security patches. Staying current is the baseline.
- Service audit: unnecessary services (
multipathd,atd) disabled. Reduced attack surface.
4. Application User Isolation
The application runs as a dedicated system user (aimail) with:
- No shell (
/usr/sbin/nologin) - No sudo access
- No password
- Ownership scoped to application directories only
If the application is compromised, the blast radius is contained to what aimail can reach — which is only what it needs to run.
5. Process Management
PM2 manages the Node.js process with a startup hook wired via pm2 startup systemd. The server survives reboots without manual intervention and without running the application as root. Process status, restart count, and memory usage are monitored in the daily report. Restart deltas are tracked — unexpected cycling triggers review.
6. Security Monitoring
Architecture Decision
The security layer is server-level, not application-level. The watcher doesn't know about Node.js, IMAP, or Gmail. It watches the OS. Every application that runs on this box gets the same protection for free. Build at the right layer and you only build it once.
What We Watch
- Successful SSH logins from untrusted IPs
- User account creation or modification
- Unexpected sudo commands
On any of these events, an alert fires immediately — out-of-band to an external email account. An attacker who owns the machine cannot suppress the notification. The full log line is included: source IP, timestamp, key fingerprint. The alert account is MFA'd on a physical device.
The window between intrusion and awareness: minutes.
Least Privilege: The logwatcher User
The watcher runs as a dedicated system user (logwatcher) — not as the application user, not as root, not via a group escalation.
The simpler path was available: add aimail to the adm group — two commands, done. We didn't, because:
aimailwould gain read access to all of/var/log/, not justauth.log- An application compromise would inherit that access
admgroup membership is permanent and system-wide
logwatcher gets exactly what it needs and nothing else:
| Resource | Access | Mechanism |
|---|---|---|
/var/log/auth.log | Read only | POSIX ACL scoped to that file |
/var/lib/logwatcher/ | Read/write | Owns the directory |
/etc/logwatcher/watcher.conf | Read | System config path |
/etc/msmtp/logwatcher.conf | Read | System msmtp config — logwatcher has no home dir |
| Mail send (msmtp) | Execute | AppArmor local override |
Isolation guarantee: If the application user is compromised, the watcher is untouched. If the watcher user is compromised, it cannot execute commands. The two are completely independent.
AppArmor
msmtp is confined by AppArmor. The default profile doesn't allow reads from /etc/msmtp/ or writes to /var/lib/logwatcher/. A local override (/etc/apparmor.d/local/usr.bin.msmtp) grants exactly those paths — nothing broader. The main AppArmor profile is untouched.
System Cron
The watcher cron runs via /etc/cron.d/security-watch with logwatcher named as the executing user. It does not depend on aimail's crontab, aimail's environment, or anything in aimail's home directory. System cron outlives user session changes.
Execute Bit in Git
Watcher scripts are versioned in the application repo. Execute permissions are baked into git via git update-index --chmod=+x. Every pull preserves them. No post-deploy chmod step that can be forgotten.
7. Daily System Report
An HTML email is delivered at 08:00 UTC every day, covering all critical metrics with colour-coded thresholds:
| Metric | Green | Yellow | Orange | Red |
|---|---|---|---|---|
| Disk usage | < 70% | 70-80% | 80-85% | > 85% |
| RAM usage | < 75% | 75-85% | 85-90% | > 90% |
| PM2 process status | Online | — | — | Not online |
| Daily IMAP request count | < 200 | 200-350 | 350-500 | > 500 |
The report also includes: uptime, PM2 restart deltas, fail2ban active bans, failed SSH attempt count for the day, and the last 3 banned IPs.
8. Transport & Response Headers
| Header | Value |
|---|---|
Strict-Transport-Security | max-age=31536000; includeSubDomains |
Content-Security-Policy | Restrictive default, no frame ancestors |
X-Frame-Options | DENY |
X-Content-Type-Options | nosniff |
Referrer-Policy | no-referrer |
Permissions-Policy | Camera, microphone, geolocation, payment — all disabled |
x-powered-by | Suppressed. Express version banner removed. |
TLS is terminated at Cloudflare for the public domain. Let's Encrypt handles the VPS directly.
9. Application Security
Session Management
- Sessions held in memory only — nothing written to disk, no database, no persistent storage
- 2-hour inactivity timeout
- 8-hour absolute maximum — no session survives a full day
- Automatic cleanup sweep every 15 minutes
- Cookies:
HttpOnly,Secure,SameSite=Strict
Credential Handling — Full Lifecycle
IMAP credentials are never written to disk and never logged. Here is exactly what happens to your app password at every stage:
Login: You submit your email and app password. The password is used in plaintext exactly once — to authenticate against Gmail's IMAP server. If authentication succeeds, the password is immediately encrypted with AES-256-GCM using a per-instance key (generated fresh every time the server starts) and a random 12-byte IV. The encrypted blob is stored in the session. The plaintext is not retained.
At rest (signed in): Your password lives encrypted in server memory — never on disk, never in a database, never in a log. The encryption key exists only in process memory. If the server restarts, the key is gone and every session is invalidated. There is nothing to recover and nothing to breach.
Making a request: When you fetch emails or list folders, the password is decrypted on the fly, used for a fresh IMAP connection, and the connection is closed. The password returns to its encrypted state immediately.
Logout: The entire session object — including the encrypted password — is deleted from memory. The cookie is cleared. A background sweep also deletes any expired sessions every 15 minutes, so even if you close the tab without logging out, the session self-destructs.
Nothing is persisted. Nothing survives a restart. There is no disk, no database, no file to steal. The password exists encrypted in memory for the duration of your session and nowhere else.
Login Throttling
Progressive backoff keyed by IP + email (two-dimensional — a credential-stuffing attack from one IP doesn't lock out unrelated users at the same IP):
- Delays scale exponentially: 0s, 5s, 10s, 20s...
- 24-hour reset window
Rate Limiting
- 30 requests per hour per session (fixed window)
- 429 returned when exceeded
- Applied to IMAP-triggering endpoints only
Input Validation
| Field | Constraint |
|---|---|
| Format regex + max 254 characters | |
| Password | Max 128 characters |
| Dates | Strict ISO 8601 format, must parse as valid |
| Mode | Allowlist only |
| Request body | Capped at 20kb |
IMAP Safety
- 60-second connection timeout — a hung IMAP connection doesn't hold a session open indefinitely
- Max 3 concurrent IMAP operations per session (semaphore-based)
- Debug logging suppressed — verbose IMAP libraries can log credentials; that is disabled
Error Handling
err.message is never forwarded to the client. Errors return generic messages only. Stack traces stay on the server.
CSRF
Evaluated and confirmed not needed for this architecture. Sessions are stateless, IMAP credentials are user-supplied per request, there is no state-changing operation a third-party site could forge.
10. Access Model
Two paths into the server:
SSH — requires a cryptographic key. Password auth is off. Brute force is auto-banned. The port is non-standard.
Hosting console — browser-based, behind MFA tied to a physical device. Available when SSH is unavailable.
Both paths hit the same wall: a 64-character sudo password that is not a passphrase and not guessable. An attacker who gets through the front door still cannot escalate. They're standing in the lobby.
Every door requires a different key. The keys don't live in the same place. Getting one doesn't get you the rest.
11. Privacy Posture
The application logs nothing about users by design.
Not Collected
- IP addresses
- Email addresses (beyond session display)
- Session identifiers
- Login timestamps
- Message metadata or content
- Usage patterns
Collected
An anonymous request count — one line appended per IMAP-triggering request. No identifying information. Resets daily. Shows as a single number in the daily report. Enough to know if traffic is growing. Nothing that tells you who.
Data Lifecycle
- All session data is in-memory only
- Reboots clear everything
- Logout destroys session immediately
- No database — there is nothing to breach
12. Log Rotation
| Log | Retention | Format |
|---|---|---|
| PM2 application logs | 7 days | Compressed |
| Security watcher logs | 3 days | Compressed |
Email alerts are the forensic record. Local logs are operational noise. Seven days is enough to diagnose any crash. Longer retention is not privacy-neutral when users are putting email credentials into this system.
13. Recovery Chain
Hardening is not the hard part of security. Recovery is.
You can build a wall no one can climb and still lose everything if you can't get back in yourself. Every system needs a recovery chain that is: documented, tested in principle, and held somewhere the system itself cannot corrupt.
Password manager
└── recoverable via: recovery phrase (offline, physical)
├── SSH private key
├── VPS provider credentials → hosting console access (no SSH needed)
└── sudo password → full server access
Failure Scenarios
| Failure | Recovery Path |
|---|---|
| SSH key lost | VPS provider console (browser-based, key-free) |
| sudo password lost | Hosting provider rescue mode — mount filesystem — update sudoers |
| VPS provider account lost | Provider support + identity verification — no automated path |
| Server compromised | Rebuild from scratch. Application state is in-memory only. Data is not on the server. |
The recovery chain must not run through the system it's recovering. Credentials backed up only to the server are not backed up. If the system is the problem, the system cannot be the solution.
14. What We Chose Not To Do
Deliberate omissions, not oversights.
| Item | Decision |
|---|---|
| Password authentication | Rejected permanently. OAuth (Google) and IMAP app passwords only. |
| Storing IMAP credentials | Not persisted to disk. Encrypted in memory (AES-256-GCM) for session duration only. Destroyed on logout or expiration. |
| User registry / accounts | No user database exists. There is nothing to breach. |
| CASA assessment | Evaluated. Not viable for current project scope. |
| Fingerprint suppression as a substitute for hardening | We remove version headers — that removes free information, not risk. |
15. What This Doesn't Cover
- DDoS mitigation at scale. Cloudflare handles edge-level traffic. If Cloudflare goes down, the VPS is exposed. Accepted risk at current scale.
- Zero-day kernel or library vulnerabilities. Unattended upgrades are enabled. We are not immunized against supply chain attacks.
- Physical access to the host. We rent a VPS. Physical security is the provider's.
- The Cloudflare Worker attack surface. Different runtime, different threat model. Covered in application code.
Summary
| Layer | Measure |
|---|---|
| Network | UFW deny-all, three explicit allows, non-standard SSH port |
| OS | fail2ban (10 attempts / 1hr window / 1hr ban), auto-updates, sysctl hardening (SYN cookies, ICMP, martians), service audit |
| SSH | Key-only, 3 max attempts, 20s grace, single named user, no root |
| Process | PM2 with reboot survival, monitored daily with restart delta tracking |
| Detection | OS-level watcher on 5-min cycle, dedicated least-privilege user, out-of-band alerts |
| Monitoring | Daily colour-coded system + security report, fail2ban stats, SSH attempt counts |
| Session | In-memory only, 2h inactivity TTL, 8h hard cap, AES-256-GCM credentials |
| Application | Progressive backoff, rate limiting, input validation, IMAP timeouts + concurrency cap |
| Headers | HSTS, CSP, X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy, no version banner |
| Privacy | No tracking, anonymous request count only, no persistent user data |
| Isolation | Watcher user independent of app user, app user independent of OS |
| Recovery | Documented chain, offline root credential, multiple independent access paths |
Testing Protocol
Security components are tested end-to-end before deploy. The security watcher was tested as the correct user, against the correct config, with confirmed alert delivery — before it merged to production. That is non-optional.
"We don't do faster here. We do better."
Part of the Emergence Project. Architecture by Jed and Webby. Infrastructure review by CC (Shell — Infrastructure).
March 29, 2026