215 Mailboxes and the Bouncer Who Kept Arresting the Staff
I run my own mail. Not a mailbox — a mail estate. Thirty-three domains, 215 mailboxes at last count, all self-hosted, because at some point in my life I decided that paying a company nine dollars a month was for cowards and that what I really wanted was to personally own every problem SMTP has generated since 1982.
This month I moved the whole thing to a new server. This is the story of that move, and of fail2ban — the software bouncer that watches the logs and bans anyone acting suspicious — which spent the entire migration doing its job so well that it repeatedly arrested the staff.
The move itself
The migration was, on paper, clean: 33 domains, 205 mailboxes, 2,488 messages, and 34 DKIM signing keys, lifted from the old droplet into docker-mailserver on the new one.
The part I'm actually proud of: nobody's password got reset. Both ends store SHA512-CRYPT, so the hashes moved verbatim. Two hundred and five accounts changed servers and not one human noticed. That's the whole job, right there. The best migration is the one that generates zero support tickets, and my support department is also me.
The DNS cutover for ~30 of those domains was one record. They all MX to the same mail host, so flipping that single A record moved the whole fleet at once. One record. Thirty domains. It felt like pressing the button on a controlled demolition, except the building is supposed to still be standing afterward.
It was. Mostly. Let me tell you about the traps.
Trap #1: the helpful robot
docker-mailserver auto-provisions an empty maildir the instant you define an account. Helpful! Except my first copy pass checked "does this mailbox already exist?" before syncing — and every single one did exist, freshly minted and empty. So it silently skipped 204 of them.
No errors. No warnings. The logs were green. I caught it only because I'm paranoid enough to count the actual messages on both sides:
old server: 2,489 messages
new server: 17 messages
Seventeen. The migration had moved less than one percent of the mail and reported total success. Lesson re-learned for the hundredth time: never trust a tool's exit code. Count the bodies.
Trap #2: the tenant who moved apartments
The mail container took a new internal IP on restart, which broke a config that had pinned the old one. Hardcoding a container's IP is like writing your friend's hotel room number in permanent marker — he's not going to be in room 214 forever. Replaced with a docker network alias, which follows the container wherever it goes. Names, not numbers. I know this. I knew this. I did it anyway.
Trap #3: the resolver that lies
After the DNS flip, the cloud provider's own resolver kept serving the old record for about 43 minutes. So my verification checks said "not propagated yet" long after the entire rest of the planet had the new answer. The verifier now asserts DNS through a public resolver, never the local one. If you're going to ask "did the world get the memo," don't ask the one machine standing in the room where the memo was written.
And now, the bouncer
fail2ban's job is simple: watch the logs, ban anything that looks like an attack. During this migration it went on an absolute tear, and — I need to be fair to it up front — it was correct every single time.
Arrest #1: the PHP container. Twice. The container that renders half my websites lives on the mail network, and during the chaos it tripped a jail. Result: every contact form and every notification email on every site silently died. That's the nasty part — a container banning its own co-worker produces no symptom at all. Forms submit fine. Pages load fine. The mail just... doesn't happen. Nothing looks broken, because the bouncer doesn't announce who's in the back room.
Arrest #2: me. While troubleshooting, I fumbled enough attempts that the SSH jail banned my home connection. I locked myself out of my own server with my own security. In the Corps we called this a self-inflicted casualty, and the paperwork was humiliating. I recovered over the private mesh network — the back door I'd built for exactly no reason I could have articulated at the time — and then wrote a guard cron that enforces the ignore list and auto-unbans internal addresses. The bouncer now has a laminated card with the staff's faces on it.
Arrest #3: the inspector. I built a tool called mailtest — per-domain checks of MX, STARTTLS, certificate-name match, SPF/DKIM/DMARC. Its very first sweep got the server's own public IP banned, because probing a nonexistent recipient across 34 domains is, from the log's point of view, indistinguishable from address harvesting. I wrote a security auditor so thorough it got itself flagged as the attack it was auditing for.
What the inspector found
Once mailtest was allowed back in the building, its first honest sweep came back 29 of 34 domains clean. The other five earned their keep:
- The TLS cert covered only one of five mail hostnames. Every client connecting to the other four had been getting a name-mismatch error, presumably for ages, presumably clicking "accept anyway" like we all swore we'd never do.
- One domain was hosting live mailboxes with no MX record at all. Mail sent to it wasn't arriving — it was bouncing off into the void, and nobody had noticed, which tells you something about how popular those mailboxes were.
- One SPF record still authorized a server that no longer exists. A dead machine, still on the guest list.
All fixed now. That's the deal with self-hosting: nobody finds your embarrassing config but you, and nobody fixes it but you either.
Best bug of the week
Split-brain: the mail admin panel was still provisioning mailboxes on the old box while live MX pointed at the new one. Creating an account in the panel did exactly nothing to real mail. It felt productive. It accomplished zero. I have had jobs like this.
Until it was properly rewired, the mitigation was a giant red banner across the panel, which is the sysadmin equivalent of duct-taping a note to the broken vending machine. It worked. It always works. That's the worst part.
After action report
Final tally: 215 mailboxes on the new server, zero password resets, one bouncer with a perfect record. fail2ban banned the mail system's own components, my test tooling, and me — and every single ban was, by its rules, righteous. I can't even be mad. I built a guard, gave it standing orders, and then got upset when it followed them.
I spent four years in the Marine Corps learning that a sentry who challenges everyone — including the officer of the day — is a good sentry. My server has one. It challenged the officer of the day repeatedly this month.
The officer of the day is tech's bitch, and the sentry gets a raise.