Sixty-Two Sites Walk Into a New Server
I run sixty-two websites. Not for money. Not for a company. For me, because at some point "I'll just register one domain" metastasized into a small nation-state of hobby sites, and nation-states need infrastructure, and infrastructure needs a landlord, and the landlord is me, and I am tech's bitch.
This month the whole nation moved. New droplet, four days, six phases: 29 static and PHP sites, 2 WordPress installs, 9 Node apps upgraded to Node 22 while they were in the moving truck, an LDAP directory with 198 identities, 10 SSO apps, a 6.5GB PDF library, and 7.9GB of music. Sixty-two hostnames picked up and set down somewhere else.
Here's the part that matters: the migration did not test my sysadmin skills. Those held up fine. It tested my definition of the word verified. That word did not hold up fine.
The rules of the move
In the Marine Corps you don't throw anything away until someone with more stripes than you says throw it away. My migration ran on the same doctrine: quarantine, soak, then delete. Nothing on the old box got deleted during the move. Sixty old docroots — 16GB — went into quarantine and sat on ice while the new box proved itself. Delete is the one command with no undo, so delete goes last, and delete asks permission.
That rule saved me zero times this move, which is exactly how many times it's supposed to save you. It's a parachute. You pack it anyway.
The 15GB git repo
The new server keeps its config in a git repo, because config that isn't in git is just a rumor. During the migration frenzy, my commits got greedy and swept up 75,358 bulk files — the music library, the PDFs, and, in a personal highlight, a live database. The repo hit 15GB. Git is many things but it is not a moving truck.
Small mercy: none of it had reached the remote. Excluded the bulk, ran garbage collection, and watched 15GB become 95MB. That is a 99.4% reduction, achieved entirely by removing things that should never have been there, which is also how most of my life improvements work.
Nine hours of public plumbing
Now the confession.
For roughly nine hours — 01:48 to 10:34 UTC — one migrated site publicly served a PHP stack trace instead of a webpage. The automated rewrite that pointed sites at the new database host matched exactly one PHP connection style. PHP, it turns out, has at least nine ways to open a database connection, and 54 files on that site used the other eight. Those files woke up on the new box, reached for a database host that wasn't there, and screamed their internals at the open internet.
And here is the verification gap that let it ship, framed and mounted for your inspection:
I checked HTTP status codes. A PHP fatal error returns 200.
Two hundred. OK. Everything's fine. My monitoring looked at a page that was actively disemboweling itself and said "site's up, boss." By that standard a building on fire is fine as long as the front door opens.
The damage audit, because you always run the damage audit: 25 requests from about 20 IPs, every one of them automated scanner noise sniffing for wp-login and friends. The password never appeared in the output — PHP redacts it in traces, a courtesy I did not deserve. The exposed database username was renamed and the old account locked the next day. Then I replaced the status-code sweep with a content-level crawl of all 62 sites — actually read the pages, grep the bodies for Fatal error and its cousins. Checking the door AND the fire.
HTTP 200, zero data
The analytics site died a second, sneakier death, and this one didn't even have the decency to error.
MySQL views carry a DEFINER — the account they execute as. My views' DEFINER was an account that existed only on the old box. On the new box, every view resolved to "who?", and the API returned HTTP 200 with a beautifully formatted payload of nothing. Status: perfect. Content: void. It's the migration in miniature — every failure wearing a green checkmark like a stolen uniform.
Rebuilt all five views as SQL SECURITY INVOKER and the numbers marched back in: 23,015 visits, 114 countries. Sixty-two hobby sites, 114 countries. The scanners of many nations love me.
The corpse that answered IPv6
One more. The DNS cutover swept every A record to the new box and forgot AAAA records entirely. Sixty IPv6 records still pointed at the old droplet. Any client that preferred IPv6 was happily visiting a corpse — and because the corpse was still technically breathing during the soak period, it even answered. I had built a museum exhibit of my old server and only modern operating systems could see it.
What the word means now
Every one of these failures passed my checks. That's the whole story. The checks were the bug.
So the checks got rebuilt into a harness — estate-verify, which grew from 38 to 49 assertions and runs every fifteen minutes. Not "did it return 200." Row-count floors, so an API serving confident emptiness gets caught. Live credential auth tests, so a dead database account gets caught. Source-leak scans, so a page wearing its stack trace on the outside gets caught. Every outage I found by hand that week would have been caught by the harness that week's failures paid for. Each bug leaves a check behind, like a rank insignia for a fight you lost.
On August 4th, the final act: 40GB archived to an external SSD, completeness audit run against the archive — because an unverified backup is a wish — and the old droplet decommissioned. Sixty-two hosts now answer from the new box, verified at the content level, by which I mean actually verified, by which I mean I finally learned what the word costs.
The estate is fine. The landlord is humbled. The machines, as always, are keeping score.
