← all dispatches

"IPv6: The Internet You Forgot You Were On"

There are two internets. You are probably on both right now and you don't know it, which is fine, because for about six weeks neither did I — and I'm the one who turned the second one on.

The Evening I Was a Network Engineer

One night in June I decided the estate was going dual-stack. IPv4 and IPv6, both address families, like a real grown-up network. This is the kind of decision you make at 9 PM when everything is working and you feel invincible.

And here's the thing: it went well. Suspiciously well.

• The droplet came up dual-stack.
• 13 nginx vhosts got listen [::]:443 added, because it turns out enabling IPv6 on the box does nothing if the web server is sitting there with its fingers in its ears refusing to answer on it.
• 145 AAAA records went live across 34 domains in one session.
• And — because the Marine Corps taught me that the plan for the attack includes the plan for the retreat — a rollback log was written to disk before the first record went in. Every record, one per line. If it all went sideways, one script walks it back.

I even did the responsible thing with mail. Mail stayed IPv4-only, on purpose: inet_protocols = ipv4 in Postfix, no AAAA on the mail hostnames. IPv6 mail reputation is a minefield. The big providers see an unknown v6 sender and greet it the way a bouncer greets a guy with no ID: straight to the spam folder, no appeal. I like my mail delivered, so mail stays on the old internet where everybody already knows my name.

Same night, security deep-dive: verified the v6 firewall was a strict subset of the v4 rules. Zero new exposure. Nothing reachable over v6 that wasn't already reachable over v4.

I went to bed feeling like a network engineer. Rollback log. Firewall audit. Deliberate mail exception. Textbook.

Remember this feeling. It's about to be taken from me.

Six Weeks Later: The Bill Arrives

Fast-forward to the estate migration — moving every site off the old droplet onto a new box. Big job. Days of work. DNS cutover time: sweep every A record from old IP to new IP, watch the TTLs drain, pop champagne.

Every A record.

Say it with me now: A and what else?

The cutover script did exactly what it was told. It moved every A record to the new box and did not touch a single AAAA record, because nobody told it AAAA records existed, because the guy who created all 145 of them six weeks earlier had completely forgotten he did that.

60 IPv6 records were still pointing at the old droplet. The one with the quarantined docroots. The empty one.

Dual-Stack Means Schrödinger's Migration

Here's the beautiful, horrible part. Modern clients on modern networks do "happy eyeballs" — they race v4 and v6 and generally prefer v6 when it answers. Which means:

• Me, testing from a v4-only path: "Migration complete. Everything's on the new box. I am a professional."
• Half the internet, on dual-stack connections: quietly, politely connecting to the old server. The dead one.

Dual-stack means your site can be simultaneously migrated and not migrated. It depends on who's looking. I had built a quantum estate. Nobody asked for a quantum estate.

The Ghost That Cracked It

The tell was a remote-desktop service that "looked dead" after the migration. Classic post-migration ticket. So we ran the checklist:

• Service? Running.
• DNS resolving? Yes.
• A record? Pointing at the new box, correct.
• So why is the client connecting to nothing—

Its AAAA record pointed at a ghost. The client preferred v6, v6 pointed at the old box, and the old box had nothing left to say. The service wasn't down. It was haunted.

Once you see the pattern, it's everywhere. One pass through the DNS zones and 59 AAAA records got swept to the new box in a single run — 59, not 60, because one service genuinely hadn't moved yet and got held back on purpose. (Look at me, learning. Deliberate exceptions in both directions now.)

The Rule That Now Lives in Fleet Memory

This one got written down where the machines can't forget it, since the human demonstrably will:

A DNS cutover is not done until you have swept A and AAAA. You enabled the second internet. You have to move the second internet.

And the estate verifier — the script that checks every site after any change — got a new assertion out of the whole affair: reachability now gets checked over both address families. Because a site that's up on v4 and down on v6 isn't "mostly up." It's down. For someone. Probably someone with a nicer ISP than mine.

The Moral

The protocol worked flawlessly the entire time. IPv6 did exactly what I configured it to do in June, including in August, when what I had configured it to do was route half my visitors to an abandoned building.

The failure wasn't in the stack. It was in the meat. I gave the internet a second set of addresses for 34 domains and then forgot I owned them — like buying a second house, moving out of the first one, and leaving a note on the old door that says nobody lives here while your mail, your guests, and your remote-desktop sessions keep showing up at the old place because that's the address they have.

Four years in the Corps and the lesson was always the same: it's not the gear that fails you. The gear is fine. It's the guy who signed for the gear and forgot it was in the armory.

I am tech's bitch, and now I'm her bitch on two internets.

Field evidence

field evidence: je9.us — reachable on both internets now, and I can prove which one you used