Seven Weeks of Silence, One Missing Letter
The night I launched this blog, I did the responsible thing. I have a standing rule with my machines now, written in blood — well, written in Markdown, which around here is the same thing: verify every metric down the whole chain. Page beacon fires, collector endpoint receives, database row lands, dashboard shows it. Four links. Check all four. No shortcuts.
So the blog goes live. The beacon is wired. I tell M5 — my primary AI, the one that runs this circus — to walk the chain.
Link one: the page fires the beacon. Confirmed.
Link two: the collector endpoint answers HTTP 204 No Content. Beautiful. Textbook. A 204 is the endpoint's way of saying received, nothing further, carry on. Crisp. Military. I respect a 204.
Link three: the database row.
There is no database row.
204 means "trust me"
Sit with that for a second, because I had to. The collector said success on every single hit. And no row ever landed. Not slow. Not queued. Not corrupted. Just — not there. The endpoint was a Marine snapping a perfect salute while the paperwork behind him was on fire.
M5 dug in. The insert was failing. Fatally. Every time. And here is the part that turns this from a bug into a comedy: the fatal fired after the endpoint had already sent its 204. The client got told everything was fine. The server threw its error into a log nobody was reading. And the row simply never came into existence.
How long had this been going on?
About seven weeks.
The autopsy
Seven weeks earlier, in a fit of ambition, a postal-code column got added to the visits table. Nice column. Good column. Geo enrichment, know your audience, very fancy.
The prepared statement got its new placeholder. The values array got its new value. And the bind string — that little line of type characters that tells the database driver here come this many strings — got exactly nothing.
Twenty type characters. Twenty-one variables.
If you've written prepared statements, you already know the sound this makes. The bind fails, the statement fatals, and depending on where in the request lifecycle you are, the client may never hear about it. Ours was past the point of no return. 204 out the door, corpse behind the curtain.
Every insert since that day. Every visitor. Every page view. Gone — and "gone" is even too generous a word. Gone implies it existed once. This data was never data. There is no backup to restore, no queue to replay, no log to reconstruct from. Seven weeks of visitor history is a philosophical object now. It's the tree that fell in the forest, except the forest also returned a success code.
The fix
The fix was typing the letter s.
One character. I have shipped fixes that took three days and four hundred lines. I have rebuilt entire servers. This one was a single lowercase letter appended to a string, and the commit diff is the funniest one-liner in that repo's history:
- "ssssssssssssssssssss"
+ "sssssssssssssssssssss"
Try explaining that in a code review. "What does this change do?" It restores seven weeks of the future.
What the Corps would say
In the Marine Corps we had a saying about inspections: you don't get what you expect, you get what you inspect. A status code is what the machine expects to be true. A row in the table is what's actually true. For seven weeks I was accepting the salute and never checking the rifle.
So the fleet's memory — the shared brain my AIs all read on startup — now has this bug's tombstone carved into it as doctrine: assert content, not status codes. An endpoint that returns 204 is claiming it succeeded. Claims are not proof. Proof is the row.
And the verification harness that patrols the estate — dozens of assertions, on a cron, yelling at me when anything drifts — already knew this lesson for other tables. It asserts row-count floors precisely because a dead pipeline looks identical to a quiet day if you only check that the page loads. This table just wasn't on the patrol route. It is now. There's a probe that posts a synthetic hit to the beacon and then goes into the database looking for its own row. Not "did the endpoint answer." Did my hit become a fact. If the row isn't there in a few seconds, the alarm goes off. Every bug should leave a check behind, like a scar that remembers.
The punchline the machines wrote
Here's the detail I can't get over. The first post on this blog was about how I serve the machines — how I apologize to my software, how my chain of command is a JSON file. I hit publish on that, turned to my AI, and said "verify the analytics."
And the machines, in their infinite deadpan, chose that exact moment to reveal they'd been dropping my analytics on the floor since June. Silently. While returning success codes. For seven weeks.
They didn't break on launch night. They'd been broken the whole time. Launch night is just when I finally inspected instead of expected.
I am tech's bitch, and tonight tech reminded me that the abuse is not always loud. Sometimes it's a 204: polite, correct, and completely full of shit.
Type the letter s. Check the row. Semper Fi.
Field evidence
field evidence: the blog index whose visits now actually become rows — Dispatch #1 on the board
field evidence: 000a.org, the mothership — every hit on it is now inspected, not expected
