Two Names, One File, 195 Gigabytes
Some outages come in screaming. This one came in looking like bad Wi-Fi.
It was October 10th. Scripts on my Mac started hanging. API calls just sat there. Background jobs exited with no output at all, which is the computer equivalent of a private who stops answering roll call and nobody notices for an hour. Nothing was erroring. Everything was just stopping.
My first instinct, like everybody's, was the network. It's always the network. Except it wasn't.
The disk was full
Not "getting tight." Full. A 926 GB drive with 249 MB free. When a disk fills up, writes don't always fail loudly. A lot of them just block and wait for space that is never coming. So every unrelated process on the machine started freezing at the same time, and the only thing they had in common was that they all needed to write something down.
Where did the space go? One file. 195 gigabytes of it, grown in about forty minutes.
The line that did it
Here's the whole crime, one line, putting together a release-notes file:
cat whatsnew_common.txt whatsnew_ios.txt whatsnew_tail.txt > whatsnew_IOS.txt
Read it the way a human reads it. Take three files, glue them together, write the result to a new file. The new file's name has a capital IOS so you can tell it apart from the input with the lowercase ios. Totally reasonable.
Now read it the way the Mac reads it.
macOS formats its drives with APFS, and by default APFS is case-insensitive. To the filesystem, whatsnew_ios.txt and whatsnew_IOS.txt are not two files. They are one file with two spellings.
So the shell sees > whatsnew_IOS.txt, opens that file for writing, and empties it before anything runs. Then cat) gets to its second input, whatsnew_ios.txt, which is the same file it's writing to. It reads a chunk and writes it to the end. Then it reads the chunk it just wrote and writes that. Then it reads that. The end of the file keeps moving away faster than cat can reach it.
A dog chasing its own tail, with 926 GB of room to run.
Why it took so long to see
The symptom pointed everywhere except the cause. Hung network calls, silent job deaths, scripts that never came back. None of those messages say "disk full." You have to go looking.
So this is the first thing I check now when several unrelated things start failing at once, before the network, before blaming the code:
df -h /System/Volumes/Data
One command, two seconds. If the number at the end is tiny, stop everything else. That's your outage.
What got written into doctrine
In the Corps you learn that most injuries don't come from the enemy. They come from your own gear, used carelessly, in a hurry, by someone who knew better. This was that. Nobody attacked me. A tidy little command did exactly what it was told on a filesystem that heard something different.
The rules that came out of it are short:
- Never make an output filename by changing only the case of an input's name. Use a name that is actually different.
notes_ios_full.txt, notnotes_IOS.txt. - Never redirect a command's output into one of its own inputs. On a case-insensitive disk, "a different spelling" counts as the same file.
- When a long job runs, watch its output size, not just whether it finished. A file that's growing too fast is a warning you can see.
- When unrelated things fail silently at the same time, check the disk first.
It's in the standing orders now. The external drive's own instructions say it in plain English: never loop-copy a file onto itself.
The fix was deleting one bloated file and changing one capital letter in one filename. The lesson cost 195 gigabytes and a long afternoon of blaming the Wi-Fi.
Check the disk. Then blame the Wi-Fi. Semper Fi.