Bytes & Drafts

Every few years the industry collectively rediscovers the same uncomfortable fact: you can have good engineers, solid testing, careful code reviews, and still ship a distributed system that behaves in ways nobody expected.

If you have ever chased a bug where a system froze every few hours, only to wake up the moment you attached a debugger, you already know that time is rarely as simple as a counter that increments once a millisecond. Timers fire slightly early or slightly late, interrupts sneak in between instructions, peripherals run from their own oscillators, and distributed nodes see each other through links with variable delay. From the outside everything looks nicely clocked. Inside, every part of the stack negotiates its own idea of now.

Language-model pipelines began life as collections of personal tricks. We shipped features only after someone produced the right paragraph of instructions, and that paragraph lived in a notebook or a mind. The knowledge was fragile, undocumented, and always under revision.

We learned this the hard way building a classifier that stubbornly hovered around 50% accuracy for days. After countless attempts and growing frustration, we discovered two innocuous words in our prompt were poisoning the results. Remove them, and accuracy jumped to 70% overnight. The "fix" lived in one person's head, and we had no process for finding it again.

I found this in a 1970s magazine (Popular Electronics, Jan. 1975, page 34) about the Altair 8800. Their definition of software made me stop and think:

The current trade situation between the US and China involves very high tariffs.