All four times, the numbers were right

The arithmetic was right all four times. And all four times the verdict flipped.

How correct arithmetic produces a wrong verdict

I did not miscount. The addition and subtraction were exact.

Every flip came from what the number was compared against.

A number alone says nothing. “Twelve seconds” is neither good nor bad. It needs a comparison to mean anything.

And pick the wrong comparison and however exact the arithmetic, the conclusion is wrong.

The ways a comparison goes wrong

Patterns from the four cases:

How it breaks	Example
No comparison at all	Ours is faster — than what?
The two sides measured different things	One includes setup time, the other does not
The environment differed while measuring	One ran with a warm cache, the other cold
The improvement came from elsewhere	Changed the tool and changed the hardware at the same time

None of the four involves arithmetic. And none of the four is visible from the numbers.

The most common one — measuring only your own thing

The recurring theme of this section.

With no control, you interpret whatever comes out. And interpretation leans toward what you already believed.

Measure only your own thing with no control and you confirm what you already believed.

Not dishonesty. When a number is ambiguous a person chooses how to read it, and the choice is unconscious.

So the prescription

Deliberately keep a control that receives no treatment.

Introducing automation, deliberately run 10% the old way. That produces a comparison value from the same period under the same conditions.

It looks like a loss — 10% does not get the improvement. Without that 10%, you never find out whether the other 90% genuinely improved.

And an estimate is labelled as one

Some values cannot be measured. Then this is written:

What goes down
The estimate
The error range
The label “estimate”
What would make it a measurement

The last line matters. Without it the value stays an estimate forever, and given time the label falls off and it reads as fact.

The line this piece leaves

The arithmetic being exact and the verdict being right are different things. All four times, the arithmetic was exact.

The detailed record starts here

Measurement usually goes wrong somewhere other than the arithmetic.

Four of my verdicts reversed over the last few months, and all four times the arithmetic was right. The scripts counted correctly and the logs recorded correctly. What was wrong was what I measured against.

Without a control, measurement confirms what you already believe

I picked up a criterion from someone else’s rule set — match the form to the failure — and ran it over my own rule documents. Prohibitions outnumbered recipes by 12.6 to 28.6 times, with recipes at effectively 0%. I wrote “my corpus is abnormal” and adopted it as the top item.

Their own rule already said what to do here.

Always include a no-guidance control. If the control doesn’t exhibit the failure, there is nothing to fix — stop, and don’t author the guidance.

My control was the reference implementation. I ran the same script over it. The reference implementation as a whole came out at 26.6×, and the document making the argument sat at 18.5×. Mine were less skewed.

The numbers were right from the start. There was no baseline. And with no baseline, I reached a conclusion anyway.

Ask whether there is anything to improve before asking how much

Evaluating a context-compression layer, I set out to measure what adopting it would gain. First I measured my own side. Cache reuse 92.5–96.3%, uncached input 1.8–2.0 tokens per call.

There was nothing left for compression to compress. The tool wasn’t bad — there was no problem on my side.

Ask “how much does this improve things” first and it always looks like an improvement. The order has to flip — is there anything to improve.

Totals hide the cause

One run of 126 cases took 937 seconds. I believed shader compilation was the bottleneck.

Instrumenting each segment gave a different picture.

Segment	Time	Share
Bridge round trips	302s	32.3%
└ one tree dump inside that	186s	19.8%

It fetches the whole tree every time it makes a call. Shaders weren’t even near the top.

A total conceals the cause. Until I measured segment by segment my guess had nothing behind it — and I didn’t know it had nothing behind it.

Counting inventory is not measuring

Holding my own operation against someone else’s four-rung ladder, I wrote equivalent into four cells. The baseline behind all four was whether it was installed.

The arithmetic was right here too. Measured with an instrument that asks “is this installed”, those four cells genuinely are equivalent, and I applied that instrument correctly. What was wrong was the instrument — I judged “it runs” with a tool that measures “it exists”.

Pressing the last cell for real returned no items, no files, no observer hooks. Possession is not operation. And unless the instrument changes, the other three cells stay exactly as they are — never pressed, still written down as equivalent. A value read off the wrong instrument is wrong in the same direction whether you read four cells or forty.

Which is why unmeasured numbers get a label

That is how the convention came about. Anything I can’t measure is published as an estimate with an interval and an [estimated] label, and to promote it to measured, 10% is held back untouched.

One recent adoption sits in exactly that state. Declaring a quality bar in one word inside a prompt — the output does look better, but whether that came from the word or from the other conditions was never separated. With a sample of one there is no interval to give.

So it is adopted, labelled [estimated], and the promotion path is written next to it — build 10% of future reports without that sentence and compare.

What I didn’t do

I have never once run the holdout. Writing the method down and having measured it are different things. One of the five sections above is still [estimated], and it went into this piece in that state.

And I never finished applying the control rule to myself. One of four “equivalent” cells has been pressed. The other three are still ledger entries as I write this.

Verdict

The four misjudgements share something. In every one, my own numbers came out well, and there was nothing to compare them against.

A control, the baseline before it, segment decomposition, and pressing the button. Four names for one job — making a place where an answer different from the one I already believe can appear. Without that place, measurement becomes a confirmation step, and confirmation steps have never once been wrong.