Run a small AI coding productivity study, then audit your own numbers

An AI coding productivity study does not have to be large to be worth doing. Ours took one day, one repository and one question. Does adding a single line to CLAUDE.md change what Claude Code actually does?

We counted nine runs of the same kind of task, across two subjects, and got a clean separation. Seven runs, under a rule worded as a property in the tool's own description, produced zero. Two runs, under a rule worded as an event in CLAUDE.md, produced two. The wording and the location moved together. We published it as Why Claude Code ignores the rules you wrote for it — linked at the foot of this article — and we still stand behind the result.

Then we wrote about it. Three articles.

Eight claims across those three did not survive being checked against the files behind them. Five of the eight were in an article that was already live.

The study itself was a house measurement, with a sample in the single digits. You do not have to settle the AI coding productivity paradox. You do not have to decide whose claims about productivity gains are overblown, either. Knowing what happens in your own repository is a different job — and it is a day of work.

The hard part comes after.

Five practices that make a one-day study worth the day

None of these are ours, and nothing below is an argument against any of them. If you are not doing them yet, start here.

1. Measure a decision you were going to make anyway. We were about to add a line to an instruction file, and wanted to know whether it earned the space. A study attached to a real decision gets finished. A study attached to curiosity gets abandoned somewhere in the middle.

2. Count an outcome, not an impression. The Claude Code productivity metrics worth having are the ones a stranger could recount from your files next month. "Did the agent call the tool" is countable. "Did the session feel smoother" is not, and it will not survive your own memory of the afternoon.

3. Put the denominator next to the numerator, always. Two out of two and ninety-four out of a hundred are both "good". Only one of them tells a reader how much to believe. A bare percentage is where a small sample goes to hide.

4. Check that your isolation actually carried your change. We ran the treatment arm in isolated working trees. Those branch from the default branch, not from wherever you happen to be standing, and our new line was on a branch. It never reached those runs. We came within one check of reporting the result backwards.

5. Write "what this does not show" while the runs are still warm. You will never know the limits as precisely as you do on the day. Writing them before you know which way the result went also removes the temptation to shape them around it.

What the five don't fix: the measurement was right and the sentence was wrong

Every one of those five protects the measurement. Not one of them protects the sentence you write about it two weeks later.

Here it is:

A number is not something you own once you have measured it. It is a claim, and a claim is only as good as your ability to derive it again.

The gap is physical rather than clever. The runs happen once, on one day. The record of them gets written the same day by someone who still remembers what the columns mean. The article gets written later by someone reading the record, which is not the same as reading the runs.

Every hop is a chance to rename a column, merge two rows, or count the same runs twice. We did all three.

Three were wrong before we wrote a word, and five went wrong in the writing

Three of the eight were already wrong in the record, which means the article reproduced them correctly.

The clearest is a fraction with no runs behind it. The record showed one arm scoring zero out of two, and the article printed that as the result of an earlier wording.

The arithmetic said otherwise. The four groups were three, two, two and two, which is nine with no remainder.

So the "zero out of two" appearing in two different tables was the same two runs, counted once as voided and once as a result. There was never a valid pair to report.

The second error hid inside a column we had stopped reading. The results table has a subject column, and one control group of three runs was on a different subject from all the others. Our "seven runs" was quietly summing two subjects.

The wording under test fires on the phrase "measured value". Six of the nine runs were themselves about measured values, so the subject may be doing part of the work. We have not separated the two.

The third error was the sentence we liked best in the whole piece. "Three runs — in different groups, independently, with no prompting — invented a way to mark how confident they were." They were two groups, not three. Two of the three had been handed the exact wording we were testing. And the later of those two had already seen the earlier one's output, because working trees are isolated and a shared board is not.

The remaining five entered while writing, and they are the more embarrassing ones.

One error is a single word in a table header. The record says "followed the instruction: 2 of 2". The draft said "tool called: 2 of 2". Our own limitations section, later in the same article, said that one of those two runs stopped before it wrote anything. I renamed the column, the claim changed underneath the number, and the article contradicted itself in public for as long as it was live.

One error is a phrase that made a file look ignored. The published version said the rule "was still ignored seven times in a row". None of those seven runs had the rule in CLAUDE.md. Five were controls with nothing added, and the line never reached the other two. Seven runs with the rule missing from the file is a different sentence, and a much duller one.

One error is a count of headings, in a different article about handing work to the next session (linked at the foot of this article). A sentence claimed "the same six headings come out every time". The file it describes lists seven items, and they are bullets inside a procedure rather than headings at all. The file is ours. Nobody had opened it.

One error is a date. A draft said a report had been written "a few days earlier". Both commits are dated 5 September and sit sixty-eight minutes apart. Relative time is a number too, and it rots the same way.

One error is arithmetic the reader could have done. A draft of our article on running Claude Code agents in parallel (also linked at the foot of this article) opened with "we needed nine agent sessions" and later said "Three runs were void". Nine was the count that reached the results table; the three void ones were outside it. The real figure was at least twelve, and both sentences were in the same piece.

Four review passes missed it, and the fifth opened the file

The standard answer to all of this is a second reader, and it is the right answer. We had one on every article, working against a written rubric, and it caught a great deal.

It did not catch these.

The six-headings claim and the few-days claim were both present in the first commit of that draft. The third pass called itself a final verification and passed the article. The fourth pass passed it again. The fifth pass found both and blocked publication.

Nothing about the reviewer changed between the fourth pass and the fifth. What changed was where it looked.

The fifth pass opened the file each number came from and counted the items in it. Both claims fell out at once.

Neither is detectable by reading the sentence it lives in, because the sentence is perfectly well formed. It simply is not what the file says.

We got the review wrong too, in the direction that hides it

One pass scored a different article by carrying forward the previous pass's numbers for two of the three sections. The reasoning looked sound. The body had not changed since that pass, so only the newly added section needed scoring.

The result was 111 out of 130, or 85.4 percent, against a pass line of 85 percent.

The next pass scored all three sections from scratch and found two points fewer in the part that had been carried forward. Only one of those two counted against the earlier verdict: the other was a flaw a later revision had introduced, after that pass. 110 out of 130 is 84.6 percent. The article we had passed by half a point had not passed.

The false assumption was not "the body has not changed", which was true. It was "a score is a score". Between those two passes the rubric had been rebuilt from 100 points to 130, and several criteria had changed what they looked at while keeping their names.

We only found it because the earlier pass had written down that it cleared the line by half a point. Without that one sentence, nobody would have gone back.

So the rule we ended up with is blunter than we would like. When the text moves, re-score all of it. When the rubric moves, re-score all of it even if the text did not.

Audit your own numbers in an hour

This works on anything you have published. It does not need to be AI coding productivity metrics research, and it does not need to be about agents at all.

1. 1. List every number mechanically, not from memory. grep -oE '[0-9]+([./][0-9]+)?' draft.md | sort | uniq -c takes a second. Then add the words that are numbers in disguise: a few, most, every, the same six, last week.
2. 2. Re-derive each one from the source, not from your notes. Open the file the number came from and count the rows yourself. Your notes are a copy, and the copy is where the first error already is.
3. 3. Make the parts add up to the whole. Add your groups together. If they already reach the total with no remainder, then a figure that looks like it has its own runs behind it does not. Ours was 3 + 2 + 2 + 2 = 9.
4. 4. Read every table header, figure label and caption as a sentence. A column head is a claim with no verb. "Tool called" and "followed the instruction" are different assertions about the same two runs, and only one of them was measured.
5. 5. Diff your assertions against your own limitations section. Both were written by you, about the same runs. If they disagree, one of them is wrong, and you can settle it without leaving the file.
6. 6. Convert every relative date to an absolute one. git log --date=short is faster than remembering, and it does not flatter you.

One question is worth more than the list, and it is not "is this number right".

If a stranger asked me to produce this number again tomorrow, which file would I open, and what would I count?

If you cannot answer that in one sentence, the number is not yet yours. Nothing else on the list will help until it is.

What this does not show

What would break this

• Someone runs the six-step audit on an article of ours we have already checked and finds a ninth claim. Then the list is not sufficient, only cheap, and we would want to know which step was missing.
• Someone shows that a careful re-read finds this class of error reliably. Our whole point is that these are invisible from inside the sentence, and one clear counterexample would demote it to a habit rather than a finding.
• The arithmetic argument turns out to be wrong — an execution log surfaces showing a valid pair of runs behind that zero out of two. Then our first correction was itself an error, which would be the most useful outcome on this list.

All three are cheap to attempt, and we would rather be corrected than quoted.

The rest of this is what we do about it. A number written into prose has no field for the date it was measured, and none for whether it is still provisional. So ours sat in a published article, looking exactly like a settled fact.

We are building the other shape. Vondet (vondet.com) is a plan and decision log that agents read and write directly. A measured value there carries its date and its status, so nobody has to infer either from the confidence of the sentence around it. Sign-up is not open yet — there is a waiting list, and for now that is deliberately all there is.

Before any of that: take the last thing you published that had a number in it, open the file that number came from, and count. Tell us whether the count matched, and especially if it did not.