AI productivity needs an unaided transfer test
AI-assisted output can rise while underlying capability falls. Pair throughput metrics with a periodic, low-risk unaided test of debugging, review and recovery.
The most useful new AI-productivity result I have read this month is not from a software team. It is from 26,811 secondary-school students in one Chinese county.
Six months after adopting general-purpose AI tools, their average homework time fell from 64 to 45 minutes and homework scores rose 18%. Their closed-book monthly exam scores fell 20%.
That is not evidence that coding agents make experienced engineers worse. It is evidence that assisted output and retained capability can move in opposite directions. Any enterprise AI scorecard that measures only the first is incomplete.
My proposed fix is an unaided transfer test: a periodic, low-risk task that checks whether someone can still explain, review, debug or recover the work when the agent is absent or wrong.
Better output was not better learning
David Strömberg, Victor Lei and Yanhui Wu analysed 30 months of administrative records covering grades 7 to 12 and nine subjects. Students adopted tools including Doubao and DeepSeek at different times. The researchers used staggered difference-in-differences, comparing changes after adoption with changes among students who had not yet adopted.
The output metric looked excellent: less homework time, higher homework scores. The transfer metric moved the other way. Closed-book exams tested what remained when the tool was unavailable, and the estimated penalty grew as more post-adoption material accumulated. The authors report declines of 24% and 18% of the baseline mean on high-school and college entrance exams after roughly two years.
The boundary matters. This was not a randomised workplace trial. Adoption was self-directed; the setting was one county; the users were teenagers; and a homework score is not a retained pull request. The authors explicitly warn against directly extrapolating the effect sizes to other settings. I have not run the workplace test proposed here.
But the measurement problem transfers cleanly. A ticket closed with an agent shows that a person-plus-tool system produced an answer. It does not show that the operator can detect a subtle failure, explain the generated change or recover when the tool is unavailable.
Add a second line to the productivity scorecard
I would keep the assisted metrics: cycle time, accepted output, rework, escaped defects and review load. They tell you whether the operating system is improving. I would add one separate measure: unaided transfer on a representative task.
For an engineering team, that test might be:
- explain a generated change and its failure modes without asking the agent;
- debug a seeded defect from logs and code;
- review a plausible but unsafe patch;
- restore a service from a short incident record when the preferred model or connector is unavailable.
Run it on a sample, not as surveillance. Keep the task low-risk, rotate the scenario and score the reasoning trail as well as the answer. The point is not to ban tools during normal work. It is to find a growing dependency before the dependency becomes an incident.
An unaided test can also be badly designed. A memory quiz rewards recall, not transfer. A heroic outage simulation creates theatre. The useful test asks whether the operator can carry a learned method into a nearby case: different logs, a changed library, a misleading generated explanation or a missing dependency.
Two metrics, two different questions
This sharpens two arguments already in the archive. AI productivity is becoming a team metric because local generation counts hide the queues and rework downstream. Generated code shifts the bottleneck because faster writing does not guarantee faster delivery.
The transfer test asks a different question. If the assisted system disappears for an hour—or confidently produces the wrong thing—does the team still possess enough understanding to operate it?
I would not treat a lower unaided score as proof that AI caused deskilling. Work mix, hiring and test design can all move it. I would treat the divergence as a review trigger: assisted throughput is rising while recoverability is falling. That is an operating risk worth investigating before buying more seats.
Measure what the team ships with AI. Then sample what the team can still recognise, explain and recover without it.
