TL;DR
Ask people how much faster AI made them and they can give you a number. In a careful randomized trial that checked that number against a clock, it ran well ahead of the work. Judge the tool on what it delivered, measured against a baseline, and keep self-reported speed in a record of its own.
What the paper develops
Ask people how much faster AI made them and they can give you a number. That number is an estimate of how the work felt, not a measurement of what it delivered, and the two can be far apart. When a program is funded, scaled, or defended on a self-reported speedup alone, the organization has let a feeling stand in for evidence.
The clearest test is a 2025 randomized controlled trial by the research nonprofit METR. Sixteen experienced open-source developers completed 246 real tasks, with and without AI. Beforehand they expected a 24 percent speedup. Afterward they still believed they had gained 20 percent. The clock said they took 19 percent longer. The trial does not show that AI slows people down. It measured expert developers, on projects they knew well, with early-2025 tools. What it shows is one documented case where capable people were wrong about their own speed and stayed wrong after the work was done. A leader should treat a self-reported speedup as an estimate, and judge the tool on delivered output against a baseline.
METR’s later study, with newer tools, estimated that AI-allowed tasks took less time, with ranges wide enough to include no change. METR calls that estimate unreliable, because developers who did not want to work without AI left the study. It believes developers are likely more sped up now, on very weak evidence. Do not quote the early-2025 figure as a rate. It remains the only trial this paper cites that checked a feeling against a clock, and there the feeling was wrong.
A survey by Google Cloud’s DevOps Research and Assessment (DORA) program looks like a second match and is not. In 2024, respondents who used AI more reported higher individual productivity, flow (sustained focus), and satisfaction, while the report’s findings on delivery stability and throughput pointed the other way. Both sides of that split are what respondents answered, so it compares two sets of survey answers, not a feeling against a measurement. A year later, the 2025 report linked AI adoption to higher throughput, with delivery instability still rising.
What the sources support together is narrower than “people overrate AI.” The feeling is not a reliable stand-in for the output, and nobody can say from outside which way it errs in a given workflow. An organization scaling AI on a self-reported speedup alone is scaling on sentiment, with no way to tell whether the sentiment is too high or too low.
What to measure instead
The alternative is ordinary, and it has three parts that work in order. First, capture a baseline before the tool changes the work. Then choose a work-output signal that reflects delivered value, such as cycle time, rework, error rate, or finished units meeting standard, and choose it so that using the tool more cannot move it by itself. Finally, review felt experience and delivered outcome in separate columns. When the columns agree, use can grow. When they diverge, the review has found the case a satisfaction survey would have counted as a win.
The tool may already be live by the time anyone asks for a baseline. There are three ways to get a comparison, listed in this paper’s judgment from strongest to weakest: keep a comparable group on the old way for a set period, switch the tool on in waves and compare the waves, or rebuild the before-picture from work records and call it an estimate. If none is possible, the output column reads “not measured.” That is different from a weak result, and expansion waits unless the organization would accept it on the assumption that the gain is zero.
The hard part is deciding which numbers do not count. Adoption rate, active users, prompts per employee, and self-rated speed can all rise while the work stays the same. They belong on an adoption dashboard, not in the evidence a portfolio uses to deepen use. A contributed opinion column in CIO magazine makes the distinction plainly: governance machinery can sit above the work, while the value is created or lost inside the workflow.
Objections, and what measuring costs
The standard has real objections. People can work to an output number, so choose finished units that met the quality bar and read a sample of the work itself. Measuring costs time, and no source prices it, so apply the standard where spending is large or a decision to scale is near. A log of who uses the tool is not a measure of what it delivered, and tracking individuals carries a surveillance cost. Results also age, because the tools change, so measure again at the next decision to expand.
None of this slows adoption. It lets a leader say yes to deeper use on evidence, and redirect rather than abandon a tool whose value is leaking inside a workflow. Felt productivity is real and worth having. It is simply not the same thing as measured productivity, and only one of the two can tell a portfolio where its money went.
What to do next
Before scaling an AI-assisted workflow, require three things. First, a baseline, taken before the tool or rebuilt from a held-back group, staged waves, or work records. Second, an output measure that heavier use cannot inflate. Third, a review that keeps felt speed and delivered output in separate columns.
Inside the white paper
- What one trial and its follow-up show about the gap between a feeling of speed and the clock
- A three-part standard, and three ways to get a comparison when the tool is already live
- The objections to measuring (gaming, cost, surveillance, results that age) and a one-page record
Sources and notes
- Joel Becker, Nate Rush, Beth Barnes, and David Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, July 10, 2025 — Sixteen experienced developers, 246 real tasks, early-2025 tools: they believed AI had sped them up 20 percent and took 19 percent longer. One small randomized trial, not a general claim about AI.
- METR, We Are Changing Our Developer Productivity Experiment Design, February 24, 2026 — METR’s follow-up: a second study with newer tools estimated less task time with wide ranges, METR calls it unreliable because of who left the study, and it says self-reported speedups can be quite unreliable.
- Google Cloud DORA, Accelerate State of DevOps Report 2024, Google Cloud, 2024 — DORA 2024, a survey of nearly 3,000 professionals. More AI use went with higher reported individual productivity and with lower delivery stability and throughput. Both sides are survey answers, so it is not a feeling-versus-measurement contrast.
- Google Cloud DORA, State of AI-assisted Software Development 2025, Google Cloud, 2025 — DORA 2025, a survey of nearly 5,000 professionals. AI adoption is now linked to higher delivery throughput while instability still rises, so the delivery-level answer changed within a year.
- National Institute of Standards and Technology, AI Risk Management Framework (AI RMF 1.0), Core: the MEASURE function, NIST, 2023 — The NIST AI RMF Core treats measurement as a function that gives management decisions a traceable basis.
- Maria Korolov, Why Is It So Hard to Measure the ROI of AI?, CIO, Jul 8, 2026 — A CIO magazine feature quoting practitioners on starting without a baseline and on measurement being lacking. Practitioner accounts, not research.
- Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, Generative AI at Work, NBER Working Paper 31161, 2023 (Quarterly Journal of Economics, 2025) — An AI assistant introduced in stages to 5,179 support agents at one company, with output measured as issues resolved per hour. Cited for the staged-rollout method, not for the size of the effect.
- Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Tom Zimmermann, Brian Houck, and Jenna Butler, The SPACE of Developer Productivity: There's More to It Than You Think, ACM Queue, Vol. 19, No. 1, 2021 — Why productivity cannot be captured by a single metric or an activity count.
- Pete Johnson, The AI ROI Gap Isn't a Model Problem. It's a Workflow Problem, CIO, Jul 8, 2026 — A contributed opinion column in CIO on separating governance activity from evidence that task-level work improved.