TL;DR
When competing AI proposals are ranked by estimated value and funded from the top, the winners tend to deliver less than they claimed. That holds even when every sponsor is honest, because ranking favors the estimates that happened to come out high. The gap grows with the number of close rivals. And a thinly evidenced winner's lead is more likely to be luck. A cut of the same share to every claim cannot change the winner. Moving each claim toward a typical figure, by more where the evidence is thin, can change it. That move also changes how a shortfall should be read.
What the paper develops
The decision
Several AI proposals want the same people and budget. The review ranks them by estimated value, or a score built from it, and funds from the top. It already trims for overselling and may score confidence. The RICE method, for example, multiplies each idea's score by 100, 80 or 50 percent [1]. The belief underneath is that honest estimates, funded from the top, deliver what the list says.
Why honest estimates still fall short
Each estimate is the true value plus an error, as likely to be high as low. The proposal that comes out on top is more likely to be one whose error came out high. So the winners should be expected to deliver less than they claim [2]. Take the worst case, with every option truly equal. The winner's estimate is too high by about 0.56 of an estimate's usual error with two options, and 1.54 with ten [2]. When a thinly evidenced proposal wins, more of its lead is likely to be error.
Why the usual discount leaves the same winner
Cutting every claim by the same share cannot change the order. Confidence tiers do reorder. But they pull every estimate toward zero, not toward what such proposals deliver. In Amazon's 2023 product tests, a correction of this kind cut the estimated total impact of launched features to 0.41 of the plain sum. It also changed the order of product groups beneath the top two [4]. That is one company's product tests. Carrying it to investment proposals is an analogy.
What to change in the review
Before reading the cases, the chair writes a typical figure and range for this kind of proposal. Each claim moves toward that figure. It keeps more of its distance where the evidence narrows the range more. The review ranks and funds on the moved figures. Record the claim and the moved figure at approval. A shortfall against the raw claim is what the ranking predicts. It does not by itself show that a sponsor oversold.
What to do next
Before reading the cases, the chair writes a typical figure for this kind of proposal and the range they would expect. Move each claim toward that figure, keeping more of it where the evidence narrows the range more, and rank and fund on the moved figures. Record both at approval and judge delivery against the moved figure.
Inside the white paper
- Why honest estimates still disappoint
- What makes the gap bigger
- Why the usual discount leaves the same winner
- What Amazon found in its own tests
- How to move the figures
- Reading a shortfall
- The objections
- What to change in the review
Sources and notes
- Sean McBride, Intercom — RICE: Simple prioritization for product managers (Intercom blog, 2018)
- James E. Smith and Robert L. Winkler, Duke University — The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis (Management Science, 2006)
- Ryan Kessler, Amazon — Overcoming the Winner's Curse: Leveraging Bayesian Inference to Improve Estimates of the Impact of Features Launched via A/B tests (Conference on Digital Experimentation at MIT, 2024)