Average accuracy doesn't tell you what you could lose

What the paper develops

In June 2021, Zillow announced a more accurate home-value model: a 6.9 percent median error, the typical miss, across more than 104 million homes not listed for sale. The release said Zillow's growing confidence in the Zestimate's accuracy had led it to start making live cash offers. Less than five months later, Zillow said it would shut the program down. It had bought 9,680 homes in one quarter and written down the value of the homes it held by $304 million.

Zillow's shareholder letter gives other reasons too: price forecasts that missed by more than it had modeled, and more purchases than its renovation and resale capacity could absorb. It also says a larger share of owners accepted its offers than it had seen before, because it was paying more than it later expected the homes to sell for. The letter never mentions the 6.9 percent. That number covered 104 million homes. Zillow's money sat in the homes whose owners said yes.

The people who say yes are not a random sample

Suppose a model prices used vans $1,000 too high or $1,000 too low, half each, so its average error is zero. A company offers the model's price minus $500 and plans to keep the $500 as its margin. The owners know their vans. Owners of vans priced too high say yes, and the rest say no. The company buys the overpriced half and loses $500 on each van, though the average error never changed.

Two studies look at the same pattern. In a simulation of a home buyer in Oslo, expected profit fell from about 6 to 8 percent to about 0 to 1 percent once owners were assumed to accept overpriced offers more readily than underpriced ones. In real purchases by iBuyers, companies that make fast cash offers on homes, profits fell as the companies moved into areas where a standard price model missed more.

Your own approval rule hides errors too. A lender sees how approved loans turn out, but not how refused loans would have turned out. A report built from your own results describes the cases you chose.

Measure the cases you accept

Treat the accepted cases as their own group and track three numbers for it. The acceptance rate against plan is the only one you see before outcomes arrive. Error is how far the estimate was from what happened. Loss is the money actually lost after margin and costs.

Cap what you commit at signing

Outcomes arrive after the commitments, so set the limit when you sign. The leader who would bear the loss sets a cap on the accepted group, in cases or dollars, and writes down the limits for all three numbers before raising it. Raise the cap one step at a time, and lower it or stop when any number leaves its limit.

Ask for the number on the accepted cases

Pick one workflow where a model's estimate sets a commitment that someone else can refuse. Write down who says yes, compare the accepted cases with the declined ones where you can see them, and set the cap and the limits.

Zillow's June release said how accurate the Zestimate was on 104 million homes. It did not say how accurate it was on the homes whose owners said yes. Ask for that number before you sign the next increase. The white paper has the evidence, the sources, and the full method.