Oversight capacity, not model capability, is the ceiling on AI scale

What the paper develops

You were told to scale the AI: add agents, add use cases, get more done with what the models can now do. The plan behind that mandate assumes the limit is the model. For a widening set of tasks, it is not.

What actually caps safe scale is oversight capacity — the finite human attention and judgment available to review, correct, and answer for what the AI does. McKinsey puts it plainly: the scale of agentic adoption will be capped by how much oversight humans can provide, which makes governance itself the bottleneck. Its early-adopter figure, two to five people supervising 50 to 100 agents, reads as impressive until you read it as a ceiling — there is a number of agents past which a team can no longer answer for the work.

The limit you hit first is not in the model; it is the number of decisions your people can still stand behind. Every new agent draws on the same shared pool of human judgment, and most companies have never counted it, so they cross it without noticing. The first proof it was there is an incident nobody had the capacity to catch.

The span of supervision

The number that makes this manageable is the span of supervision: how many agents, decisions, or units of output one accountable human can genuinely own. It is the AI-era version of span of control. It shrinks when stakes are high and errors are hard to spot, and grows when they are low and obvious, so estimate it per workflow rather than assume one number covers everything. The test is simple and uncomfortable: for each agent, can a named human actually say what it did this week and answer for it? When the honest answer becomes no, the span has already been exceeded.

Raising the ceiling

The span is not fixed, and raising it without lowering the standard is the central move. Layered oversight raises it: critic agents that challenge outputs, guardrail agents that enforce policy, and compliance agents that watch regulation, all under named human accountability, with monitoring, red-teaming, and incident response wired in so drift and failures are caught by the system rather than by a person watching everything. And oversight calibrates review to risk: reviewing every action only recreates the bottleneck the AI was meant to relieve, so humans move above the loop — sampling the low-stakes, concentrating judgment where errors are costly or hidden, and escalating the uncertain. Layering raises the ceiling but keeps one — the oversight agents themselves answer to a human and cost some oversight to maintain.

Treat oversight as a portfolio decision

Because oversight is finite and shared, where to spend it is a portfolio decision. Every AI system competes for the same pool, so the question is never whether one deployment is worth doing on its own, but whether it is worth the oversight it will consume given everything else that oversight could cover. The highest-value move is often better allocation rather than more review: shifting oversight off a low-risk system that was over-watched and onto a high-risk one that was under-watched raises safe scale without adding a reviewer. Leaders who feel out of oversight capacity are frequently spending what they have on the wrong AI, so the first move is usually to reallocate rather than to add.

The failure signature

Exceeded oversight shows a recognizable signature before the incident that usually announces it. Ownership in name only: an owner on paper who cannot account for what the system has been doing. Review that has become rubber-stamping: approval standing in for inspection that is no longer possible, which produces a false record that the work was governed. And surprise: incidents surfacing from corners of the estate nobody was really watching. The corrective for all three is uncomfortable because it means slowing something down — reduce the live AI footprint to what oversight can genuinely cover, or add oversight before adding more AI.