Work, adoption & judgment · Field note

AI should make people better thinkers, not just faster producers

TL;DR

A polished AI answer can hide who formed the view, tested it, and owns the final decision. The evidence for a fix comes from students, so the paper proposes a one-quarter pilot.

What the paper develops

Suppose your team now turns in AI-assisted work faster. One recommendation reads well and reaches review early. A leader asks which assumptions carry the decision and what evidence would overturn it, and no one can answer without opening the chat again. The tool did more than help write the analysis. It supplied the first view, the challenge, and the wording of the decision.

This paper's position is that, for work that depends on judgment, people should form and defend a view before AI answers. The evidence is thinner than the position. The strongest results come from students, and no large workplace trial has tested the idea on experienced people doing their own expert work. So the position is the paper's inference, not a finding, and the paper ends with a one-quarter pilot.

The same model can support a different habit

Two randomized trials show why the interaction matters. Bastani and colleagues studied nearly 1,000 high-school students. Students with a standard AI tool later scored 17 percent below a no-AI group on an exam without the tool. A guarded version of the same model led students through hints and largely avoided that loss. Kestin and colleagues tested a custom AI tutor in introductory physics at Harvard. The AI group had more than twice the median learning gain of a class using active learning, in two topics over two weeks.

The trials used different students, subjects, and designs. Neither tested the first-view step described below or experienced workers, so neither proves a workplace will get the same result. The closest test with working adults is a randomized experiment with 52 software developers learning a new library. Those who used an AI assistant scored 17 percent lower on a quiz about it. Developers who asked for explanations tended to score higher than those who handed over the coding, but they chose their own approach, so that part was not randomized. It was one task, about an hour of work, and a preprint.

A review step needs skill and ownership

The usual phrase, “human in the loop,” is too weak on its own. It does not say whether the person can spot a poor assumption, check the evidence, or reject the answer. It also does not say who makes the final decision. A late signature is not the same as review.

Workplace evidence makes that distinction useful. In a field experiment with 758 consultants, AI users did better on tasks inside the model's tested range. On one task outside that range, they were 19 percentage points less likely to reach the correct answer. People whose results fell tended to accept the AI output and question it less. In a study of medical students, a wrong AI explanation sounded as convincing as a right one.

Make people form a view before they ask

For high-judgment work, start with a short pre-mortem, adapted from Gary Klein's team technique of imagining a plan has failed. Before opening the tool, the worker states the answer they expect, names the assumptions that must be true, and says what evidence would change their mind. AI then challenges that first view. The worker checks the challenge and owns the conclusion.

A reviewer who knows the subject then reads the record and checks four things. Was the first view written before the tool answered? Are the assumptions specific enough to be wrong? Did the worker act on each challenge? Does the final decision name its owner and say what changed? The reviewer does not decide whether the conclusion is right. The reading takes about ten minutes, as a planning estimate. This scoring is the paper's design, and no study has tested it.

Support the habit in three places

Training should teach people to test assumptions and evidence, not just write prompts. IT and operations should give them a safe, usable tool and require stronger checks when the cost of error is high. Managers should ask what changed the person's view and who makes the decision. If any one of those pieces is missing, deadline pressure is likely to push the work back toward ask, receive, and paste.

Pilot it for one quarter, and expect objections

Choose one function where judgment already matters, such as strategy, risk, client advice, or product decisions, and where someone can read a record with knowledge of the subject. Name one executive owner. For one quarter, require the three-step pre-mortem on selected work. Compare speed, rework, errors, and review quality with the function's earlier quarter. That comparison can show cost and fit, not proof.

Expect objections, among them that the step slows the work, senior experts may reject it, and a written first view may anchor people. Plan for them: the owner and the most senior reviewers use the step first, the pilot tracks how often the final decision differs from the first view, and it stops if neither rework nor errors fall. Then decide whether to expand, change, or stop. AI should help people think better, not make their thinking disappear behind a polished response.

What to do next

For high-judgment work, have people state their first view, key assumptions, and what would change their mind. Then let AI challenge it. Pilot it for one quarter in one function.

Inside the white paper

  • What the student trials and a small developer experiment show, and what no large workplace trial has tested
  • Why a human signature is not the same as skilled review, and how a reviewer scores the reasoning
  • Five objections, and how to pilot a three-step pre-mortem in one quarter of real work

Sources and notes

  1. Microsoft and LinkedIn, 2024 Work Trend Index Annual Report. Survey of 31,000 people, 31 markets, Feb–Mar 2024. The 78% figure for bringing your own AI tools includes any personal tool use supplementary to employer-provided tools and may overstate fully unsanctioned behavior. Only 39% of AI users had received AI training from their company — this survey reports widespread use of personal AI tools at work and limited company training among AI users.
  2. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakci, O., and Mariman, R. (2025). Generative AI without guardrails can harm learning. PNAS, 122(26), e2422633122. Pre-registered RCT; ~1,000 students, 9th–11th grade; Turkey; IRB-approved. August 2025 correction updated one author's affiliation only; findings unchanged — this pre-registered student trial compared standard AI, a guarded tutor, and no AI.
  3. Kestin, G., Miller, K., Klales, A., Milbourne, T., and Ponti, G. (2025). AI tutoring outperforms in-class active learning. Scientific Reports. RCT with crossover design; 233 enrolled, 194 eligible Harvard undergraduates; introductory physics; fall 2023. Effect size 0.73–1.3; p < 10⁻⁸. Critical limitations: selected population; two topics; introductory material; authors do not presume the result extends to complex synthesis and higher-order critical thinking — this student trial compared a custom AI tutor with active learning in two introductory physics topics.
  4. Shen, J. H., and Tamkin, A. (2026). How AI Impacts Skill Formation. arXiv:2601.20245 (v2, February 2026). Preprint, not peer reviewed; the authors are affiliated with Anthropic. Randomized experiment; 52 software developers (26 with an AI assistant, 26 without), recruited through a crowd-work platform, most professional or freelance, learning a Python library in about one hour. AI group scored 4.15 points lower on a 27-point quiz (a 17 percent difference; d = 0.738, p = 0.01); no statistically significant speed-up. The higher-scoring interaction patterns were identified afterward and were not randomized — this randomized preprint of 52 developers learning a new library found lower quiz scores with an AI assistant; the interaction-pattern comparison was not randomized.
  5. Dell'Acqua, F., et al. (2026). Navigating the jagged technological frontier. Organization Science, 37(2), 403–423; first circulated as HBS Working Paper No. 24-013 (2023). Pre-registered; n=758 BCG consultants. "Blindly adopt" wording is from the working paper; its retainment measure (copying AI output) was associated with better in-frontier performance. BCG co-designed a study finding BCG consultants perform better with AI; pre-registration and multi-institutional authorship partially mitigate this — this field experiment compared consultant performance on tasks inside and outside the model's tested range.
  6. Groot, T., and Valdenegro-Toro, M. (2024). Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models. Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), ACL, pp. 145-171. Tested GPT-4, GPT-3.5, LLaMA 2 and PaLM 2 (and two vision-language models); finds high calibration error and models "overconfident most of the time." Models of 2023 — this workshop paper tested four 2023 language models and found them overconfident most of the time.
  7. Khanna, M., Wang, Z., Wei, L., and Xue, L. (2025). When Medical AI Explanations Help and When They Harm. arXiv:2512.08424 (v1, unreviewed preprint). n=257 medical students; 3,855 diagnostic decisions. Explanations raise accuracy when the AI is right (about 73 percent of cases) and lower it when wrong (about 27 percent); high-competence participants with appropriate uncertainty lose the most when the AI errs (-12.4 percentage points, p < 0.01), overconfident novices gain (+9.9 net) — this preprint of 257 medical students found that AI explanations helped when the advice was right and hurt when it was wrong; it is not a study of expert work.
  8. Lee, H-P., et al. (2025). The impact of generative AI on critical thinking. CHI '25, Yokohama. Microsoft Research. "Task stewardship" is the authors' own framing. Limitations: self-report; cross-sectional; critical thinking not objectively measured; causality not established — this CHI study links confidence in AI with lower self-reported critical-thinking effort; it does not establish causation.
  9. Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), 6. n=666. Correlational; recommends balanced AI integration, which the study did not test. No objective performance measures — this correlational study links frequent AI use, cognitive offloading, and critical-thinking scores; it did not test a better way to use it.
  10. Autor, D., and Thompson, N. (2025). Expertise. NBER Working Paper 33941. The inference that AI is specifically automating the non-expert tasks of knowledge work is the author's application of their analysis; Autor and Thompson do not make this specific claim about generative AI — this labor-economics paper explains how automation of expert and less-skilled tasks can affect employment and wages; the AI application is the author's inference.
  11. Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19. Tested in: Veinott, E. S., Klein, G., and Wiggins, S. (2010). Evaluating the effectiveness of the PreMortem technique on plan confidence. ISCRAM, Seattle (lab study, college students, hypothetical outbreak plan; read through secondary descriptions, not the original proceedings). The adaptation here — individual cognitive priming before AI engagement — extends Klein's team-based technique and has not been independently validated in that specific form — Klein's team pre-mortem and a lab study with college students inform the paper's untested individual adaptation.
  12. Ma, S., et al. (2024). "Are you really sure?" CHI '24, Honolulu. Three studies; "Think the Opposite" was tested in Study 2 without AI assistance, and Study 3 found that confidence calibration reduced under-reliance on AI but not over-reliance. The intervention is similar to the pre-mortem prompt protocol and was developed independently. Limitation: income-prediction task in a lab setting — these studies tested confidence calibration and a think-the-opposite prompt, with mixed effects on how far people followed the AI.