TL;DR
A program can reach go-live with every delivery control green and still never be checked for the value it was funded to produce. The controls that run a program can end at go-live, and the value arrives afterward. A missed benefit can then go unseen. The fix is to settle at approval who answers for the value, how it will be measured, and when it will be reviewed.
What the paper develops
A program can go live with every delivery control green and still never be checked for the value it was funded to produce. Delivery governance, meaning the schedule, budget, and scope controls that run a program, can end at go-live, and the value arrives afterward. The argument is narrow. Delivery also fails, and no source reviewed here shows which stage most transformations fail at. The point is that after go-live nothing may compare the results with the business case, so a missed benefit can go unseen, and the next program is approved on the same unchecked kind of business case. Benefits realization, the work of confirming that a program produced its expected result, is not a new idea. The Project Management Institute (PMI) tracks it as a maturity measure, and U.S. federal technology guidance describes how to review it. PMI's 2018 survey found one organization in three reporting high maturity at it, which is a self-report.
The fix is to settle five things at approval, before anyone is reassigned. A benefit owner, an operating leader outside the delivery team, answers for the value after go-live. A measure tracks the result the program was funded to change, such as fewer repeat contacts and not logins. A baseline, the starting reading of that measure, is taken before the work starts. A benefit review is booked for a date after launch with a decision attached: accept the benefit, invest more, change the approach, or stop. Each result then goes into a record the next business case can be set beside. The paper adds ways to separate the program's effect from other change, to time the review, and to handle a program that has no baseline, and it says what owning the benefit costs.
The paper also says where it is weaker. Many changes affect the same result, so what a program can credit itself with is a contribution, not a precise share. Measuring costs money. The PMI figure is self-reported, and none of the sources tested whether booking a review at approval raises the value a program realizes. The five-thing design, and the requirement that the owner sit outside the delivery team, are the paper's own.
What to do next
At the next approval, add one page. It names the benefit owner, the measure and its baseline, the review date, and the decisions the review can take. Then check the largest program that went live in the past year for the same four things.
Inside the white paper
- Why delivery governance can end before the benefit is due, and what that leaves unchecked
- Five things to settle at approval: owner, measure, baseline, review date, and record
- How to separate the program's effect from other change, when to review, and what it costs
Sources and notes
- Project Management Institute, "Pulse of the Profession 2018: Success in Disruptive Times," 2018 — PMI's 2018 survey of project practitioners, executives, and directors of project management offices reports that one organization in three has high benefits realization maturity, which is a self-report. It names managing projects on time, scope, and budget without tracking strategic goals as a challenge many companies face.
- U.S. General Accounting Office (now Government Accountability Office), Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G, March 2004 — GAO's 2004 guide to managing federal technology investments describes a review after launch that compares actual results with estimates, with a general guideline of 6 to 18 months adjusted for when the benefits are expected. It is federal guidance, not an outcome study.
- Michael Bloch, Sven Blumberg, and Jürgen Laartz, "Delivering large-scale IT projects on time, on budget, and on value," McKinsey & Company (with the University of Oxford), October 1, 2012 — McKinsey and the University of Oxford compared predicted and actual results for more than 5,400 IT projects. In a 2012 article, large ones averaged 45 percent over budget, 7 percent over time, and 56 percent less value than predicted. The study covers large IT projects, not transformations in general.
- Patrick Forth, Tom Reichert, Romain de Laubier, and Saibal Chakraborty, "Flipping the Odds of Digital Transformation Success," Boston Consulting Group, October 29, 2020 — BCG combined 825 executives' ratings of their own digital transformations with 70 client programs. About 30 percent met target value with lasting change. The score is self-rated and includes timing, and the report links outcome tracking to winning programs without testing cause.
- U.S. Government Accountability Office, Designing Evaluations: 2012 Revision, GAO-12-208G, January 2012 — GAO's guide to designing evaluations explains that an observed outcome mixes a program's effect with outside factors, and that comparison groups help separate them. It was written for public programs.
- Bent Flyvbjerg, "Quality Control and Due Diligence in Project Management: Getting Decisions Right by Taking the Outside View," International Journal of Project Management, 2013 — Flyvbjerg reports that front-end estimates of costs and benefits often differ significantly from actual results, drawing mostly on large transport projects, and proposes checking a forecast against the track record of similar ones.