Sunday, September 22, 2019

We find that oversight increases delays by 6.1%–13.8% and overruns by 1.4%–1.6%

Hmm. From Oversight and Efficiency in Public Projects: A Regression Discontinuity Analysis by Eduard Calvo, Ruomeng Cui, and Juan Camilo Serpa. From the Abstract:
In the United States, 42% of public infrastructure projects report delays or cost overruns. To mitigate this problem, regulators scrutinize project operations. We study the effect of oversight on delays and overruns with 262,857 projects spanning 71 federal agencies and 54,739 contractors. We identify our results using a federal bylaw: if the project’s budget is above a cutoff, procurement officers actively oversee the contractor’s operations; otherwise, most operational checks are waived. We find that oversight increases delays by 6.1%–13.8% and overruns by 1.4%–1.6%. We also show that oversight is most obstructive when the contractor has no experience in public projects, is paid with a fixed-fee contract with performance-based incentives, or performs a labor-intensive task. Oversight is least obstructive—or even beneficial—when the contractor is experienced, paid with a time-and-materials contract, or conducts a machine-intensive task.
This is valuable research but there is an apples-and-oranges flaw at the core. Take that claim:
We find that oversight increases delays by 6.1%–13.8% and overruns by 1.4%–1.6%.
You could leap to the conclusion that government bureaucrats supervising a project are ineffective, costing more in delays and money. And that possibly is true, but cannot be concluded from the evidence in the abstract.

As a management consulting running multiple business units with dozens of projects, for clients with dozens of their own strategic projects, I have some experience in the filed.

If a project goes south, you cancel, you muscle through with the existing team (taking it on the chin in terms of schedule and cost), or you bring in a new team. But even if you bring in a new team of top performers, they are still likely to end up over-schedule and over-budget, not because they did a bad job but either because the project was badly structured in the first place or it was so far in the hole by the time the leadership is changed that you cannot recover. All you are doing is backstopping the losses on a bad project.

What I found interesting in the conclusion was that this seems clearly much more an issue of procurement and project structuring than it is of project management per se.

Why would you let a contract to a contractor with no experience, on a fixed-fee basis, with performance-based incentives? Fixed fee contracts are the most difficult to deliver. The contractor has every incentive to cut corners and take risks, while the purchaser has much less skin in the game to ensure that is delivered on time and on budget.

If those are the projects that go south, that is a procurement issue, not a delivery issue.

The other interesting conclusion that emergency oversight is most effective when "the contractor is experienced, paid with a time-and-materials contract, or conducts a machine-intensive task."

The standard procurement wisdom is all about avoiding time-and-material contracts.

I don't know that there are any universal conclusions to be drawn from this research. There are too many variables unaccounted for. You really want to know what to do in order to get the best outcome for the lowest cost. All this is doing is looking at what to do when projects are already failing.

Even that time-and-materials conclusion may be misleading. When another contractor has failed on a project, particularly one that is incentives-based, it is a signal to the market that the agency owning the project is unable to deliver with contractors. Perhaps it is different in government markets where I have much less experience, but in private markets, an incentives-based project that is failing and the contractor has been displaced, is rarely replaced with another incentives-based contract. The new contractor knows that there is likely a client issue putting the project at risk and will insist on time-and-materials. Of course oversight is better with a new and better contractor, the time-and-materials may be a small data effect and/or a spurious correlation.



No comments:

Post a Comment