Street under water, flood

Overflow...

Why do Development Projects Fail so often on Deadlines and Budgets?

Time to Read 9 min (Summary 3 min)

When we look around our industry and our clients' industries, we see many projects that fail due to schedule and budget overruns. Or where factors more time and money are grudgingly approved so that the other goals can be met.

And understand me correctly, we are not talking about overruns of 10 or 20%, we are talking about factors of two to ten, which can also call into question the profitability considerations for a product.

What are the Symptoms of Such Projects?

In retrospect, there we see exactly two symptoms: the start of production is delayed and/or the budget is massively exceeded. Are there any indicators beforehand by which these projects can be recognized at an early stage?

  • Watermelon reporting: The traffic lights in project reporting are all green, but unfortunately red on the inside. This means that every project participant knows better about the problems of the project than the management, than the boss.
  • "It always takes three times as long": overruns by factors are so normal that planning is already including them.
  • Chaotic agility: "We develop agile" misunderstood as "everybody does what he wants". It is not clear to the project team which concrete goals they should achieve for which milestones. Only the goals for the next sprint, for the next two weeks may be clear. Roles and responsibilities are blurred, important product decisions are not made and so are replaced by the developers by mostly wrong assumptions.
  • "Draft": this is the state of all documents and artifacts, including basic requirements well beyond the project start. Much is still "nice-to-have" or optional at the end of the project.
  • Estimates and risk lists are still those from the project start, if they exist at all. And most importantly, there are no actions based on changes to estimates and risks, only short-term responses to the most pressing issues at the moment.

And if all projects in an organization show these symptoms? Then we brace oursevles for the

Excuses for Time and Budget Overruns

  • Changes: We think that everything should be said about this in the age of agile development. "Embrace change" means that the project is set up to handle a certain amount of change.
  • Technology issues: These happen all the time, and they can really bring down a project. We believe that most of them can be dealt with through good risk management. And our experience shows: if you have technical problems, solve the human problems...

What are the Causes? And the Corresponding Solutions?

As an engineering firm, we also have fallen into these traps, and we too sometimes exceed the effort by a few percent. As a service provider who often offers projects at a fixed price, we had to adapt our way of working to never overshoot by more than a few percent, because a larger overshoot would break our back financially. We have compiled here the most important solutions we have found.

Disclaimer: these solutions are often very painful and therefore not suitable for everyone! This is also the reason why they are not implemented everywhere.

Bicycle overloaded with firewood

Reason Nr. 1: Unclear Intent and Lack of Decisions

i.e. unclear goals on each level.

  • The overall goal, the sub-goals of the sub-projects and the actual goals of the milestones are not clear to the whole team.
  • The goals of the project, the teams and the individual developers are not aligned.
  • Too many goals are pursued at the same time, each goal still has three options, there are no clear decisions for or against it.
  • Unnecessary goals are achieved because the development project is technology-driven instead of customer-driven.

Solution: Write down Clear Operationalized Goals

  • Operationalized goals means that it is clear what the product will do when it reaches the goal, or at least what the status of which documents and other artifacts in the project is.
    • It also means making clear what the product will not do, what the non-goals are.
    • We find it most efficient to work with intentions:
      • What does success look like?
      • What do we want to achieve here?
  • Writing down means that the goals are formulated in a document accessible to everyone or, even better, on a team Wiki page.
    • Attention, trap: In many teams there is a very digital approach to goals: either nothing is written down, or then a huge "requirement management" formalism is created. Again, the middle way is the best, a few bullet points on a wiki page are better than nothing at the beginning, but also better than formally correct requirements, which are only finished at the end of the project...
  • This does not mean that it is possible without requirements: requirements are indispensable as detailed goals.
    • It is also indispensable to understand that changes of requirements are part of every project.
    • As soon as requirements are written down, they can be discussed. If these discussions with all stakeholders from product management to developers are well lead, then the requirements will "automatically" reach the right level of detail.
  • Decisions: just decide and put the results in writing.
    • We look at decisions as experiments, then they become easier: With the decision, we formulate a hypothesis. Then we test this hypothesis in practice, in an experiment. As soon as we see that we have taken the wrong hypothesis, we adjust it. If we have taken the right one, we don't have to adjust anything.
    • If deadlines are to be met, it is better to decide instead of waiting for additional information ("paralysis by analysis").
  • Change management: What does not correspond to the objectives is a change, clearly identify it in the project and estimate the impact ("impact management"). This serves as a decision support for whether the effort in time and money is worth the change.

By the way, the clear formulation of goals and requirements invalidates the argument "Own employees and freelancers are easier to manage than an external (sub-)project because we don't have to specify anything": Own employees also have to know what they have to do!

Reason Nr. 2: No Realistic Estimation

  • In many organizations the tools to make a meaningful estimate are missing, a number is written down from the gut.
  • Often neither time nor employees are planned for a reliable estimate.
  • Sometimes nobody wants to know the real effort, because the project is already sold at the price, the budget was planned without an estimate...
  • Starting with a realistic estimate, no project will be approved.

Solution: Perform a Realistic Estimation

  •  Realistic estimation means
    • to carry out an expert estimation, preferably with Planning Poker
    • two-point estimation (see e.g. estimation tools)
    • whenever possible attribute-based estimation (e.g. the implementation can be estimated well by most developers, the effort for design, testing, integration... are better multiplied as factors ("attributes"). If an organization has very stable processes, completely attribute-based estimates are also possible, e.g. development time per page of specification.
  • If it is necessary for internal reasons, it makes sense to perform the "tuning" to a target value only after the realistic estimation, one can then always fall back on the first estimation.
  • The effort really used in the project must be documented, so that the estimate can be checked after the project end and the factors for next estimates can be extracted from the project.
  • A simple risk management with damages and probabilities of occurrence is performed.
    • The resulting risk belongs added to the estimate: If 10 risks of 500 hours of effort are estimated with 10% probability of occurrence (i.e., a risk total of 500 hours), then one of the risks can be expected to occur. This means that the project will have to spend 500 hours on it. Where will it get the 500 hours if they are not planned?
    • The risks, damages and probabilities must be tracked through the project, for example, on a monthly basis. And it's a pretty good indicator for an intervention if the overall risk doesn't decrease over the project...

Reason Nr. 3: No Clear Way of Working

  • Each project is managed differently by each project manager, each team member uses the tools differently and understands the processes and terms differently.
    • Officially, everyone has ISO 9001, everyone does Scrum... In reality, everyone does what they think is good. And "agile" is too often the excuse for "chaotic".
    • It follows that no project can learn from another. There are good ways of working (also called processes) in individual projects. From which, because they were never written down, the other projects cannot profit.
    • Processes and their optimization is the task of a quality department. The engineers and especially the management do not feel responsible for it.
  • Lack of responsibility/"ownership": It is not clear who is responsible for the  completion of each product of the project.

Solution: Enforce and Optimize Processes

  • Whatever the basis, agile, V-model, stage-gate or waterfall, project management processes must be enforced.
  • Prepare the project: technically (architecture and design), organizationally and temporally (planning).
    • Otherwise it will be like when all (bricklayers, plumbers, carpenters, electricians ...) start to build the house together (without foundation, without plan, only with a vague idea)...
  • Define in writing the persons responsible for everything that is to be done in the project.
  • Define terms in writing, create a glossary.
    • It can be argued afterwards what a package test and what a module test are. If it is defined, then it is clear to everyone in the project right from the start.

Reason Nr. 4: No Reliable Information on Project Status

  • Often in the management there is no idea about the degree of goal achievement during a project.
  • There is a lot of controlling in the rear view mirror about accumulated efforts and costs in the project management system, however no information about the real project status and the problems (e.g. occurred and identified risks).

Solution: Define Intermediate Goals and Talk to the Team

  •  Iterative delivery, plan and track intermediate goals.
    • The best practice here has been the agile idea of delivering a subset of the features as soon as possible every, say, four weeks.
  • Update estimations and especially the risk list regularly, e.g. every four weeks.
  • Try asking a team member about the status.
    • If the culture allows for an unfiltered answer....

Reason Nr. 5: No Effective Management

  • Acceptance of the "fact" that development cannot be planned, ostrich behavior in case of overruns.
  • "Salami slicing" is implicitly accepted, every project is budgeted too low and then re-financed.
  • Belief in Parkinson's Law: we must limit time and budget as much as possible, otherwise the team has too little pressure.

Solution: Demand Predictability

  • Require estimates and plans from the development team, demand improvement in meeting the planned goals.
    • If salami slicing is needed, use it just right to define intermediate goals: Each slice properly estimated and with clear targets.
  • Do not accept overruns by factors.
    • But that often means accepting higher initial estimates. Which in turn has the advantage that financial decisions about product developments are made on a realistic basis.

 

And the effort for all this? In our experience, it's pretty much 10% of the project effort, much less than the 200 to 1000% without these measures....

And: the above also applies to many other technical projects (digitalization, IoT, IT...).

Andreas Stucki

Do you have additional questions? Do you have a different opinion? If so, email me  or comment your thoughts below!

Author

Questions & Comments

Great insights! Wonderful blog you’ve provided us. I’ve also explored related themes in an ultimate guide on IT staff augmentation on my site. It might add depth to this discussion on budget overruns in software development and key factors to watch out for. Appreciate your thoughts!

Thank you very much!

Yes, this is a recurring theme, in IT and embedded...

Do you have Additional Questions?

* These fields are required

Projects? Ideas? Questions? Let's do a free initial workshop!