Are bugs more expensive to fix in production?

Almost everyone who works in the field of software engineering will likely have heard that the cost of fixing a bug grows exponentially as it progresses through development phases: A bug in a requirement specification is easily fixed. Bugs that are only discovered during implementation are much harder to resolve, but still easier than bugs that have made it all the way to production. The authors of this paper from 2016 refer to this as the delayed issue effect (DIE).

Many engineers and academics to this day believe strongly in the delayed issue effect (side note: This claim was confirmed by a survey conducted as part of the study. I left it out of my summary, but you can read more about it yourself in TFA.). And why wouldn’t they? Not only does it make sense and is it commonly observed in practice, but the phenomenon is also frequently referenced in software engineering literature.

However, the evidence for this effect is both sparse and old. Nearly every citation about DIE can be traced back to a report from 1976. The researchers managed to find only nine sources that provide actual evidence for the claim that the cost of fixing bugs grows exponentially. Four of those sources are related to the original 1976 report, while the other five were published between 1995 and 2007. At the same time, they also found a few studies that report smaller linear increases in the effort required to fix delayed issues.

To put that claim to the test, the researchers examined data from 171 agile software projects conducted across the globe between 2006 and 2014. All projects made use of the Team Software Process (TSP), a project management approach developed at the Software Engineering Institute (SEI) at Carnegie Mellon University. As part of TSP, developers keep detailed time-tracking logs, which they use to improve the quality of task estimates. Organisations that use TSP agree to provide this data to the SEI for use in research, on the condition that it is not traceable to their source.

The TSP data shows that the times to resolve issues usually do not differ significantly between development phases. A high percentage of defects (44%) are found and fixed in early phases (side note: This is to be expected for projects that have invested a lot into software engineering quality assurance practices. For NASA robotics projects, the percentage even is as high as 58%!), even before a single line of code is written. Nowhere in the results do the researchers see the very large increases that are widely reported in papers documenting DIE, and nearly all the increases that can be seen are insignificantly different. Even when the dataset was stratified into subsets, no exponential increases could be found.

Why do these findings differ so much from what we previously thought to be true? There are a couple of plausible explanations, many of which have to do with when the phenomenon was first reported. In 1976, programs were still written using punch cards and only ran in non-interactive environments. A lot has changed since then: nowadays we have more powerful hardware, deploy changes continuously rather than in huge chunks, and most importantly, use agile development processes that are designed to mitigate the delayed issue effect.

It is therefore not surprising that the researchers did not observe the delayed issue effect. At the same time, it is important to consider that this effect may continue to be a problem in some very specific cases, such as high-assurance software, architecturally complex systems (side note: Bad architecture decisions made early in the life cycle may be too expensive to change!), or in projects with poor engineering discipline.