For a very long time, many years now, we were all told that the later in the process a bug is found and fixed – the more it costs. This was written in hundreds of articles, shown in charts, and lectured in seminars. It became the obvious truth.
It probably was seriously first discussed by Barry Boehm, who researched software development economical aspects. We’re talking almost 40 years ago.
But is it really still the case in 2019? Is it a simple truth that the price of fixing a bug in later stages is higher? Aren’t there changes in 40 years that changed the economics of software testing projects?
Software Development Done in a Previous Era
It should be acknowledged that when this claim was first made, it was truth. In a software development world where the different project phases were totally separated, testing being last in line, it made sense.
It also made sense in light of what it meant to fix code, how it was once done, and perhaps more – the time it took to implement it and release a new version of the fixed software. Just imagine sending a technician with a CD to your customers, to update the installed version with the new one…
So the limitations of software development languages, the development cycle and the means/tools for testing and deployment that were available in that time were the cause of this.
And today it isn’t the same any longer.
Challenging Claims From Past Decades
So much has changed, and still is – just faster – since that time. We no longer develop with the tools and languages used back then. We no longer separate the different groups and phases of a software development project: design, requirements, development, testing, etc. These are usually all mixed up, team wise and time wise.
- Agile methodologies for software development breaks the so called ‘agreement’ of the time-lined stages Design -> Development -> Testing. With Agile, it is all done in short cycles, development and testing combined and immediate deployment.
- The automation of the CICD processes enables rapid correction of bugs, so there’s no ‘big price’ for a bug found and fixed.
- Automated testing enables quick detection of any malfunction, any bugs, creating an environment of fast bugs discovery, and thus lower price for the bugs process (finding, fixing and releasing).
There are companies that took this approach even further. They are releasing versions, that have known bugs, just to be ‘live’ and ahead of the competition. Then, taking advantage of the Agile process, they fix them quickly and release the fixes immediately. They don’t see higher costs in that, but rather an opportunity to lead and improve while in production.
True Bug Fixing Costs
There’s no one truth. It all depends.
Barry Boehm himself indicated, in 2001, that at least for small non-critical systems, the ratio (between finding a bug in the development stage vs. finding it while already in production) is more like 5:1 [Boehm, Barry and Victor R. Basili. ‘Software Defect Reduction Top 10 List,’ Computer, v. 34, no. 1, January 2001, pp 135-137] than 100:1 (which Boehm mentioned in his first publication as the ratio).
So the true costs of your project’s bugs, are dependent on a many parameters.
In between them the scale of project/application/software, what version is it (1.0? The first release to production usually means higher costs if bugs found later), your development and testing methodologies (Agile anyone…), the significance of application/software to your company, the type and number of users (how will they react to bugs, which related to how fast you fix these bugs and how do you interact with your users).
What Can You Do
We believe that instead of going back to the old approach of a bug costs that are dependent on when it is found, groups should consider a whole different approach. That is to try to minimize the cost of the bug fixing, regardless of the time it was found, a parameter that becomes less and less relevant.
Possible action items to take in order to reduce the cost of bug fixes are:
- A faster CICD process
- An automated CICD process
- Integrate testing as early as possible in the life cycle of the product development process (design, development, distribution)
- More coverage for automated testing
- A more TDD-driven development process
- More Unit testing, and of more types
What’s your experience? Do you see costs also as reputation costs, not only actual funds spent? Share your stories with us below.