Yes, it has been two weeks since my last post. During this time we worked very hard on fixing and releasing Testuff 1.1 for the general QA public.

Unfortunately, it took us longer to release Testuff 1.1 than what we had anticipated. So, after it was finally out, we gathered round the round table and did the aftermath. This is what we found out:

  • We didn’t take holidays into account. The month of September was plagued with Jewish holidays. Even though we do work quite often on non-working hours and on weekends, the holidays still slowed us down, making us go to family dinners and sleep more with lot’s of food in our stomachs.
    – Take holidays into account for the next milestones.
  • Incomplete feature design. We wanted to implement an improvement in the infrastructure. Unfortunately we found out way too late into the coding that the design had failed to take into account certain scenarios and thus the feature could not be implemented properly. The code was then reverted, coding time flushed more or less down the drain.
    – instead of trusting the design with one man only, discuss it with more people as early as possible and include QA in the process.
  • Last minute major changes. Due to some performance problems in the My Tasks feature, we made last minute major changes to fix them. This caused an array of bugs and extended the testing and fixing cycles beyond what we thought.
    – better to wait for the next version or patch with such major changes. This way the release candidate can be stabilized faster for release.
  • Lack of communication. Even though I sit in the same room with the devs, we didn’t communicate about the new version enough. Turns out there were some infrastructural changes that caused bugs, and we are going to have to release another patch to fix them.
    Conclusion – hold testing meetings and discuss changes in the new version and what should be tested in particular.
  • Too many failed validations. They might not be a major time killer, but they sure are annoying and most of them can be eliminated. We discovered that some bugs were closed as fixed even though they weren’t fixed, they just didn’t reproduce for the dev. Another thing we found out is that sometimes the dev didn’t understand the bug and just closed it. The bugs had to be reopened, refixed, and retested.
    Conclusion – if a bug fails to reproduce or is incoherent, talk to QA before closing it.
  • Side effect bugs. Certain bugs, even though that they themselves were fixed, caused side effect bugs that could’ve easily been fixed had they been noticed following the initial fix. These kind of bugs increased the testing cycles we needed to stabilize the version.
    Conclusion – after fixing a bug, take a minute to poke around and see if anything related went wrong.

We have learned some lessons because of this version and I hope they may be of use for you as well. Did you also run into similar situations? What other lessons have you learned from delayed releases? Share it with us in the comments or write a post about it and link back to this one.