I f*#$ed up. I wrote a simple script to send post sale emails to our customers, letting them know we are here to help in case they have any questions or suggestions. The script was supposed to pull out registrants who registered over 2 weeks ago and to whom we have not sent such an email yet. The relevant database field was “zeroed” of course so that we wouldn’t send hundreds of emails to users who are long gone.

Everything seemed OK. I even tested a modified version of the script to check the email comes out correctly by sending it to myself, and did find some bugs. But lacking a more “real world” test, say an example DB with the names and emails of everyone in Testuff, turned out I had a rather silly and annoying bug.

After I activated the script for real, I got back an email from a semi-angry customer who said that good support starts with getting the customer name right. I wasn’t sure what he was talking about until I saw the email that he forwarded me. It started with Hi Chris which wasn’t his name. In the words of the great Homer Simpson: D’oh!

Turns out the following line of Python code was wrong:

postsaleText = postsaleText.replace(textToReplace, firstName)

As you may understand, in the post sale email template I had some sort of fixed text, in this case Hi FIRST_NAME, that was to be replaced with the first name of each customer in a loop. However, with the above buggy line after the first time it was replaced FIRST_NAME was gone and instead Chris, the first name of the first customer that came up in the loop, was pasted in. After that the replace function is useless since it will never find FIRST_NAME again. Thus everyone received a mail that starts with Hi Chris, which wasn’t their name, except for Chris of course.

Here’s the correct line, with the additional fix of replacing postsaleText later on with newPostsaleText in the relevant lines:

newPostsaleText = postsaleText.replace(textToReplace, firstName)

This goes to show that developers just shouldn’t test their own code. They assume things work and fail to see their own blind spots. It takes another pair of eyes connected to some other brain to find these faults. Even a simple piece of code (the script just has 171 lines) has bugs, and despite my own QA experience I wasn’t able to find all of them, blind to my own blind spots.

Another conclusion is that “real world” testing is really important. If you want to find actual bugs that your actual customers may encounter, you need to use actual data for testing instead of a sterile environment. The headaches of setting up a real world environment will save much bigger headaches later, the kind you get when nasty bugs are discovered by your users and cost your company lot’s of dineros.

I’ve seen people asking lately if software companies actually need QA, if it’s not all a big waste of money. Your developers, as good as they may be, write bugs in their software. They can’t find their own bugs. Even if they go testing each other’s code, they don’t have the kind of software breaking thinking & skills that QA do and just won’t find as many bugs. Not to mention developers are usually further removed from the user experience while QA are much closer to it, thus finding “trivial” bugs that developers may ignore altogether, or worse, bugs in the software design.

So, if you’re OK with buggy software that customers will simply abandon in favor of the competition, if you wanna save a few salaries and you don’t mind losing market revenue, go ahead, make my day.