Several months ago, we published an article about choosing software testing methodologies. Which approach yields the best results (i.e. the most critical defects, the highest quality reports, the most critical fixes that require the greatest attention, a faster cycle, etc.)?
The article received quite a lot of attention. And one reader followed up with a very interesting question. “After selecting the ‘best’ methodology, are you better off doing internal software testing or opening up the process for public beta testing?”
It’s a terrific question because there are compelling reasons for using both of these approaches.
Making the Case for In-House Software Testing
As professional software testers, we at Testuff are naturally biased. And it’s probably safe to assume that most of you reading this are as well.
Here’s what we like about in-house software testing:
- You can leverage the power of expert testers who know exactly what to look for.
- You benefit from a dedicated team that works exclusively on debugging (instead of ad hoc testing by scattered users, many of whom don’t have any type of professional training).
- This dedicated team knows the code, the functionality, and what “should” be. They understand how the product is supposed to perform.
- In-house software testers have direct access to the rest of the development team (including marketing, accounting, management, and all the other departments required for successful product launches). As discussed in a previous article, this communication is essential for quality assurance and delivery.
- Internal testing produces fewer redundancies. Once a defect is found and corrected, development can continue. With public testing, however, you often get bogged down in repetitive comments – too many cooks all pointing out something that only needs to be discovered once.
Honestly, it’s hard to beat these benefits. In-house software testing is an incredibly valuable asset, which explains why testing represents 80% of most software development budgets.
Let’s take a look at some of the benefits of public beta testing.
Making the Case for Public Beta Software Testing
One of the reasons why companies use beta releases is to create buzz. You can generate excitement with a “soft” launch before officially releasing your product.
From a marketing standpoint, this is fantastic. But does public beta testing produce better results?
Let’s take a look:
- Depending on how large the “public” is, the law of numbers suggests “they” will discover more bugs than any in-house testing team ever could. Simply put, crowdsourcing works, which is why Wikipedia is far more comprehensive than the Encyclopaedia Britannica.
- Unofficial software testers will kick the tires from every angle. They’ll explore functions and find bugs that you never thought imaginable.
- Public testers use different machines, platforms, screen resolutions, and operating systems to discover these bugs. This is a pretty powerful advantage – especially if your in-house team is all working within the same general “environment.”
- Public users rely on one of the best software testing methodologies ever created. They’re actually using the product in practice – not in theory. Isn’t this the ultimate software test?
- Equally important, they’re using the software across a broad range of applications – often beyond the scope of anything your team thought was possible.
- Although public testers don’t necessarily have official training, they come in flexible team sizes and offer 24/7 global testing versus localized 8-hour shifts.
- Outsiders don’t suffer from groupthink. They can offer reality checks and let you know whether or not the product is worth developing (before you sink more resources into the project).
- Last but not least, public beta testing is free. You can amass an entire army of testers without spending a dime.
These are all pretty powerful advantages. But are they enough to make public beta testing the clear winner?
Which Testing Approach Should You Choose? In-House or Public Beta?
Public beta testing is free. It has tremendous marketing value. And it also has the potential to find more critical fixes than even the best in-house software testing team ever could.
As a result, it seems to be the clear winner. Nothing beats free.
But we strongly recommend using both testing groups. And here’s why.
Realistically, you’ve got to do internal testing no matter what. If you release buggy software, you’ll alienate even the most understanding and patient beta testers.
Beta testing is only truly free if the product is ready for the public. Yes, you can dramatically reduce your total software development costs if you leverage the power of volunteers. But there’s a huge social cost when you bring in these outsiders prematurely. The potential damage to your company’s reputation far exceeds any financial savings you might enjoy.
So the best approach is to have the internal team find everything it can. Exhaust all possibilities, explore every angle, and don’t stop until you think the product is ready. Then (and only then) release your software in beta form, and let the public take a stab.
Agree? Disagree? Please share your comments down below.