Software Test Management | Testuff

Why so many people don’t like testing?

admin — Mon, 13 Feb 2012 14:22:33 +0000

Last week, I’ve attended a good, interesting conference – Agile Practitioners 2012. One of the sessions was held by Orit Hazzan (An associate professor and head of the Technion’s department of education in technology and science).
I thought to summarize her lecture here for your benefit, adding some notes of myself, as its probably relevant for you, your team and touches an important aspect of software testing.

So, why is it that so many don’t like testing?

Cognitive, Practical

There are several reasons for the resentment testing might cause. These reasons can be explained in a few ways, some work-related, some human-behaviour-related.

Time pressure – Work in general, and projects specifically, usually run under strict time constraints and time pressure. Perhaps they shouldn’t but facts are that they do. Testing, as a seen by many in the organization, slows down the project, holding it from moving forward. Not only that, even those who test, the testers, don’t like being those who prevent us from reaching our milestones/schedules.
Negative feedback – when working the traditional way, first developing then testing, the process starts with something you build then ends with something negative, need to break the code and re-write it. Testers are bound to give negative feedback to the written code, to developers. Again, no one like to hear and get such bad feedback, and most don’t like being those who deliver it.
Responsibility – with the develop-then-test methodology, still widely used, those who create the application (developers) are not responsible for checking it works… it is up to others (testers) to do it. This responsibility transfer is unlikely in most other jobs, or even at home :-)
You do it, make sure you do it right.
Classes at work – developers are a higher class, testers the lower. This simple – sad – truth, should be said loud and clear. Unfortunately, both groups still share this belief in many groups. And who wants to belong to the lower classes? exactly.
Conflicts – the individual tester is experiencing a conflict doing their work. This is a conflict between the group’s goals and his goals. The group has a project to complete, schedule to meet, budget to stay in. The tester is all about finding problems, issues, where does the project fail and should be stopped until fixed.
Managing testing is difficult – working the traditional way, with testing located at the end of the line, raises challenges for the QA manager. What to test, when to test, how much time is required and how much time to allocate for each test or group of tests, understanding the requirements and the new developed application/features/code.

Breaking the circle

What can we do to improve the process and find solutions for all these problems? How can we make software testing be more attractive, more accepted?
The answer is found in the description of most of the issues. In most you can see that the separation between the groups of the project – developers, testers (even business people can be counted here) – is a main source for them.

So, let’s get everyone on the same team.

The business people, the analysts, developers and testers. All of them should be in a team, working on a project, or part of it. They should be working together from the beginning, when requirements are discussed, explained and written, through the design, then development and testing.
A collaborative relationship rather than a competitive relationship with a software team will drive better results for all, for the project first.
Call it development driven testing, or any other way you wish, as long as you can see the benefits of such practice and how it can solve the problems in discussion. let’s go back to them and review the new process and how it helps:

No time pressure – Testing is done continuously, from the first phase, never postponed to the end of a development stretch. It’s done in smaller units, with visible immediate results. Development and testing move forward simultaneously.
Positive feedback – This is a nice one. In the first case the process always ends with a negative feedback, but with testing done while developing (and fixing) the end result is always success.
Think of code that is written according to an already written test – it is bound to work, and the test will pass. It just can’t otherwise.
Responsibility – all of the team is now involved in the testing process, writing tests, understanding the tests, and developers are practically executing tests as part of what they do.
No classes – since not only the testers are testing, the importance of the testing is now higher, developers are part of it, accepting it completely different than before. Working all on the same team, brings more equality, better communication, better understanding of the qualities and challenges of each team member.
Less conflicts – the individual tester is now part of a group, which is not a testers group only, but a group/team which works to complete a goal of creating a new working code.
Easier testing management – the QA manager has now more information, from day 1, can better plan the resources, the tests, what’s need to be tested and how.

No Miracles

Miracles happen only in fairy tales, and development projects are defiantly aren’t such…. to accomplish a better project, and with in it, better software testing, managers need to work, and find the best solution which will fit their vision, their organization and their specific project.
Changing the way of work, to the described suggested way above, cannot be lightly done. It takes time, people need to understand it and how it works, and how it will help them.
This involves a new culture in an organization. New procedures and work flows. Some will find it hard to accept maybe, as is with new things brought to an existing environment. Since a basic part of this methodology is based on cooperation managers must find the right way to convince people it’s a good thing. Forcing it will fail the idea itself.

We believe it’s worth the effort.

Test Editor, Labs and API improvements

admin — Mon, 30 Jan 2012 14:19:54 +0000

2012 is here, and our first version for the year (Testuff 1.38) is out.

We’ve started 2012 in full force, and plan to cover a few areas in Testuff for which we have some great enhancement ideas (Requirements is one to mention). This year will be one were we will launch new services, adding to your complete testing experience.

Here’s the full list of enhancements for Testuff 1.38:

Test Editor Enhancements:

Control the Font Size and Color
Hot-Key Ctrl+F for Search & Replace functionality

API Additional Supported Options:

Report a new defect with an attachment
Update test status and time progress

Better Planning in Labs:

Assign new tests from current runs in a lab plan
Right-click to go to a test from a run

And more:

Reports Share option is now available on all reports
Search option on our website’s Online-Guide

Visit our Website for a full list of all new features in all versions since 2007.
Testuff. There’s always something to wait for.
______________________________________________________________________________________
Tips of the month:

Work on several tests at once. Select more than one test/run and right-click the selection for speed editing.
Exclude tests from requirements. If you don’t need a whole suite to be set as a requirement, you can select tests, in that suite, and exclude them.

* Take a look at the list of tips we gathered here, for your convenience.

Absolute Beginners

admin — Tue, 24 Jan 2012 14:18:32 +0000

When we first look at a new thing, we are bound for that first impression. This first impression is sometimes the barrier, stopping us from starting a conversation with someone, choosing something to eat from the counter and of course buying a product. “No second chance for first impression” is a well known saying. For a SaaS business it might be a major differentiator between success and failure.

Most of our potential customers are anonymous for us, and their purchase decision process is often the same as with all SaaS online sellers:

The customer needs a service/tool.
Searching the web (usually Googling for key words matching the services they need).
Evaluating several of the found alternatives, to see which one best fits their needs.
A purchase decision for one of the options evaluated is then made.

This is of course the short version of the process, which can be a longer, more complicated one, in some organizations (involving people from different departments, internal demos, questions sent to the vendor, quote requests and other).

I will not discuss the process of how to be one of these user-found-me-on-the-web alternatives. The online activities of advertising, the SEO, social media, and website quality are all aimed to bring your product/service/tool to the right place to be included in the evaluation process of potential customers. I personally don’t believe anyone has found the Holy Grail of how to do it right (yet?). I will also not discuss here what makes a good first impression, or what and how to make your tool look better compared to the alternatives.

The purpose of the post is to list several techniques of how to get a second chance for first impression… that’s right. It’s possible.

What we want to do is learn from first time users, how do they see the application on that first impression, what are their thoughts and what can they say after the first few minutes/then hours of using it. Improving this first impression experience will give us a second chance… with the next new potential customers, on their first impression of our service.

“Real” User simulation

Asking a friend to try your product, or paying a company to get you a focus-group, have both the same meaning of bringing new users and learning from their reaction, real-time actions while working with the product for the first time, their questions while doing so and their direct feedback.
You can learn by watching them, asking them questions, and even use professional tools like “eye movement tracking”. If you are under budget restrictions (aren’t we all..), you can do it all online, and send a survey to users, hoping that results will be meaningful. The downside with all of this effort, and this type of users, is that they are not your actual real users, and much of their input might be irrelevant to you or to your real potential user/customer.

Asking for feedback from users

Asking our current users how to improve the process, one they already saw, is another option. These are users who voted already for you, they are usually happy to help and give feedback, and they already know a bit about your app. The problem here is that these users are not exactly relevant to your target audience for that first impression we are looking to improve. You are looking for that fresh new look at your product, that first-time when feelings and impression play the main part, not the logic, expertise or knowledge. Your current users/customers are the experienced users. More than that, they probably have, at this stage, a positive view about your product (after all they became customers). But you are looking for those with either no set mind or even those with a negative position. Another disadvantage of this effort is the survey content, the questions asked. Our experience shows that users will always give you ideas and feedback according to your questions. They will rarely bring up a new approach to the table even if asked for it, and it will be difficult for them to go back to their initial impression and get the data you need from them, about what worked back then, that made them stay and continue the evaluation.

High priority for “how-to” support emails

Another place to get good, solid information is an easy one to implement. Support emails are full with information and knowledge, businesses will usually pay to have. Why not use it if it’s already there?
Whenever a user sends a question or request, it should raise several follow-up questions by you:

Why do they need our assistance, maybe our product is not clear/intuitive enough?
What should we improve? Is there anything missing?
And for me, the most important thing, is asking the user what were they trying to do and why.

In many cases the user might ask a simple “how-to” question. You can answer the question, explaining how to do it and ‘close’ the case. That’s wrong. Consider asking them what were they trying to do, and more important – why. What is their process, work-flow, and how do they use your application for that matter. You will find that you can improve many areas in your solution, change how things work and make it useful for the (real) users.
While this method seems not strictly related to our first impression discussion, it can be so. By continuously improving you product you will have a better product and obviously a more user friendly one, making a better impression even in the first look.

Using freelance testers

A recent practice I learned from using freelance testers (using online resources such as oDesk and others) is that they are a great source for that first-look-impression feedback. You can hire many testers for a relatively low budget, and ask them to use (test) any part of your product. Since these are experienced testers, they will send you great, valuable feedback which will not only help you with the testing but will provide ideas and insights to get better first-time impression. Direct them right, sending the exact expectations, and get that feedback right back.
Although not what we defined earlier as ‘real’ users, they do have the QA skills and will know to tell you about the usability of your product, some possible improvement directions, perhaps a few bugs found on the way :-)

All of the above mentioned options, to get that so-important feedback on how to improve the first time impression are valid, good to use, with their advantages and disadvantages. At Testuff we found that giving high attention to a customer’s email, wisely asking them those “what” and “why” questions, lead us to improvements that most of our users liked. And using freelancers testers gave us that useful fresh look of our product, and led us many to good new enhancements ideas.

Better Test Planning

admin — Mon, 02 Jan 2012 14:22:39 +0000

Last version (Testuff 1.37) for 2011. An exciting great year for Testuff, hopefully for you as well.
This version includes a few new enhancements, which will make it easier for those interested in planning their testing cycles better. Below is the list of what’s included.

2012 is here, and we are already working on the first version for this year…

Here’s the full list of enhancements:

Better Test Planning:

View All Tests assigned to a lab, on one list.
Improved Filtering. Use advanced text-search.
Assign tests to users and labs from the Tests Tab.
Assign tests to users and labs from Search Results.
Get specific Objects IDs (Branch, Suite, Test, Defect).

Share Suites with Non-Testuff users.
API Enhancements:

Get Data from your account using the API.
Report Run Status.

Better, informative error messages.

Share your reports with others. Use the share link on the bottom of every report.
Customize your bug reports. The Defect Template is a great way to get your bugs look exacly in the way you want them.

* Take a look at the list of tips we gathered here, for your convenience.

2012 – SaaS Test Management Solution Year

yoav — Tue, 20 Dec 2011 14:04:36 +0000

As we come to the end of our most successful year, it is summary days. We are going into our 5th full year, and this is already the fourth summary we write. Time flies.

In each of the previous years, we wrote a short summary about how the year was, and what did we do that year. We talked about our plans for the next year. Since this is holidays season, tradition and all in the air, we’ll continue ours (tradition) as well…

As we’ve mentioned last year, December 31st is no different than January 1st for us same as it is, we’re sure, for you. Work continues, old and new apps to test, existing and new features to try, old and new colleagues to work with. However, in the spirit of the time, new year and all, we sit back for a few minutes, stop the daily routine and think about what happened here and what do we wish would happen….

And so, keeping the same format of last years here it is:

What did we do

We have managed, yet again, 12 new versions, as planned. Not obvious, not easy, but thanks to our developers and testers we have made it. These versions were rich with useful features. Just a few as example:

Share reports
Synchronize Defect Status directly from your tracker
Defect Template
Integration with Clarizen, Youtrack, ChiliProject, activeCollab, Basecamp
Improved integration with all bug trackers: custom fields support and more
New API
Audit Trail
New Reports
Test editor Spell Checker
Template Tags
Server framework upgraded from CherryPy+Sql Alchemy to Django
Added internal bug tracking functionality to Tesuff for those who don’t use one
Added Customization for a few objects

And there are so many more. If you are a user, you know them and have witnessed and enjoyed the monthly progress. If you are not a user…. well, time you became one.

What else? It’s a repeat of 2009, 2010:

2011 was another year of growth for us. Many new customers from many countries, have joined our community.
We worked hard to keep on the good service and smooth operations. From the feedback we get it looks we manged not bad.
Our support team had kept their average response time of less than 3 hours to any incoming email.
We’ve continued to improve our back-end infrastructure, added data centers in several countries and prepared ourselves for another year of growth.

What didn’t we do

We also failed on a few things. There are one or two tricky bugs which we are hunting for some time now, so far with no luck. There’s progress, and we know we will catch them, it is only frustrating while it lasts.

What will we do

We can promise that:

We will continue to be #1 SaaS Test Management Solution.
There will be 12 new monthly significant versions.
Customers will still be the #1 source for what should be included on these versions.
We will introduce new services.

What we won’t do

As always, we won’t be annoying any customer or registered user with too many emails.
We won’t stop enjoying what we do :-)

Happy Holiday Season To All !

Making the Difference

admin — Tue, 06 Dec 2011 14:59:34 +0000

“Happy families are all alike; every unhappy family is unhappy in its own way” Leo Tolstoy

“Happy” Testing Teams

What makes one testing team better than the other? Is it their manager? The methodology they use? The quality of the testers – experience and education included? Or perhaps the combination of all and more?
You would usually want to think that a “Yes” is in place here. But is it really? To an extent it surly is, however if we compare a large number of testing teams, we should – statistically – expect to get the same quality of testers (as testers and as people, whatever that is…), same manager, a few same methodologies used by most teams. You get the point.
On average teams, the bulk of them will fall under the same ‘total score’ for quality. Forget about the exceptions at the top and bottom of the list. Odds are you are not in either :-)

The fact you have the best team players, or the biggest budget, or the best support from above and around you doesn’t necessarily say you will do well. Just see what happened to some sports team who bought, for unreasonable amounts of money, the best players they could get, just to find out a few months later, how big the mistake was.

Don’t fool yourself. We hear those QA managers, each with his ‘answer-to-all-problems’: if they just gave me that automation tool and agreed to implement it, if they just gave me more resources, if I just had a few more days/weeks/months, if only the project manager listened to me…. stop and think. What if this happened? would that make the big difference?

Why would one team perform better than the other? Some groups are found to be more productive, get better results of their testing work, and in a shorter time.

“Unhappy” Testing Teams

Testers have a long list of reasons they give for poor delivery:

The schedule was too short
The budget was too small
The team was inexperienced
The developers didn’t corporate, and communication with them was impossible
The tested application was not ready even for testing

and the list goes on…

You know it :-)

For each project, and each cycle one or two can be picked of the list and used to explain. So, those “happy” groups from before, how do they get the job done under the same conditions? We agreed, as the assumption for this discussion, that the average grade (or quality points) of a typical testing group is the same as the other group. So, how come one is happy and the other isn’t?

Drive, Consistency and Work Ethics

Whichever team you have, whatever methodology you decide to work with, tools you use, budget allocated or scheduled defined – if you set correct 3 things, you are bound the get the best results possible for the situation:

Drive

“Those people blessed with the most talent don’t necessarily outperform everyone else. It’s the people with follow-through who excel.” Mary Kay Ash

Make sure the team is working as a group, understanding the end goals, knowing how to get there. Make sure they have the Drive otherwise they will just an ‘OK’ job, not a good or excellent job. If your testers don’t love testing, replace them.
Give them cool tools to use, appreciate what they do, make sure they are part of the project and not just a side-kick of it.

Consistency

“In any team sport, the best teams have consistency and chemistry” Roger Staubach

Choose how you want to work and go with it. Don’t run after every passing new fashion, with lots of buzz-words explaining them. We think that common-sense is your best friend, but any methodology you choose, stick with it. Same goes for much of how you run your projects. This doesn’t say you can’t improve, change, or be creative. There’s no contradiction between the two.

Work Ethics

“Striving for success without hard work is like trying to harvest where you haven’t planted.” David Bly

You can have the best plan, a good team, you chose your route and you know what and how to do to get to the results and goals you set yourself and your team. If you and your team don’t work hard to get there, day in and day out it doesn’t worth anything. Your team must come to work, not just get to the office…

We’ll discuss in one of our future posts, a totaly different angle of happy testing groups, and how to make them happy.

Testuff Continues to Work For You

admin — Mon, 05 Dec 2011 14:24:53 +0000

Testuff new version (Testuff 1.36) follows up on the previous version, and further improves your Defect Management and Reporting.
After allowing editing of defects and file attachment we have made the extra mile and included an option to update the defect status directly from the tracker, with a simple one-click link.
That should make your communications with the developers an easier task :-)

Another great new feature can be found in the Bug Reporter – you are now able to build the report however you want it. Select the fields, add permanent titles and more. We gave you full control.

Checkout also the new integration with Clarizen. This great project management tool has just been added to our (very) long list of supported bug-trackers.
And that’s not all. How about sharing reports with non-Testuff users? and he list goes on….

In the last version of the year, we’ll be racing to complete a few of the feature requests of this year that are not yet implemented.

Here’s the full list of enhancements:

Share reports with colleagues, clients, managers even if they aren’t Testuff users.
Synchronize Defect Status directly from your tracker.
Build your own Defect template and Customize it, while reporting it to your bug tracker.
New tracker on our long list – Integrate with Clarizen.
Easily Export Defects to Excel.
API Improvements:

Query for items:

/suite/ get all, get by branch
/test/ get all, get by suite
/lab/ get all, get by branch
/run/ get all, get by test, lab, user and branch
/defect/ get all, get by user

Report a new defect

Those little things that’s makes the user happy:

ctrl+A is now supported for labels, run configurations (in their setup windows) and in the defects screen.
Easier Run Configuration management and assignment.

Visit our Website for a full list of all new features in all versions since 2007.
Testuff. There’s always something to wait for.
______________________________________________________________________________
Tips of the month:

Click the pie chart in the Requirements screen, to get the list of tests for that requirement.
Use the ‘Teams’ feature (in a project’s settings) to better manage your testers and projects.

* Take a look at the list of tips we gathered here, for your convenience.

Those little things that make our work easier

admin — Thu, 10 Nov 2011 20:49:08 +0000

Every application we use has those little things, that make the user’s work so much easier. It can be a keyboard shortcut, a hidden option, or a quick way to make the most out of a feature.
We once had a post written, by a former team member of our support, explaining how to change the background color on a PowerPoint presentation in a specific scenario where it wasn’t obvious – the feedback was amazing. Emails, comments and loads of website visitors who were looking for this over the net and got to this post…..
Those little things.

For over a year now we have been sending Testuff usage tips on our newsletter. Two of them every month.
We’ve decided to sum them up all here, after getting questions from users asking “where can I get the full tips list?”. Truth is that we don’t (or didn’t) have such a list, as the tips were written each month, following our own ideas or our support team mentioning questions that showed a few less known Testuff options and features.
Usually users easily find the various options and features Testuf offers, as these are all visible on-usage and intuitive in their location in the app. Still, some help is always a good idea :-)

So, here it is. Let us know if you find it helpful or if you have a few tips of your own for us to add. We may add the list to our Online-Guide soon.

Testuff Tips:

Copy & paste does miracles in Testuff. Use it for a test or a suite, or even a group of them (ctrl and shift work). It works between branches and projects as well.
Soft-Link is a great tool for better structuring and organizing of your testing.
Bug tracker setup allows you to connect each Testuff project to a matching tracker project.
A test in the lab will be automatically re-assigned to the tester who runs it, regardless of any other tester’s list of tests. You can use it, for example, when not sure about assignments when creating the lab.
Watch for the icon in Testuff. Click on it to get more information about a test.
When reporting a defect, you can drag & drop files from a folder to the entry fields of the bug reporter file attachment section.
Click on a column’s title to sort the table by this column.
A test can be edited while running it. Use the “edit test” link on top of the runner.
To re-organize your suites, you can Drag & Drop a suite into other suites.
To pass more than one test at the same time, mark a group of tests in the Lab, then right-click them and select “Mark as passed”.
Ask your team member to run a test directly from the Tests view by a simple right-click–>assign option.
Use the “Forgot your password?” link on the login screen, to create a new personal password if yours is lost.
Click the bulb icon on the top left side of the Test Runner to get the history information of the test you are running.
In Testuff home screen, bottom right side, you can see the version you are working with. It is highly recommended to always work with the latest version.
Follow us on Twitter. Sometimes we tweet interesting stuff there…
Twistuff can be great group tool, for messages and mutual follow up. It can be a manager let-me-know system as well.
Testuff is built to be intuitive for you to use. This means you should try to do all those things you got used to, such as highlight multiple items with shift or ctrl, use copy & paste for tests/suites, use the mouse right click to discover the many options on each screen, and on each item.
Subscribe to our data centers status page. it can come up handy one day.
Double-click a search result item to go directly to it.
Select your spellchecker preferred dictionary. Go to Settings–>Languages. Can’t find what you need there? Let us know and we’ll add it.
Test Runner window is “always on top” by default, however you can control this, in the Settings–>Look and Feel tab.
Table columns can be customized – sort the table with a click on a column’s title, adjust the width and use hide/show option to select the columns to display.
Use Export test/s to Excel to quickly edit them, and import back using overwrite import option.
Use Template Tags for defect custom field value reporting.
Keyboard Shortcuts work in the test editor. Use them to speed up your work.
When assigning tests in the lab, select more than one Run-Configuration to create a run for each with one click.
To change your monthly subscriptions (licenses) number go to your Settings window in Testuff and click on the Change link next to the number of licenses.

New Level of Managing your Defects

yoav — Mon, 31 Oct 2011 14:37:18 +0000

Testuff 1.35, your new monthly version, is all about Defect management. We have focused this time in the defects area, making it easier to track defects within Testuff, and at the same time improving some more the integration to the bug-trackers.

You can now edit a defect after reporting it, attach files to it and follow up on status changes for the defect. There’s a new integration also, this time the new member on our long list is Youtrack.

The list of enhancements includes a few more:

You can now do more when reporting defects:

Edit a defect
Add attachments to defects
Report the new Severity field on defects
See Defect statistics in Overview tab
Integrate with your Youtrack bug tracker
Use the updated Rally API (version 1.27)
See the failed steps on report

Audit trail:

Enabled on Defects, simply right-click and select history

Improved test organization:

Supports Cross-project Soft-links
Full suite path is now shown in My Tasks tab
Exclude-from-requirement multiple tests at once

In the coming versions we will continue to enhance the defects tracking in Testuff, and will work to implement the promised feature, for updating a defect status directly from the tracker.

When assigning tests in the lab, select more than one Run-Configuration to create a run for each with one click.
To change your monthly subscriptions (licenses) number go to your Settings window in Testuff and click on the Change link next to the number of licenses.

Is Automation Overrated?

yoav — Tue, 11 Oct 2011 17:06:56 +0000

No doubt that automation testing is sexy and impressive – it seems to be a great cure for the annoying process of more laborious testing. Imagine that you can write the crappiest, laziest piece of software and a magic automation tool painlessly finds any and all defects and helps you fix them.

Sadly, this is not possible. But you probably already knew that. Nothing compares to a good human tester with extensive experience, advanced skills, and basic intuition. However, the human approach is not without its drawbacks either. Human testers are more expensive, and that’s only if you can actually find and train them. Keeping them working at the same company for a long time is even harder still.

But this post is not about the tester – it is about automation testing, when to use it, and when to avoid it. After all, automation testing is not always a bad thing. You don’t have to pay for training, and you never run the risk of sick days or the possibility that your automation tool will look for greener pastures. I would just caution anyone who embraces automation as a silver bullet – a panacea to fix all ills and completely replace actual human testers.

However let’s cover when automation is a true benefit. I can think of two distinct scenarios – “Not GUI” and “Boring Parameters.”

Not GUI
This is the classic area in which automation thrives. Developing server side components, protocols, and devices doesn’t require an actual person to send raw commands to the device and test for functionality. Of course, developers naturally do this during the development stage, but to require a physical tester on-site whose sole purpose is to type “sudo blah blah blah” is a phenomenal waste of time if you have an automation tool that can perform the same duties. Automation testers never get bored, rarely make mistakes, and can share their results instantly.

Boring Parameters
This scenario applies to testing in which algorithms generate different results based on different parameters. A good example is the complex algorithm used to calculate your monthly cellular bill (minus all the sneaky fees that invariably require a “human” element). This type of testing can involve more parameters than NASA uses in spaceship design (270,000 according to Bruce Willis in Armageddon). In these cases, automation tools are better equipped to handle the vast number of boring and monotonous inputs than a typical human tester can comfortably manage.

The approach that many automation tools use is something I call Sanity GUI. Such tools provide you with a great framework for recording your application GUI, allowing you to run it and generate scripts that the tool can replay ad infinitum.
The rationale behind this approach is simple – you can do a “real” run in your application, and the tool will record it and allow you to replay this run at any time in the future to ensure that everything still works properly. Moreover, the automation tool will never tire or complain, managers cannot “fault” you for not doing the tests, and the tool itself is actually quite enjoyable to use.

I have no argument against this general rational. On the surface it makes perfect sense. However, in my own testing experiences across a broad range of applications and environments, I have discovered that “Sanity GUI” brings certain disadvantages. In effect, when using automation tools in the above example, I often end up spending more time and money. In nearly all cases, recording is a breeze, script repeating works like magic, result comparison is a snap, and my managers love the neatly generated reports. And on top of all this, the technology is getting better and better, bringing improvements on nearly all fronts. So what is the problem exactly – what is there not to like?

Simple. The problems typically start on Day 2.

After recording the scripts, it becomes increasingly cumbersome (i.e. expensive and time-consuming) to keep them updated. Each time the GUI is modified, the scripts routinely fail. Moreover, there is usually no direct benefit from running the script through the automation tool since one typically tests the application from the GUI. Very rare is the automation script that will discover something that I hadn’t already discovered on my own through normal application testing.

But for me, the worst thing about automation testing is the de-emphasis it places on living, breathing testers. Not only do I waste time using a tool that finds defects I already know about, but automated tools also find “false” defects due to the constantly updated GUI. Every minute spent sorting through obvious defects or false positives is a minute taken away from real testers who run analyses using the actual application to find “real” defects.

When all is said and done, hours of automated testing may need to be followed up with “hands on” human testing, which begs the question – “why automate in these specific cases at all?” It’s like paying people to clean your yard with the full knowledge that you’ll have to go over their work when they’re done and essentially start from scratch as if they hadn’t done any work at all. This is both a waste of money and time.

To Automate or not to Automate?
As I mentioned before, automation is not a bad thing – in most of the industrialized world, automation is a buzzword that connotes efficiency, reduced costs, and speed. But when deciding whether or not to use automation, we should remember that “automation” itself is not a goal. Rather, it is the efficiency, speed, or reduced cost that automation can potentially bring when designing or producing a superior product. In other words, the product itself is the ultimate aim, and one should decide what role automation can play in moving this aim closer within reach.

In our industry, testing (and not automation) is the ultimate goal. In times when automation can yield superior tests with lower costs, better results, and fewer backside problems, then such tools are clearly beneficial. However, if using automation tools results in more clean up, more human testers (after the fact), and an inferior or costlier product, I caution you to exercise good judgment. Real testers might represent higher upfront costs, but if the end result is a product that makes users happier (and your company more profitable), then clearly automation does not represent the best course of action.