on-demand software test management

Even though I am ultra busy testing Testuff 1.0, that’s coming out very very soon, I decided to take some time out and to conclude this crazy series of posts.

We’ve seen why automated testing rocks. As I’ve stated before, it’s more accurate, work faster, harder, for a longer period of time, for less money, with no complaints and no quitting. We’ve also seen why human manual testing rocks. Humans are self maintaining, adaptive to the environment, creative, recognize objects better than machines, have built-in natural language processing, and also excellent learning abilities.

So what is the summary of it all? Fire all humans and replace them with automated testing machines? Throw all automation out the window and hire some testers for cheap from India? No!

The world doesn’t actually divide into black and white, but we quite often mistakenly think it does (a bug in our brains? The misguiding of Western society? Both? None? who knows). We don’t have to be crazy like that Israeli company who fired all testers in favor of automation, though who knows, maybe for their needs of completely repetitive well-known testing it was the right move. We also don’t have to reject any kind of automated testing at our company and stick to flesh and blood testers only. The answer is that we can enjoy both worlds, the shades of gray in between the black and the white.

Be a cyborg. Part man, part machine, it has all the advantages that humans do, but some off the hook technological enhancements that allow it to have beyond human capabilities. To take the analogy back, this means that we enjoy the best of both worlds - the automated testing world and the manual testing world and use them both to our advantage:

1. Automate boring, repetitive, high accuracy tests. Humans don’t like to do them and don’t do them as well as the automation does. By automating these kind of tests your QA will be happier since you will lift the annoying load off of their shoulders and you will get better testing results via the automation.
2. Give humans creative testing tasks. As a complimentary item to the previous one, challenge the human testers, use their natural learning skills, adaptivity and creativity to find bugs way outside the scripted tests. Don’t let them spend their time only on pre-scripted scenarios. Let them do some Exploratory Testing, new test designs, maybe even learn some programing so that they could write some useful utilities.
3. Work the automation overtime, not the humans. Having people do overtime for too long under pressure increases work stress and basically drains all the motivational juice out of them. This is why high-pressure testing, such as sanity, might be better put on the automation. However, watch out not to pressure the humans who execute the automation!
4. Use low maintenance automation. It’s a real pain in the ass when the automation requires too much maintenance in terms of time and effort. Don’t be afraid to go back to the drawing board and design your automated testing from scratch. Check why it takes such high maintenance and either solve those issues or move tests to manual execution if necessary.
5. Humans test some things better than automation. Given the current technology, computers don’t know how to do many things as well as the humans. In the testing world this is especially true about GUI. Sure, there are solutions for web testing and functional testing for simple GUIs, but if your app is more complicated than the average software Joe you will quickly see that the automation doesn’t do. If these automated tasks suck for your testers, maybe even consider off-shoring them to some Indian testers who wouldn’t mind doing some repetitive tasks for good cash.

If you still don’t believe me that you should become a cyborg, watch the trailer for Robocop and see how cyborgs improve law enforcement in crime filled future Detroit. No man and no machine would do to clean Detriot of its thugs, but only a man machine combination that is Robocop:

  

Here’s a great tip if you work a lot with the Windows Command Prompt aka cmd.exe. Wouldn’t you just love it if you could easily copy-paste stuff in the Command Prompt window? Well, you can!

1. Open the Windows Command Prompt via Start > Run > type cmd > OK.
2. Right click it on the task bar and choose Properties.
3. In the Options tab, make sure Quick Edit Mode and Insert Mode are both checked:
   
4. Press OK.
5. The Apply Properties dialog will appear. Choose the second options, Save properties for future windows with same title, and press OK.

Done! Now you may:

• Copy text in the Windows Command Prompt by holding down the left mouse button, marking the relevant text, and pressing ENTER.
• Paste text by pressing the right mouse button.

BTW, if you like this post or any other post that I wrote, please help me spread the good word by bookmarking the post in digg, delicious, or reddit. Small icons are available at the bottom of each post for easy bookmarking. Thanks!

  

The machines have made their move in the previous episode. To sum it up, they are more accurate, work faster, harder, for a longer period of time, for less money, with no complaints and no quitting. Sounds just about perfect doesn’t it?

But wait! It is time for the humans to fight back, to show why in certain aspects their manual testing is way better than any testing automation can be. Let’s see what they got going for them that the silicon based life forms cannot beat.

Human Testing Pros

1. Self maintenance. Humans are (usually) capable of taking care of themselves. If there is some sort of system malfunction like an illness, hunger, sadness, or whatever, humans can at least attempt to solve it by themselves. The automation cannot do so. It needs someone to maintain it. If some sort of malfunction happens, human intervention must take place. Be it a network cable gone loose, a fallen web service, an air-con malfunction, the machine cannot go about taking care of itself.
2. Adaptivity. Testing automation lacks any kind of intelligence. If, for example, a connection failed to open, it won’t go about investigating what happened. A whole series of tests could fail because a network cable went loose, thus rendering hours of testing automation useless. A human on the other hand can use her wits to investigate what happened, find that loose cable, and go on with the proper testing. Also, if some change in the AUT was made the automation will probably fail. The human on the other hand will easily determine if the change is acceptable and act accordingly.
3. Creativity. Whereas the computer follows the scripted scenario religiously, people look in between the scripted lines and outside of the test altogether. They will find bugs even in things they weren’t supposed to test and be able to come up with new tests for the future. Of course, let’s remember it was the humans who programmed the testing automation to begin with, it isn’t able to design tests on its own.
4. Object recognition. Even a five year old could easily beat any computer running any algorithm when it comes to recognizing objects. Thus, and despite so many efforts made, testing automation has problems testing GUIs. A human on the other hand doesn’t really care how a UI was implemented. She can easily recognize buttons, windows, scrollbars, or anything else and get testing with it. She can see if the colors are OK, if images are displayed properly, videos are streamed in good quality, etc., things that the testing automation has big trouble doing.
5. Natural language skills. Humans may give and receive abstract instructions in natural language and act. A human tester receiving an instruction such as “test the site also works with IE7″ can easily go about doing so. The testing automation must be programmed with exact instructions in a programming language. It takes much more time to give instructions to machines and involves testing and debugging, once again by humans, to make sure the automation “understands” the instructions properly. Not to mention that automation cannot test something like a help file, that it contains good help with correct grammer. It just doesn’t understand the human talk, not out loud, and not even if we hand it in a text file.
6. Learning. Humans can learn new features, new technologies, different ways of doing things, and append new knowledge to their hard disks. Testing automation cannot do so. It needs the programmer to step in and give it exact instructions what to do. Otherwise it will never expand its knowledge base and just keep doing the same things that it knows forever, not doing anything new at all.

Once again, feel free to respond if there is something missing in the human onslaught agains the testing automation. Do join me once again for a summary of the fight between human and automated QA!

  

There seems to be some confusion about our company name.  When I tell people that I work for Testuff, I always have to add in a pre-scripted bit to make it all clear. “It’s ‘Test’ and ‘Stuff’ in one word”, I emphasize and add some hand gestures to show what I mean. When I’m physically there I seal the deal by giving out my business card, thus making it clear how to spell our name and how to reach our site.

However, I’m not always the one to speak of our name. Friends, relatives, and most importantly users may mention Testuff to other people. Unfortunately telling a person the name of Testuff doesn’t conjure that kind of spelling in one’s head. It’s more likely one would presume that we are called Test Stuff, or TestStuff, and then go look it up in a search engine.

Luckily searching for Test Stuff on Google brings us in the first results page, but we are nowhere to be found on a TestStuff search. So our name may be costing us new users, most of which probably won’t be persistent enough to keep looking and change the search keywords to find us.

The moral of the story - if you are about to start your own company, make sure its name is easily spelled by doing some real world testing. Mention it to a few people, ask them to spell it back to you, and check that it fits with your spelling vision. Sure, it’s important for a name to be unique, catchy, and in context, but ambiguous spelling might cost you word of mouth clients, like it may be costing us. As a matter of fact, I really hope this page will be indexed by Google and other search engines real soon so that people searching for TestStuff but are actually looking for us will finally find what they are looking for.

  

Continuing where the previous introductory post left off, we continue to discuss the epic battle between the ruthless testing automation machines and the brave manual testing of the human QA army. To make it fair on both sides and let them wage their battles, let’s see what makes each of them so great and vice versa not that great. Automation goes first.

Testing Automation Pros

1. Accuracy and rigorousness. Testing automation is extremely accurate and rigorous. Be it counting (try to count to 20,000 out loud and see how far you get without making mistakes or getting tired), mathematical calculations, even the computer’s memory seems much more accurate than our human memory. While a human may skip a detail here and there in a manual test, the automation doesn’t accept any funny business. An FTP connection wasn’t established? Fail the test! The exact test message didn’t appear? Fail the test! Of course as long as the test was written properly, automation is always on full concentration, never gets lazy, and doesn’t miss a thing.
   
2. Speed. The automation performs tests faster than a human. It just runs through the tests without caring about any interrupted emails or Facebook messages, easily shuffling between whatever tasks it has to do with a flick of the command line.
3. Testing 24/7. Theoretically at least, putting aside malfunctions. OK, so maybe it’s more like 20/6, but still, automation can work many more hours than humans can and at times where most humans don’t like to work, like nights, weekends and holidays. It doesn’t need any rest, no lunch breaks since it gets a constant supply of delicious electrons right from the power plug, no paid vacations. It just works as long as it doesn’t break down.
4. Cheap cheap. An automation and maybe one person to operate and maintain it is cheaper than paying for a small team of testers who would have otherwise had to do the get the same testing done manually. Not much doubt about it. Spend a few thousand bucks for some machines, feed it aircon and electricity, get it maintained by a pro, and that’s it really. So you need more machines and more people to operate the automation? That’s OK, how many testers would you have needed otherwise and how much would they have costed? A whole lot more.
5. No complaints. It doesn’t matter how dirty, repetative, boring, or time consuming the testing is. Machines ‘gladly’ go at it (even though they don’t seem to have emotions), without any need for challenges or promotions or conditions or a raise.
6. No quitting. Your star tester may leave the team in favor of a better paying job at another company. Automation won’t quit on you, at least not by choice, at least not as far as we know. While people come and go, the automation is there to stay and continue doing its job loyally. It may malfunction, then again so do people when they get ill, angry, fall in love, or whatever obstructs them from doing their job.

Did I miss something? Disagree with one of the points? Make sure to comment about it, or forever hold your silence. Do join me next week when the humans strike back after the machines’ emotionless attack as we take a look where manual testing of human QA exceeds automated QA.

  

One of the songs by New Zealand’s Flight of the Chonchords, which is by the way an amazing show that music and Borat fans must watch, is about how in the distant future, the year 2000, the humans are dead. Robots used poisonous gasses to poison our a$$es. Robotic beings rule the world and the only dances left are the robot and the roboboogie:

Humor aside, some winds in the QA world are blowing this way, bringing forward the machine uprising, or Judgement Day if you will. I’ve recently been informed that a major company in my country fired an entire team of software testers because the automation can do their job!

I can imagine their linear line of thought. Testing automation is cheap. We don’t need to weed out thousands of dollars for monkey manual tester salaries. Instead we would only need to pay for the electricity the machine and the aircon take up, plus one person to operate it. It does the same job, maybe even better since the machine can run tests 24/7. The results could even be more accurate since the automation doesn’t miss a thing.

So what do you think about such a line of thought? There is an ongoing hype pro automation, that automation is here to replace all the manual testing that we do. Whatever repetitive action a person does when testing manually, the automation can do. Do you agree?

Having had some automation running and programming experience myself, I’m going to dive into this issue in a short series of posts. I hope you won’t be shy, and leave some good comments what you think about the whole man vs. machine issue in regards to testing, and in general if you wish as well. Stay tuned for more!

  

Today is a day of celebration - Testuff hit the 1000 registered companies benchmark! We never thought we’d make it so far so soon, but here we are with our crazy vision of test management and a 1000 followers behind us who at least at some point thought it’s a good idea.

So, thank each and every one of you 1000, and may we see thousands more!

  

Not sure how many of you noticed the blog’s sidebar, if at all, but there is a bunch of cool stuff there! Seriously!

The king of cool is a rather useful link called Dev & QA Tools. It contains a nice list of the wonderful tools we use in Testuff for RnD and testing. I’m sure you may find some gems there, especially since I have just updated it with powerful security testing tools. Giddy-Up! I’d also love it if you could leave a comment on that page about the tools that you enjoy so that we could all have the creme-de-la-creme utilities at our disposal.

Another link you might want to check out is Twittuff ( = Twitter + Testuff) under the links section. Hype or shmype, I use it to post tidbits about what’s happening in Testuff. Amongst other things, you’ll find there fresh news from our desks for all you hyper information junkies.

Also under the links section, I’ve taken the liberty to share with you some QA blogs that I read and enjoy on a daily basis, at least whenever they bother to post. Hope you’ll enjoy them too.

  

Sending out the word of mouth about Testuff to the further reaches of cyberspace has yielded some stimulating feedback. Out of all things that are Testuff though, what seems to really get people are the smallest things of all - the Testuff icons.

Selected testimonials below:

The GUI is a bit colorful and kitschy - it’s recommended to consult a GUI expert.

I know in your blog you mention it, but the icons are terrible for a business-oriented application. Far too cutesy. I wouldn’t want to demo this to my VP for budget approval.

…regarding the icons. I’m all for a different look but they are a little too fair to one side, professional can be boring but it’s the ‘professionals’ who will sign off on the money to purchase the product.

Let’s see those pesky icons again:

As I mentioned in the first post, when seeing the Testuff icons for the first time I too thought that they are a bit childish, that they look like game icons or IM icons and I recommended to replace them. What in the world is a monkey and some emoticons doing in a test management application!? This is unprofessional was my immediate thinking.

My problem, and I suspect that this is why we get such remarks about our icons, is that we’ve been conditioned. We’ve been conditioned by GUI designers, or lack thereof, that icons in test management software, or any professional software for that matter all look the same. They are boring. They have the same sleek, shiny, generic unisex look with gray all around.

For instance, just take a look at 3 different IDEs - Eclipse, Visual Studio, and Borland Power Builder. Putting aside functionality and everything else but the GUI design, don’t they look pretty much the same? See for yourself, compare and contrast:

Yes, but icons have to look professional since the decision makers who will decide whether to buy Testuff or not are big time managers… It may be hard to believe, but behind every professional and big time manager, well, maybe behind most big time managers, there is a human being. And this human, does he or she want more boredom? More gray? More of the same? Maybe some of them do. Or at least pretend so. That’s because those above and around them pretend that that’s what they want too.

At one of my previous workplaces there was a mess hall where we would eat our chow at lunch time. That place was designed to keep you out of there and send you running back to your desk ASAP. All the windows were covered so that you couldn’t see at all what was going on in the outside world. The colors decorating this great hall were gray and black with a few small exceptions of the company logo in yellow and orange. Everyone sat at big tables and didn’t enjoy their food to the constant buzzing in the background of people talking about work.

Say there was an alternative mess hall. One with big open windows to the outside, with smaller booths to hang out, a place that looked like one of those colorful lounge bars with funky pieces of vintage decor you wouldn’t see elsewhere and unusual lighting. Which one do you think most employees would prefer? That gray prison which makes you want to escape back to your desk in despair, or the lounge bar? I clearly say that the answer is door number 2. Let the accountants, the kiss up “example givers” and whatnot have their gray no-outside world mess hall.

So if we like color, fun, and uniqueness in our lives, and we sometimes go to extreme lengths to get it, like traveling half way across the world, or spending a lot of money on things that look great, why the hell do we settle for boring GUIs? I’m afraid that the answer is just because it has been this way so far, and blimey, we got used to it.

Yes, the Testuff GUI will alienate some people as being “unprofessional”. That’s perfectly fine. We don’t write Testuff for these kind of square minded boring go with the flow kind of people anyway.

The kind of people we write Testuff for are the alive and kicking. The thrill seeking testers who love life and expose the application under test as fuller of holes than Swiss cheese. The kind of people that have fun doing QA, that see colors in the gray world that society tries to impose on them.

Testuff along with its funky icons is written for these in mind, to be different, funky, and colorful. Not to look like all the other test management software out there, and not to be like the other test managements out there. Testuff was founded on the premise that most test management software sucks and feels as a burden for the QA that are forced to use it. However, the stronger premise is that it doesn’t have to be this way. Test management software can kick ass just as good as other software, and look great while doing it.

I apologize if this may seem like some sort of marketing speech, but believe it or not this is the philosophy behind Testuff. Plenty of people may reject it. Let them do so, allow them to dis our icons. But if somewhere in there you relate to some of the things I said, give those cutesey icons some time, and they may grow on you. Maybe you’ll start to see things different, at least to break away from the boring GUI = professional GUI conception that has been imposed upon you.

  

I’ve got some hot material to blog about, and I will do so, soon In the meanwhile why not enjoy a real world testing scenario of a buggy water bed: