Think of your company, project and team and you’ll see it applies nicely.

Built right

Teams that are built right, have more success. Germany, Spain, Netherlands are teams that come from countries with a tradition of working right with players, starting at a very young age. Nothing is left to luck or dependent on talent. hard work, teaching the basics and long term plans brought them to the top.

• Hard work – working with kids, taking them all the way, step by step, until they are ready and old enough to join the “real thing”. There are no compromises about training, more than once a day for many years.
• No room for luck – starting with little kids, all the training, diet instructions, psychological aspects and physical conditions are all designed and monitored by the best experts. Plans are made carefully and trainers, coaches, doctors and others are following each step making sure everything is done as it should be.
• Talent is not enough – a team can’t rely on one or even a few talents. A team has to be a team, each one contributing his part in the total effort to achieve the team goals. Building the right mixture of roles, and making sure all are in social harmony is important as much as each individual talent (and take Argentina as an example for talents vs. team work and achievement).

Work Gradually

Many teams try to skip the work and time of gradual and right process. They spend money (lots of it) on bringing the best players in an effort to win NOW. In most cases it doesn’t work. The examples are many – take Real Madrid football club and its failures to win the championship in the Spanish league, although bought all available expensive players. On the other hand, Barcelona has one of the best youth football academic clubs and they enjoy the products of it every year in the shape of 1-2 great new players (so many examples here, best known is Lionel Messi who grew in the Barcelona football club academy).

• Don’t just bring the best there are out there. Make sure your environment and culture encourage learning, professional growth and get yourself better “home players”.
• Make sure your “players” can work together. Sometimes 1+1 is more than 2 and sometimes it might be less. A good team is built slowly, making sure each of its members is the right fit, not only for his role, but in the greater picture of the team as well.

Team, Team

We’ve mentioned above already the need to work as a team. France was kicked our early because they were not a team, had many internal issues between players and between players and management. Same happened to the Netherlands in a few previous tournaments, were although had many talents and great opportunity failed to win.

Your team is a group of individuals who work together. They have to be synchronized, pull the wagon to the same direction. Each one has to know what is his part of the overall work, where can he contribute mostly and how. They must be able to help each other, overcome mistakes together and take advantage of good ideas and opportunities.

They also must trust their team leader and have confidence he leads them on the right direction. You know what happens if the coach says one thing and players do something else, each one what he thinks best.

Were would we be without Passion

It was obvious during the games that teams with less talents can try to close this gap with a passionate game. Have you see the South Korean running? Did you watch Uruguay “burning” the field or the USA squad giving it all?

Now imagine you have an A team with a few great talents, with this kind of passion….

Make sure your team is passionate about their work. Involve them, give them good example, make them believe – and they’ll bring their best act.

See you in Rio in 4 years

We like to think that since we are part of this industry, and have a dialog with others in it (such as customers, experts, competitors and others), we know what’s currently going on and where is it going to (as much as this can be known). However, it’s always good to get some feedback on this “knowledge” from researchers, who study the industry in a more methodological and thorough way.

The most common prediction is getting more and more votes. A recent study by TechNavio Insights shows that testing – like many other software development units (and IT to this matter) – it is moving to the cloud.

Is it a bird? Is it a Plane? It’s Testing

The general term of Cloud hides behind it a few different terms related to testing. There’s the SaaS (Software as a Service), the TaaS (Testing as a Service) and the PaaS (Platform as a Service, on-demand application development platform). The new research shows that many companies, including big-size ones, understand the importance and the opportunities of the Cloud and are building their services to be available in this new form.

This goes even further when we look at new offerings (from new or existing companies) that are designed to be used as over-the-net, SaaS, Cloud applications (and you can pick your favorite term). Obviously customers are not happy to pay anymore the amounts previously paid for software, in advance, and then to add servers, people to maintain them, yearly application maintenance, upgrades, etc. Why should they, if they can subscribe rather than buy, pay for the usage only rather than commit and at the same time save all the hassle of installations.

What started with the hardware is very quickly spreading to software, and specifically to software testing. Look at all the tools available now online – test management, bug trackers, automation, mobile units virtualization, project management. You name it. So far it was the new small and mid-size vendors, but we expect the big players to join. They have no other choice. 

Need for Proof

Of course. Vendors of software testing application will have to stand to the same quality those “regular” apps have. Security has to be even tighter, and performance and availability must be superb. The challenge is huge, even before mentioning the different approach to service (including support, upgrades and on-going maintenance) that these vendors have to adopt.

Unlike installed solutions, the ease of switching to another solution provider in the Cloud will enforce vendors to take their standards to a higher level – at least those who will want to stay in the game.

It’s the golden era for customers – they will soon get better solutions, easier to manage, better services and in a lower price. The opportunities for customers are huge – solutions of TaaS will enable them to re-size their testing teams at will and reduce costs significantly. They will be able to use many testers in a short notice and for a limited time and perform much of the testing efforts faster. We believe there will be vendors offering solutions, which will be combination of community access (testers) and products (testing platforms and solutions).

Software testing has a bright future. After recognized in the past few years as an important part of the development effort, testers being part of the project teams and participating in all phases starting with the design, we will see more companies doing testing (easier and cheaper), doing more testing (testers available online and on-demand), using Cloud based solutions for the testing process. 

We’ll be there.

Usually after a short dialog, they find the best way for them to use Testuff. We’re quite proud that Testuff does indeed fit with many different methodologies. However, the question I always wonder about is: Why do most people (i.e. testers) think that somewhere, over the rainbow, there’s a magic methodology which flies…. In my mind, it is almost obvious that no methodology can replace common sense.

         “Common Sense Is Not So Common”, Voltaire

Keeping the testing process efficient, simple and manageable is sometimes more important than finding and adopting a methodology, which doesn’t necessarily fit in with the company, its processes, people and attitude.

This post is not intended to be philosophical and does not attempt to create yet another testing methodology. The intention is to simply try and describe my “common sense” testing approach for those who want a simple 5 step guideline.

5 Steps to follow                                      

The first two steps involve some preparation:

1. Get your requirements! no mater if you are a one-man-band or a global multinational organization, if you don’t have any requirements, you can’t test. You can even have “The site should not crash, and it should be accessible using FF and IE” as your only requirement. But if you have nothing, you may be able to do a great development work but not when it comes to testing.

2. Write a simple test script for each requirement. The test script is an instruction how to ensure the requirement is fulfilled. You can write it in a simple language: “Check that all links work”, “Check that all links have a proper title when the mouse hovers over them”. You could also use any format of step/expected result. Just write something easy to understand. I don’t have any particular advice on how to organize your test scripts. Some prefer to group them by requirement (each test under its requirement), some by the product areas, some by type (GUI, DB, etc). I prefer to put them in a place where I can find them

The third step is a mental step, just to bring you to a “testing” mode and make sure it is done correctly. Think zen.

3. Freeze the test environment. No matter if you have separate environments for development, testing, release and production or if you use only one environment for all (which is probably a bad idea). The golden rule is to freeze everything before and during testing. If someone changes your test environment whilst testing, all your effort goes down the drain and you might as well not do any testing at all. Now for the actual testing process.

There’s a little twist that will make your testing better on every cycle:

4. Start testing (and improve your scripts). Just run the Application you test and follow the test scrips you wrote in step 2. If the Application fails on a particular step of the test script, simply open a defect to be fixed later by the development team. If you test something that was not specified on the script and find a defect then (This is the continuous improvement part) add it to the script and fail the test. If you find the script irrelevant or not up-to-date then update it.

* Sub-step #2 enables you start with a very basic test script, even an empty one! and fill it in as you progress with testing.

   That’s it. All done.

Another step, to be carried out outside of the testing cycle, which will help you to get better coverage of your application is:

5. Prepare for the next testing cycle. Whilst running the application in production, you will no doubt find some defects. Don’t worry! everybody does… The key is however to avoid these defects on the next release of your application. You can make sure you improve by updating the relevant scripts for the processes in which the defect was found in the first place. This way, next time you test the application you make sure to check that the defect was fixed. It also serves well for sanity / regression testing purposes.

That’s all.

The testing world has some great methodologies, and I’m sure you can improve this “common sense” five step process. The main reason for writing this post was to encourage you to start testing your application! Instead of hiding behind “it’s too complex”, “I don’t know how to”, “We need an expert to drive our strategy” excuses, simply jump in and get started. This is what Testuff is all about.

Bug validations can be considered quite a tedious task. Finding a bug is quite fun, a treasure hunt except you don’t get the luxury of a map with X marks the spot. Bug Validations however do include that X, but in most cases after digging the ground you won’t find a treasure, or in other words a bug. Yet, what if you took a few steps to the right and dug again? Looked underneath a rock? Shaken a tree? There might be treasure there.

I do agree that validations are somewhat important to make sure that bugs have actually been fixed, because from time to time the devs do mess up (see my previous post). However, instead of being bored out of your mind as an automatic validating machine, use validations as a jumping board to find new bugs:

• The bug may have been fixed locally, but not globally. Some bugs may reproduce again in a different screen or slight different configuration. Ask yourself where else the bug may appear other than the specified screen and configuration mentioned in the bug report. Take a few minutes to test it.
• Changing the code probably inserted new bugs into the system, unintended side effects. It is up to you, the validator, to find them. How? Keep your eyes open, especially around the bug. Do some exploratory testing in the area, make sure no additional unintended harm was done.
• Invent and execute new tests. Use validations as an excuse to do additional testing, maybe even for a “stable” section of the software that hasn’t been tested for a while. There could be old bugs there, performance issues, overlooked security issues, silly spelling mistakes everyone has overlooked, or even room for enhancements. And all this without bearing direct relation to the fixed bug.

Happy bug hunting!

Unfortunately, it took us longer to release Testuff 1.1 than what we had anticipated. So, after it was finally out, we gathered round the round table and did the aftermath. This is what we found out:

• We didn’t take holidays into account. The month of September was plagued with Jewish holidays. Even though we do work quite often on non-working hours and on weekends, the holidays still slowed us down, making us go to family dinners and sleep more with lot’s of food in our stomachs.
  Conclusion – Take holidays into account for the next milestones.
• Incomplete feature design. We wanted to implement an improvement in the infrastructure. Unfortunately we found out way too late into the coding that the design had failed to take into account certain scnearios and thus the feature could not be implemented properly. The code was then reverted, coding time flushed more or less down the drain.
  Conclusion – instead of trusting the design with one man only, discuss it with more people as early as possible and include QA in the process.
• Last minute major changes. Due to some performance problems in the My Tasks feature, we made last minute major changes to fix them. This caused an array of bugs and extended the testing and fixing cycles beyond what we thought.
  Conclusion – better to wait for the next version or patch with such major changes. This way the release candidate can be stabilized faster for release.
• Lack of communication. Even though I sit in the same room with the devs, we didn’t communicate about the new version enough. Turns out there were some infrastructural changes that caused bugs, and we are going to have to release another patch to fix them.
  Conclusion – hold testing meetings and discuss changes in the new version and what should be tested in particular.
• Too many failed validations. They might not be a major time killer, but they sure are annoying and most of them can be eliminated. We discovered that some bugs were closed as fixed even though they weren’t fixed, they just didn’t reproduce for the dev. Another thing we found out is that sometimes the dev didn’t understand the bug and just closed it. The bugs had to be reopened, refixed, and retested.
  Conclusion – if a bug fails to reproduce or is incoherent, talk to QA before closing it.
• Side effect bugs. Certain bugs, even though that they themselves were fixed, caused side effect bugs that could’ve easily been fixed had they been noticed following the initial fix. These kind of bugs increased the testing cycles we needed to stabilize the version.
  Conclusion – after fixing a bug, take a minute to poke around and see if anything related went wrong.

We have learned some lessons because of this version and I hope they may be of use for you as well. Did you also run into similar situations? What other lessons have you learned from delayed releases? Share it with us in the comments or write a post about it and link back to this one.

• Software Testing Club
• SQAForums
• LinkedIn
• Israeli QA Forum (in Hebrew)

The people have spoken. All of the above discussions were great and yielded some powerful feedback, so I’d like to say thanks to everyone who participated.

Below is a summary of the most common opinions about recording test and bug videos following the above discussions. Even though Steve Swanson beat me to the punch with his summary about recording test videos I would like to present my own. Here we go!

Most people don’t record test videos for tests and bugs

1. Text and screenshots are sufficient.
2. Video file sizes are too big and take up too much space on our servers and databases.
3. It takes too much time and hassle to take videos of tests and bugs.
4. Videos can’t be searched.

Some people do record test videos

1. As an aid to bug reports, not a replacement.
2. When doing Exploratory Testing.
3. For hard to reproduce bugs.
4. For complicated scenarios.
5. For bugs that are graphical by nature.

Conclusion – videos of tests and bugs are useful in certain cases, but aren’t necessary in others. They really help as an addition to a textual bug report, but cannot completely replace it.

Therefore, we shouldn’t expect to see test and bug videos being recorded all the time, and maybe even not most of the time. Yet according to our statistics we rarely see test videos being recorded!

Some people seemed to have a certain resistance to testing videos mainly due to technical constraints. However, most of them simply aren’t true.

1. Text and screenshots are sufficient
   As quite a few people noted in the discussions, there are cases when text and screenshots aren’t sufficient. Such is the case with hard to reproduce bugs, complicated bugs, graphical bugs, and more.
2. Video file sizes are too big
   We use our own custom and top secret compression algorithm to make test and bug videos really small. Download the Testuff Video Player and check out an example bug video of a silly bug in Testuff 1.0 with the forgot my password feature (to be fixed in Testuff 1.1 . The 11 second video only takes about 74 KB. With most bug videos being around 10 seconds, there shouldn’t be a worry about file sizes.
3. Video files take up too much space on our servers and databases
   Testuff videos for tests and bugs are actually uploaded to the Testuff servers, so it’s our worry if they take up too much space.
4. It takes too much time and hassle to take videos of tests and bugs.
   When running a manual test in Testuff, the test runner already includes the video recorder controls. All you need to do is target a window or area to record, and hit the record button. If some step fails, the video is automatically attached to the bug report. See the video recording help section for more info.

Of course, there is a whole lot more that could be done with testing videos to make it even better and simpler, and we have plenty of interesting ideas. Still, assuming that we did solve these technical constraints in a decent manner, we should see more videos being recorded.

The only conclusions that I can draw from the above is that either people are way too lazy to make videos, or the idea is still too new so that only a few stray sheeps are using it while the rest of the herd continues along the known path.

Well, if you aren’t recording test and bug videos today and haven’t even tried, you might be missing on something. There isn’t a need to record a video for every bug, but quite often they come in very handy. Give it a try. Besides, recording videos can be fun and magical too. I highly recommend you to watch Be Kind Rewind and see why:

We’ve seen why automated testing rocks. As I’ve stated before, it’s more accurate, work faster, harder, for a longer period of time, for less money, with no complaints and no quitting. We’ve also seen why human manual testing rocks. Humans are self maintaining, adaptive to the environment, creative, recognize objects better than machines, have built-in natural language processing, and also excellent learning abilities.

So what is the summary of it all? Fire all humans and replace them with automated testing machines? Throw all automation out the window and hire some testers for cheap from India? No!

The world doesn’t actually divide into black and white, but we quite often mistakenly think it does (a bug in our brains? The misguiding of Western society? Both? None? who knows). We don’t have to be crazy like that Israeli company who fired all testers in favor of automation, though who knows, maybe for their needs of completely repetitive well-known testing it was the right move. We also don’t have to reject any kind of automated testing at our company and stick to flesh and blood testers only. The answer is that we can enjoy both worlds, the shades of gray in between the black and the white.

Be a cyborg. Part man, part machine, it has all the advantages that humans do, but some off the hook technological enhancements that allow it to have beyond human capabilities. To take the analogy back, this means that we enjoy the best of both worlds – the automated testing world and the manual testing world and use them both to our advantage:

1. Automate boring, repetitive, high accuracy tests. Humans don’t like to do them and don’t do them as well as the automation does. By automating these kind of tests your QA will be happier since you will lift the annoying load off of their shoulders and you will get better testing results via the automation.
2. Give humans creative testing tasks. As a complimentary item to the previous one, challenge the human testers, use their natural learning skills, adaptivity and creativity to find bugs way outside the scripted tests. Don’t let them spend their time only on pre-scripted scenarios. Let them do some Exploratory Testing, new test designs, maybe even learn some programing so that they could write some useful utilities.
3. Work the automation overtime, not the humans. Having people do overtime for too long under pressure increases work stress and basically drains all the motivational juice out of them. This is why high-pressure testing, such as sanity, might be better put on the automation. However, watch out not to pressure the humans who execute the automation!
4. Use low maintenance automation. It’s a real pain in the ass when the automation requires too much maintenance in terms of time and effort. Don’t be afraid to go back to the drawing board and design your automated testing from scratch. Check why it takes such high maintenance and either solve those issues or move tests to manual execution if necessary.
5. Humans test some things better than automation. Given the current technology, computers don’t know how to do many things as well as the humans. In the testing world this is especially true about GUI. Sure, there are solutions for web testing and functional testing for simple GUIs, but if your app is more complicated than the average software Joe you will quickly see that the automation doesn’t do. If these automated tasks suck for your testers, maybe even consider off-shoring them to some Indian testers who wouldn’t mind doing some repetitive tasks for good cash.

If you still don’t believe me that you should become a cyborg, watch the trailer for Robocop and see how cyborgs improve law enforcement in crime filled future Detroit. No man and no machine would do to clean Detriot of its thugs, but only a man machine combination that is Robocop:

But wait! It is time for the humans to fight back, to show why in certain aspects their manual testing is way better than any testing automation can be. Let’s see what they got going for them that the silicon based life forms cannot beat.

Human Testing Pros

1. Self maintenance. Humans are (usually) capable of taking care of themselves. If there is some sort of system malfunction like an illness, hunger, sadness, or whatever, humans can at least attempt to solve it by themselves. The automation cannot do so. It needs someone to maintain it. If some sort of malfunction happens, human intervention must take place. Be it a network cable gone loose, a fallen web service, an air-con malfunction, the machine cannot go about taking care of itself.
2. Adaptivity. Testing automation lacks any kind of intelligence. If, for example, a connection failed to open, it won’t go about investigating what happened. A whole series of tests could fail because a network cable went loose, thus rendering hours of testing automation useless. A human on the other hand can use her wits to investigate what happened, find that loose cable, and go on with the proper testing. Also, if some change in the AUT was made the automation will probably fail. The human on the other hand will easily determine if the change is acceptable and act accordingly.
3. Creativity. Whereas the computer follows the scripted scenario religiously, people look in between the scripted lines and outside of the test altogether. They will find bugs even in things they weren’t supposed to test and be able to come up with new tests for the future. Of course, let’s remember it was the humans who programmed the testing automation to begin with, it isn’t able to design tests on its own.
4. Object recognition. Even a five year old could easily beat any computer running any algorithm when it comes to recognizing objects. Thus, and despite so many efforts made, testing automation has problems testing GUIs. A human on the other hand doesn’t really care how a UI was implemented. She can easily recognize buttons, windows, scrollbars, or anything else and get testing with it. She can see if the colors are OK, if images are displayed properly, videos are streamed in good quality, etc., things that the testing automation has big trouble doing.
5. Natural language skills. Humans may give and receive abstract instructions in natural language and act. A human tester receiving an instruction such as “test the site also works with IE7″ can easily go about doing so. The testing automation must be programmed with exact instructions in a programming language. It takes much more time to give instructions to machines and involves testing and debugging, once again by humans, to make sure the automation “understands” the instructions properly. Not to mention that automation cannot test something like a help file, that it contains good help with correct grammar. It just doesn’t understand the human talk, not out loud, and not even if we hand it in a text file.
6. Learning. Humans can learn new features, new technologies, different ways of doing things, and append new knowledge to their hard disks. Testing automation cannot do so. It needs the programmer to step in and give it exact instructions what to do. Otherwise it will never expand its knowledge base and just keep doing the same things that it knows forever, not doing anything new at all.

Once again, feel free to respond if there is something missing in the human onslaught against the testing automation. Do join me once again for a summary of the fight between human and automated QA!

Testing Automation Pros

1. Accuracy and rigorousness. Testing automation is extremely accurate and rigorous. Be it counting (try to count to 20,000 out loud and see how far you get without making mistakes or getting tired), mathematical calculations, even the computer’s memory seems much more accurate than our human memory. While a human may skip a detail here and there in a manual test, the automation doesn’t accept any funny business. An FTP connection wasn’t established? Fail the test! The exact test message didn’t appear? Fail the test! Of course as long as the test was written properly, automation is always on full concentration, never gets lazy, and doesn’t miss a thing.
   
2. Speed. The automation performs tests faster than a human. It just runs through the tests without caring about any interrupted emails or Facebook messages, easily shuffling between whatever tasks it has to do with a flick of the command line.
3. Testing 24/7. Theoretically at least, putting aside malfunctions. OK, so maybe it’s more like 20/6, but still, automation can work many more hours than humans can and at times where most humans don’t like to work, like nights, weekends and holidays. It doesn’t need any rest, no lunch breaks since it gets a constant supply of delicious electrons right from the power plug, no paid vacations. It just works as long as it doesn’t break down.
4. Cheap cheap. An automation and maybe one person to operate and maintain it is cheaper than paying for a small team of testers who would have otherwise had to do the get the same testing done manually. Not much doubt about it. Spend a few thousand bucks for some machines, feed it aircon and electricity, get it maintained by a pro, and that’s it really. So you need more machines and more people to operate the automation? That’s OK, how many testers would you have needed otherwise and how much would they have costed? A whole lot more.
5. No complaints. It doesn’t matter how dirty, repetative, boring, or time consuming the testing is. Machines ‘gladly’ go at it (even though they don’t seem to have emotions), without any need for challenges or promotions or conditions or a raise.
6. No quitting. Your star tester may leave the team in favor of a better paying job at another company. Automation won’t quit on you, at least not by choice, at least not as far as we know. While people come and go, the automation is there to stay and continue doing its job loyally. It may malfunction, then again so do people when they get ill, angry, fall in love, or whatever obstructs them from doing their job.

Did I miss something? Disagree with one of the points? Make sure to comment about it, or forever hold your silence. Do join me next week when the humans strike back after the machines’ emotionless attack as we take a look where manual testing of human QA exceeds automated QA.

Humor aside, some winds in the QA world are blowing this way, bringing forward the machine uprising, or Judgement Day if you will. I’ve recently been informed that a major company in my country fired an entire team of software testers because the automation can do their job!

I can imagine their linear line of thought. Testing automation is cheap. We don’t need to weed out thousands of dollars for monkey manual tester salaries. Instead we would only need to pay for the electricity the machine and the aircon take up, plus one person to operate it. It does the same job, maybe even better since the machine can run tests 24/7. The results could even be more accurate since the automation doesn’t miss a thing.

So what do you think about such a line of thought? There is an ongoing hype pro automation, that automation is here to replace all the manual testing that we do. Whatever repetitive action a person does when testing manually, the automation can do. Do you agree?

Having had some automation running and programming experience myself, I’m going to dive into this issue in a short series of posts. I hope you won’t be shy, and leave some good comments what you think about the whole man vs. machine issue in regards to testing, and in general if you wish as well. Stay tuned for more!