Exploratory Testing with Testuff

How to work with Testuff when using Exploratory Testing as your methodology

There are many different approaches to software testing, with most methodologies requiring the user to develop preconceived ideas about potential defects and their causes. However, one branch of software testing offers the benefit of truly objective investigation. Whether used as a standalone methodology or in tandem with others, such Exploratory Testing allows you to uncover unknown defects as you prepare software for the marketplace.
The purpose of this article is to describe how you can use and benefit from Testuff’s test management solution for your exploratory testing.
When it comes to exploratory testing, the quality of a test management tool is measured by two main criteria:

  • Speed. The tester’s valuable time must be used and managed as efficiently as possible. You want a tool that allows users to easily and quickly start testing.
  • Documentation. The extent, ease, and depth with which tests and results can be documented. The ultimate goal centers around one’s ability to reproduce the defects found, without any additional information requests and communication between testers and developers. This includes the:
    • Ability to open a test with no content, just as a general idea or direction.
    • Ability to write the test while executing it, based on actual actions taken.
    • Test execution recording. A way to show the developers the findings/defects found in a comprehensive and visual way.

Now let’s see how you can use Testuff for your exploratory testing.

______________________________________________________________________________________________

No Time to Explain… Just Test!

Have you been in the situation when there is lack of product requirements or insufficient time for formal testing? And yet, you still need to ensure that the quality of the product is high?
Many inexperienced testers simply begin hunting out bugs without any specific strategy. With this approach, you can find a lot of defects, but there is no guarantee that you’ll test the most important functionality properly. This careless chase is often called ad hoc testing.
More skilled engineers remember that the goal of the tester isn’t to set a world record in the number of defects found. Rather, the focus should be on objectively analyzing the quality of new products – a process that requires walking in with no preconceived notions. There are many methodologies for accomplishing this, but exploratory testing is one of the most effective. Although it relies more on execution than on planning, the process adheres to clearly defined steps that allow us to more thoroughly understand the product, tests, and results – from start to finish.
Director of the Center for Software Testing Education and Research, Cem Kaner, originally coined the term “exploratory testing” in 1983. According to Kaner, the process includes the following steps:

process

Exploratory testing not only helps us learn how a given product works. It also helps us understand priorities, define requirements, and test the most critical components of that product in the first order. The process is iterative, allowing one’s understanding to grow with each subsequent cycle. This process makes it easier to isolate and test the software’s core functionality.

Software Testing Tools

Cataloging testing results across paper notes, electronic documents, and issue-trackers isn’t always convenient. For exploratory testing, we need a tool that allows us to:

  1. Document the requirements we’ve found.
  2. Add brief descriptions before expanding tests.
  3. Execute tests.
  4. Record the new results.
  5. Share these results.

Given the complexity and importance of these tasks, you need a software testing platform built around sound requirements and robust documentation. The tool must include:

  • Issue trackers
  • Screen capture functionality
  • The ability to add additional files and notes
  • Reporting capabilities (for sharing)

Furthermore, the software test management platform and issue tracker software must be seamlessly integrated.
In the past, you would need several different tools working in tandem to deliver these results. But now, there exists a unified platform that integrates all of the above. Testuff is an all-in-one solution that removes the need to rely on numerous tools, editors, trackers, and electronic files. In addition, the Testuff suite includes video recording capabilities, allowing you to send information directly to your favorite issue tracker. All in one place for your exploratory testing needs.
As an added bonus, Testuff integrates all of these exploratory testing steps into one unified screen. Let’s review.

Learn

First, we need to learn how the product works. Go through the main use cases and isolate the user roles, the objects they work with, and the workflows required. When learning the features, focus on the results (i.e. output) to better understand what input data steps are needed to produce those result.
In Testuff, keep everything you’ve learned in the Requirements section. You can add a sketch of these requirements and then expand them with additional information.

To quickly add the set of found requirements, follow these steps:

  1. Go to the Requirement tab.
  2. Click Add requirements now!
  3.  

    req1

  4. Type all requirements, separating them from each other by pressing Enter.
  5.  

    req2

  6. Click OK to save the requirements.

You can easily add additional information to any of the requirements while learning more about the product. Simply double-click any requirement in the list or right-click the requirement and select Edit.

req3

Testuff integrates with external requirements management tools. If you already have part of the requirements stored somewhere else, you can easily import them directly into Testuff. The synchronization is bi-directional, so any requirements found and recorded in Testuff can also be sent to the external tool.
It’s worth noting that Testuff allows you to add requirements in any format you wish, whether it is a feature in the backlog, user story, use case, or even a classical IEEE record. You can also attach documents and images to the requirements.
You will need the help of someone who is intimately familiar with the product (e.g. Product Manager, Product Owner, Business Analyst, etc.) to set the testing priorities for the requirements and verify whether you understood the requirements correctly. Change the priorities accordingly in Testuff.

Design

With the basic product information clearly defined, the next step involves outlining future tests.
In exploratory testing, time is usually critical. And thus, you want to be able to design the tests as quickly as possible. Fortunately, Testuff removes the need to jump from one tab to another – all exploratory testing work can be done right in the Requirements tab.
The first thing to do will be to create Suites, based on the requirements you have. Right-click a requirement (or a group of them) and select Linked suites->Add suite and link it. This will create a suite, with the name of the requirement, in the Tests screen. This suite will be Linked to the requirement, which means that Testuff will handle the relation between the tests in the suite and the requirement.

Now that you have the suites, you can start creating a list of tests. This is straightforward:

  1. In the Requirements tab, select a requirement you want to test.
  2. Click the Assign more tests tab.
  3.  

    req4

  4. Right-click the All available tests option.
  5. Select Add new or Quick add tests in the pop up menu.
  6.  

    req5

  7. Click the suite you want to include the new test in (Select suite dialog).
  8.  

    req6

  9. Type the name of the test, steps, and expected results. The example down below is a description for a simple “positive” test. Click Save & Close when you are done.
  10. req7

For exploratory testing, we already have enough information to begin executing our tests. We can only make educated guesses about the tests and expected results at this moment. We’ll validate these data in future steps.

Execute

Once the initial tests are created and linked with the requirements, we’re able to run the test and learn more about our assumptions. We want it to be easy, and fast. Testuff does exactly that for you – run, record with video, report defects and comment.

  1. In the Requirements tab, click Tests in this requirement tab.
  2. Our goal for now is to verify how the product works and record the test steps. The execution protocol is important for keeping the “explored” knowledge and interpret the results. We might also find some defects during this stage.
    It is recommended that you use Testuff’s video recorder to easily log everything for future reference.
    Follow these steps:

  3. Right-click a test and select Run.
  4. req8

The Test runner window will open.

req9

Now initiate the video recorder to capture all of your actions while testing:

  1. Click icon1 or drag the icon2 icon, and trace out the desired screen capture area. Click icon3 to start the recorder. Test your application while Testuff captures all clicks, mouse movements, new screens, and messages with the video recorder tool. Everything is carefully documented so you don’t need to worry about keeping any logs.
  2. Note that new recorder controls become active:

    req10

    These controls allow you to Pause, Stop, and Save the video recording process.

    When you have finished the test, click icon4 to stop the recording.

  3. The recorded video, which is now linked to the test, is essential to the exploratory testing process. It allows you to easily reproduce the steps taken during the test – without overlooking any of the details. In addition, the video also allows you to analyze whether the tests and results match up correctly.
  4. Writing and re-writing of tests, while running them, is an essential capability of a testing tool when doing Exploratory Testing. It’s a Must have feature.

  5. Click Edit test to describe test steps with their expected results.


  6. req11

    Interpret

    Now that you have enough information, you can create the final test cases for further regression testing:

  7. Play back the recorded video to identify test steps and expected results by clicking icon5.
  8. The video player not only shows the captured video, but also cursor trails, mouse clicks, input data, and keystrokes.

    req12

    If you have found any defects while executing the test, click the icon6 icon (Report a bug not tied to a specific step), or select icon7 near the specific step of the test to tie the bug report to that exact step.

    req13

  9. Type the bug report description in the Bug Reporter. Captured video is already available in the Watch the video box.
  10. req14

Keep in mind that any tests you’ve created and defects you’ve found are only assumptions until they’ve been confirmed. Refer to someone who knows more about the product to help you understand whether your test cases, steps, and expected results are accurate.

Complete this process on those requirements with the highest priorities. And make sure the tests for these requirements are confirmed and executed. You are now ready to do another round of the Learn, Design, Execute, Interpret cycle. Repeat this cycle to obtain more knowledge about the product, and test these new assumptions as needed.

process2

When you have a set of ready test cases, you can delegate the test execution to other team members.
In the Test in this requirements tab, right-click a test and select Assign… in the popup menu. Designate the desired tester from within the list.

req15

In this way, the exploratory testing process easily scales up, allowing multiple team members to add their own input.

 

Additional Notes

  • Watch out for the Pesticide Paradox – a common situation in which repeated testing of the same defects often results in a decreased ability to isolate said defects. In order to ensure the effectiveness of the tests being performed, you must frequently revisit and rewrite the tests. Otherwise you risk glossing over unexposed code and missing out on some of the core functionality of the product.
  • Also, focus your attention on defect clustering per the Pareto Principle. In general, 80% of defects are caused by 20% of modules. So if you find a disproportionate number of defects somewhere in the product, it makes sense to add more specific tests to verify this functionality further. Fortunately, Testuff allows you to easily expand both requirements and tests, without sacrificing traceability between the two.
  • Working from the Requirements screen is a great approach, however you can get the same results while using other screens as well, mainly the Tests screen.
  • Here’s a “cheat sheet” that can help you remember each step of the Testuff exploratory testing process. It covers the entire exploratory testing lifecycle, from isolating requirements all the way up to reporting defects.



process-summary