During a conversation I had with a seasoned QA manager in a large communications company, I tried to present the Exploratory Testing approach. His reaction was surprising: “but we are already doing this, when we finish running the tests or when we are short on time, I cut my testers loose and let them freewheel and that’s how I find most of the bugs.”
This description is one way of performing ET, but it is not very good way. Isn’t it unreasonable that the most productive approach (most bugs found) is applied in an uncontrolled and unmethodical way, while order and method are kept for the less efficient approach?
These types of reactions result from misunderstanding the Exploratory Testing approach. In this article we attempt to explain ET principles by analyzing common misconceptions about ET.
The most well known definition of exploratory testing (ET) was coined by James Bach and Cem Kaner:
Exploratory testing is simultaneous learning, test design, and test execution
This definition of exploratory testing was the prevailing definition for many years. The new descriptive definition says:
“Exploratory testing is an approach to testing that emphasizes the freedom and responsibility of each tester to continually optimize the value of his work. This is done by treating learning, test design, and test execution as mutually supportive activities that run in parallel throughout the project.”
First of all, Exploratory Testing is not a technique, it is an approach. Techniques, such as automated testing or boundary testing apply the Exploratory Testing approach. A technique is way of doing something; approaches are broader notions, they’re like additives that are applied to techniques.
Many people think that Exploratory Testing is an approach that does not require documentation, but, in fact, it can be very documentation oriented, explicitly structured, and highly rigorous. The only difference between Exploratory Testing and Scripted Testing is that in ET the three main activities that comprise testing (learning, test design and test execution) are done in parallel, are mutually supporting activities, but are not necessarily consecutive. This doesn’t mean you can skip or avoid doing any one of them. All it means is that instead of doing these three activities separately, in Exploratory Testing all three of them are carried out in a mutually supportive way. For example, when executing a test, new information and ideas might come up that can help the tester both in creating new tests and in better understanding the product (that’s why in many organizations the testers are the most knowledgeable people about the product).
All testing is structured! The question is: how is it structured? Unskilled exploratory testers often think that ET is not structured because they are testing unconsciously, i.e. different tests are conducted in each test cycle.
Indeed, poorly executed ET can result in incomplete coverage, just like incorrect Scripted Testing can lead to the same outcome.
In many organizations, designing a test really means translating or re-writing the requirements documents into test cases and test documents. This can be done using a fully automated approach that doesn’t involve any thinking. Managers might think that this ensures coverage of the system under test, but in most cases it only covers verification of the system and does not validate that the system really works. In most cases, bugs cannot be predicted from the requirements without carrying out a complete analysis of the system. For example, if we “translate” the requirements into test cases we will be unable to find problems resulting from the influence of activity in one module on function in another module, nor will we be able to test the system in real-life conditions that are not mentioned in the requirements. Also, any missing element in the requirements document will get embedded in the product without giving testers a chance to find it.
One method of ensuring coverage of the system under test is by using Session-Based Test Management (SBTM). SBTM divides the entire test into sessions. Each session has a Test Mission and a charter that together define the boundaries of the test (what should be tested). Using SBTM can help test-teams ensure coverage of testing while, at the same time, providing testers with the freedom of thinking that is necessary to ensuring full coverage of the system. More information about Session-Based Testing can be found in James Bach’s website at www.satisfice.com.
An ET practice that provides improved coverage is “state tables”. This practice is based on linking the session logs to the requirements, thereby creating transparency charts that indicate in what session each test case was tested. It’s important to mention that testing all the requirement test cases doesn’t prove that the system has been checked; however, testing all the requirements using the ET approach does increase confidence that most of the system’s bugs were found.
There is no such thing as an “ET tester”.
ET doesn’t really change the main activities involved in testing. One way of looking at it is that ET only changes the timing of the test activities; instead of learning the product, designing the tests, and then executing them, all activities are performed simultaneously.
Most people that are capable of planning tests are also capable of executing Exploratory Testing. However, if you hire a tester that has only execution skills (“monkey tester”), he won’t be capable of working according to the ET approach. But then, at least in our experience, he won’t be capable of any other testing either, regardless of the approach being used…
Exploratory testing is like chess; learning how to play it takes very little time, but learning how to play it well is a much more significant proposition. The exploratory tester needs a large arsenal of testing techniques in order to perform good and efficient exploratory tests.
Reporting is an example of an important skill required for ET. In SBTM, the tester is required to provide a test report that includes notes of what he was testing, his ideas, and the mission behind each test. Even experienced testers lack reporting skills and find that “taking notes” interferes with their thinking. Testers must also practice how they take notes. Many testers tend to telegraphically summarize their testing like this: “I tested the system”. Others tend to be too detailed (Click OK, type John in the Name field etc.). The notes should not only cover what you have done but also provide details of what you had in mind, and what were you trying to achieve in each test.
Another skill applied in ET and requiring training is Focusing and De-focusing. When testing a system, testers might focus on a specific module they are testing and fail to see the overall picture. Part of the training required for ET is developing the ability to focus and then de-focus. This is very easy stuff to explain but achieving this capability requires training and a lot of practice.
One of the main arguments against ET is that it does not allow management to control what is being tested. Well, actually, we believe this is the main argument against Scripted Testing.
Scripted Testing usually produces a mass of test documents sometimes amounting to thousands of pages or more. In most cases these documents are not inspected thoroughly, and only sample tests get reviewed.
In ET, at the end of each session, a Test Report is produced containing a charter of the test conditions, area tested, detailed notes on how testing was conducted, a list of any bugs found, a list of issues (open questions, product or project concerns), any files the tester used or created to support their testing, percentage of the session spent on the charter vs. investigating new opportunities, percentage of the session spent on creating and executing tests, bug investigation /reporting, session setup or other non-testing activities, and session start-time and duration. Although the list of items included in the test report seems long, it usually only amounts to a few pages that are much easier to inspect and control and thus, provide better visibility and transparency on what is being tested.
In one of the organization we worked in, we defined a Test Lead position whose main task was to review the Test Reports. At the end of each session, the test lead formally reviewed and analyzed the test report, to make sure the testers had done testing properly and had covered all the necessary scenarios. The test lead used the notes taken by the testers to make sure they had understood the system under test and had been able to think of ideas of where it might fail.
In practice, this was the first time the tested content had gone through a real review …
Screen recording tools, such as Spector Pro, are very useful in ET Reporting. The screen recording tool records each operation done by testers by capturing a bitmap image of the screen every couple of seconds. This reduces the reporting load, leaving testers to focus their effort on coming up with new ideas for tests.
Before comparing the resources required for Exploratory Testing to those needed for Scripted Testing, we must answer the question “how much time will we need to test the system”. Well, there is no real answer to that question, but before we try to answer it, we will have to answer a different one: what is the quality level we are seeking? If the answer is a bug-free system, then the answer to our first question is, an indefinite amount of resources. Every system and organization has different quality expectations and these are reflected in the amount of resources allocated to QA. In addition to the company QA standards, resource allocation also depends on the system’s complexity, the number of builds we will have to test, the quality of the code, etc.
So, the main issue is not how much time it takes to test, but how efficient our testing will be. This can be evaluated by counting the number of defects found in an hour of testing and setting their level of severity.
The efficiency of Exploratory Testing done right is much higher than that of pre-documented tests. There are many reasons for this, and here are few of them:
Many people estimate their test efforts by counting test cases and test steps, and monitor testing progress by calculating how many test cases have been executed. Counting test cases is like testing the performance of a company by the amount of files stored on its servers… The question is what is in a test case and how much time is needed to execute it. A certain test case can take five minutes to execute in one iteration and half a day during another iteration, depending on the amount of bugs found and the ideas and risk factors discovered by the tester.
In SBTM (Session Based Test Management) the test effort is measured in blocks of uninterrupted testing time. A normal session takes 90 minutes during which the tester refers to a charter that details what the test is trying to accomplish. The metrics in SBTM break each session into three discrete parts that do not overlap: Setup, Test Design and Execution, and Bug Investigation and Reporting. This method can provide the management with better evaluations and metrics and can be used to improve planning and evaluation in the future.
There is no such thing as 100% scripted testing. Even in cases where testing is limited to creating test cases and then executing them (and them only), exploratory testing is being applied. Not at execution level (unless bugs are found in which case bug investigation requires exploration by the testers), but for planning and designing the test cases.
Our experience shows that even in organizations that only employ scripted testing (or at least that’s what they believe, since purely scripted testing does not exist in reality), when testers are asked “how many of your bugs were discovered by executing the scripts?” the answer is never 100%. In fact, most bugs are usually found during the testers’ “free time”, when they can carry out additional tests that are not pre-defined.
It would be too arrogant to say that no Scripted Testing is ever needed. Scripted Testing can be very useful as a guideline and for assuring coverage by the testers, as long as their freedom and responsibility are preserved. In one of my previous projects, I blended Scripted Testing and Exploratory Testing by allocating the first 15 minutes of a session to executing scripted testing that covered the basic (sanity) functionality while the rest of the session was dedicated to executing exploratory testing.
The intelligent testing manager must find the correct balance between ET and ST that will lead to the best possible outcome for his project.
The problem with executing pre-defined scripts is that it assumes that we already know where the bugs might be hiding in the software before we have started testing. This is actually called Checking; real testing means investigating product behavior and defining risk zones based on data-analysis and information collected while exploring the product. If we want to investigate well, we cannot assume that the most critical risk has already been identified.
In addition, pre-documented testing causes inattentional blindness, resulting from our being so caught up in reading and understanding the test scripts, that we end up missing obvious bugs that are staring us in the face.
The more complex the system is, the more test-cases can be created to test it. This is the reason that for complex systems a certain degree of risk analysis is performed when deciding which test cases will be performed and which will be ignored However, once the tests have been selected, they are the only ones that will be executed. As a result, defects residing within the sphere of the rejected tests may be missed.
One of the main advantages of using ET is that it gives the tester the freedom to select different scenarios in each test cycle. This leads to better coverage of the system; more defects can be found, resulting in higher quality of the system under test. So the bottom line is: the more complex a system is, the more suitable it is for ET!
