Everything seems to be well organized in Agile testing processes. The process of testing is under complete control. It begins with the design, unit, and integration testing performed typically by developers and continuing with automated and manual functional or non-functional testing. In the end, there is user acceptance testing and exploratory testing. All of these techniques are managed, mapped, evaluated, and described. Covered by the process. We can say: the process is perfect. Perfect in theory, indeed. But what about the real world? What about the real life of the application in the bad, bad, unexpected, chaotic world out there?
● Exploratory Testing (and Its Limits)
In fact, only user acceptance testing and exploratory testing are techniques that cover and evaluate an application from the user's perspective. Other testing techniques concentrate on the functionality of the application itself, primarily. User acceptance tests may be a little more problematic when you are thinking of some kind of application that has a million or more customers, and you are trying to simulate an average customer experience. Exploratory testing is more experimental and more valuable from the user's perspective because it works with a wide variety of possibilities and scenarios.
But in Agile methodology, even exploratory testing becomes problematic too, mainly because of the methodological frame it is locked in. There is the whole process of preparation, planning, context, and even certain sections of the product or paths for testing are set before the execution of exploratory testing.
I think that we must ask ourselves the question: what exploration remains in exploratory testing in Agile development, when all these limitations, prerequisites, frames and limits, even expectations are set? The main benefit of the approach is that it can cover scenarios that were not covered by more traditional testing approaches listed above. One positive of exploratory testing is also the fact that when testing in pairs (e.g. developer + tester, or tester + tester) more and more areas of possibilities are covered. But there is still something missing anyway.
● Let's Introduce a Little Anarchy
The set (also mindset), the operational space, the frame of agile testing is still not enough spacious to cover the set (also mindset), the operational space, the frame of the users' world(s). That is the problem of any methodology. How to fully and systematically cover something that transcend anything systematic? How to, as much as possible, reproduce the real scenarios that may occured?
I suggest that we need something more, something beyond the methodology and beyond methodological testing process to achive the goal. We need to introduce little anarchy to the perfectly covered testing process, and this something will help us fulfill the goal: a perfect software accepted by our customers.
● Ad-Hoc Testing
Ad-hoc testing is a good candidate. It is chaotic, it is beyond any rules, it has no goal, it has no target. The only rule for ad-hoc testing is that of space and time. We need only some environment and some time, plus some application to test, to perform ad-hoc testing. We do not need an experienced tester, we do not need a plan, nor any scenario; in fact, there must not be any scenario, any plan, any model, or any expectation. Otherwise, there is no ad-hoc testing possible.
These are the main differences between exploratory testing (in Agile methodology) and ad-hoc testing:
Exploratory testing needs (more or less) experienced testers, ad-hoc testing can be performed by anybody, similar to the expected user of the application. If the application under the test will be available for all users on the market, if the application is not limited to a specified type of user (domain administrator, security expert, financial expert, etc.), the variety of testers is almost endless. You can use non-testers for performing these tests, administrative workers, managers, students, volunteers from different specializations, even employee's family members. Because it is application created for them, the application created for everyone, it is an application for the masses, so try to test it with as many possible types and kinds of users as possible. Let the newbie tester in your team test the application on their very first day, without any training, without any education, and record the testing. Let the testers from different departments within your company's portfolio exchange the applications for testing...
Exploratory testing needs a specified target, a specified section of the application under the test for testing, a context in which the testing will be performed; ad-hoc testing is just a chaotic event. But remember: there is always some kind of order, even in the most chaotic chaos. Just ask quantum physicists. So sit down and play with the application. Do not think much. Try to not think as a tester at all. Try to be as intuitive as you can. There are no limitations for you as the tester. There will be none for the users. Try to be spontaneous. Do not concentrate the testing only on the application itself, but also test the environment it is running on. How does it operate with the operating system? What happened when a similar application was started? What happened when other applications were started? What happened when you acted unexpectedly? Do something you had never thought of before.
● The Cons of Ad-Hoc Testing
Exploratory testing is described in detail. The notes are taken during testing. Everything is archived. It is very important because when an error or failure occurs, the incident is traced and all the information saved will be used for a quick solution to the problem. It is also very important because it makes your manager(s) are happy. They can see it and they can control it. They know how much time it costs, and how much money it costs. Colorful graphs may be created and presented. That is a little problem with ad-hoc testing. Writing notes may be helpful, but may also be a big problem here, because the intuitive and spontaneous flow of actions is broken. Any pause, any concentration on the testing process itself is not welcome here. The solution may be a very good memory of a tester, or another tester just sitting next to the ad-hoc tester on the run, or - and I think this is the best solution available - the whole process of ad-hoc testing is being recorded by some kind of screen recorder and/or keylogger. With such a tools there will be no problem in letting the ad-hoc tester do their job and fully concentrate on it.
Also, the purpose of ad-hoc testing must not be to find as many bugs, or errors, or failures as possible. This kind of testing, when recorded and analyzed afterward, may be helpful also for further development of the application: make the functionality of the application more user-friendly, make several design changes, move some functionality into a different section of the product, etc. And we must also remind ourselves of some psychological benefits of this non-traditional kind of testing. It can be used as kind of "mental hygiene" for testers. After all the systematic approaches, after all the strict and systematic tests and scenarios, ad-hoc testing may help testers to unleash, to relax, to be not-so-systematic for a moment. It can be a kind of relaxation, a vacation, and also a kind of game for them. Imagine, for example, a role playing game. The tester will play the role of a customer. Our tester became a busy manager in some big company who just goet a new laptop with our application installed.
Our tester will become an accountant who is working on his computer where our application is installed. Our tester will become a hacker who is trying to break into our application. Our tester will become a bad, bad administrator who will look at our application with a highly critical eye, and who is trying to beat out of it what they can get. Our tester becomes an ordinary young person browsing Google Play store looking for our application... There are no limitations.
● "Ad-Hoc Testing Friday Afternoons"
Imagine that your testers are allowed to create an ad-hoc testing on Friday afternoons. What does it mean? It means that on Friday after the lunch, testers will have an ad-hoc testing session. The recording device is on, their mind is ready, let's start. During this session, they will be allowed to do whatever they want, according to the software and the environment. They will just be experimenting. They will be someone else. They will be in different roles as described in the previous section. They will try whatever comes to their mind.
Various situations, various environments, various operating systems, various different or similar applications, various different users logged into the system, online, offline scenarios, etc. The afternoon is for intuition, imagination, and chaos. The afternoons of real-life. If some unexpected issues occur, these issues will be recorded and analyzed after the session, or on the following Monday. If no issues were found, this is also good: maybe some new possibilities for future testing were discovered, testers did some kind of different actions than expected, some mental hygiene, something non-rational and non-systematic happened.
Ad-hoc testing is not exploratory testing. These two are quite different, and I think, both needed approaches in the Agile testing process. Ad-hoc testing is a kind of transcendence beyond the methodology, beyond the reason, and beyond the planned activities that exploratory testing can never cover. With ad-hoc testing, the imitation of the real world usage of the application with its non-rational, non-methodical, non-predictive scenarios may be covered during the testing process. This approach may help to create even more reliable, usable software, and may be also be beneficial for the testers.