Pesticide paradox and maintaining test case effectiveness

What is the pesticide paradox?

On most projects, there are situations in which the more we test a product, the more immune it becomes to the bugs we try to find using our test suites. After the same tests are repeated over and over again, you eventually stop finding new bugs. In such a situation, the functionality, even though described in detail in test cases, may not be tested thoroughly enough and users may find serious bugs.

Just like insects, the software bugs under test can develop resistance to the same pesticide, in other words, form immunity to existing test cases. This phenomenon is known as the pesticide effect. The term was coined about 30 years ago by American engineer Boris Baser. The essence of this effect is that using the same methods repeatedly becomes ineffective over time, as the pests that survive become immune. 

Why do test cases lose their relevance over time?

Of course, the simplest reasons can be situations where changes are made to the project, new functionality is added, or the old one is optimized. In this case, test cases need to be updated even without taking into account the pesticide effect.  But even if the project doesn't change much, it is still necessary to update the test suite to find new bugs, which are no doubt present in any product.

There are several reasons why the same set of test cases stops helping find bugs. 

First, it is simply impossible to initially predict all available scenarios for testing. Even if the software under test is quite simple, it is still unrealistic to write an exhaustive number of tests that will find all the bugs.  Not to mention complex systems with a huge set of incoming data, dependencies, and scenarios. That is why any new ideas on how to test a product should be checked and added to the tester's arsenal.

Second, the tester gets used to the project over time and becomes familiar with all the problem areas of the tested software, and there is a risk that they might miss defects or even mistake the defect for a previously agreed functionality feature.  When a tester has been working on a project for six months or more and needs to repeatedly perform regression testing and go through the same set of tests, it's not an easy task to keep a fresh perspective on the product.

How to maintain test cases?

It is very important to use a set of different approaches and techniques during testing to look at the software from different angles: by looking at the project structure, its functions, and the data it processes. It is also essential to understand how users will interact with the software and what possible problems they may have.  After all, you can write hundreds of test cases and still miss an important bug. There are many techniques that help testers thoroughly examine software from different angles. 

For example, there are a variety of mnemonics that help testers expand the number and effectiveness of test designs. One of the most popular mnemonics is SFDPOT (San Francisco Depot), created by James Bach. Each letter in SFDPOT represents an aspect of the product that needs to be tested:

• Structure – identifies the components of the software.
• Function – what the software does, its functions.
• Data – what kind of data it processes.
• Platform – on which platforms the software is supported.
• Operations – how the software will be used.
• Time – how the software will behave over time. 

Another mnemonic, I SLICED UP FUN, created by Jonathan Kohl for mobile app testing, reminds us of what needs to be tested: 

• Input (possible ways of transferring information to the application);
• Store (app store requirements);
• Location (user location);
• Interactions/interruptions;
• Communication;
• Ergonomics;
• Data;
• Usability;
• Platform;
• Function;
• User scenarios;
• Network.

There are many other mnemonics, for example:

RCRCRC (Recent – Core – Risky – Configuration sensitive – Repaired – Chronic) – for helping with regression testing.

COP FLUNG GUN (Communication – Orientation – Platform – Function – Location – User Scenarios – Network – Gestures – Guidelines – Updates – Notifications) – for testing mobile apps.

Seeing the product from different angles can result in many good test cases. Using mnemonics helps reduce the risk of forgetting to pay attention to an important part of the software, prevents the chaotic generation of test ideas, and allows the development of a more structured approach to testing and/or test design.  After completing the mnemonic checklist, the tester will know that the main aspects of the product have been tested and nothing has been missed. Needless to say, mnemonics are not an exhaustive test plan, but a useful tool for generating ideas. 

If you want to avoid a situation where the existing test suite does not correlate with the current state of the software and is no longer relevant, you need to update it with tests every time the software changes and is updated with new functionality.  

Here are some tips to help keep your test cases up to date:

1. Edit existing test cases when the software changes.
   Consider that you may need to edit your test cases in the future when you first design them. Therefore, try to write test cases that can be easily corrected. The future you or your colleague will thank you for this.
2. Move outdated and unnecessary test cases from the current test suite.
   As soon as you see that the functionality tested by a test case has been removed, move the test case to the archive, where it will wait for the possible return in the future. It is not recommended to delete test cases at all, as there is always a possibility that the functionality will be returned at the request of users, managers, or for any other reason.
3. Add new test cases needed to test the new functionality.
   Every time a test team is assigned to test a new feature, it's worth writing at least a basic set of tests that can be expanded later. Of course, there are situations when the task needs to be finished yesterday and the product needs to be tested as soon as possible, but even in this case, you should write down which scenarios are going to be tested, at least in the form of test case names without filling in all its attributes. This will help you to be more thoughtful about testing now, and later when you have more time you can finish the test cases.
4. Create test cases to test scenarios related to bugs found by the test team or end users.
   This will help add interesting cases to the set that were not considered during the test case design process.

By expanding their set of test cases, testers can find new interesting and important bugs, look at the software from different angles, and be assured that the product will be released in a high-quality way. This way, testers will not be bored with the project, as they will not have to run the same tests that will bring fewer and fewer benefits.