Thinking traps in testing

In this article, we will discuss thinking traps, or, in scientific terms, cognitive biases. 

There is a direct connection between a person's perception and their way of thinking, as well as the judgment of other people and situations. Someone’s thoughts and thinking are not always objective, which can lead to systematic errors and deviation. This deviation from what is seen as reality are cognitive biases or thinking traps. 

Wikipedia provides the following definition: 

Cognitive bias (or cognitive distortion) is a systematic pattern of deviation from norm or rationality in judgment that may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, and irrationality about other people and situations. In other words, thinking traps refer to standard, stereotypical deviation in human thinking based on distorted beliefs.

Living in their own reality based on the perception of the surrounding reality, people acquire certain behaviors and relationships with others. Often, illogical behavior, inaccurate judgment, and illogical interpretation are the result of these thinking traps.

Software is created by people, tested by people, and typically used by people. Each person in this chain is susceptible to their own cognitive distortion.

Distortion affects all testing, both manual and automated. To achieve effective test case creation, execution, and defect finding, it is necessary to have an understanding and knowledge of one's own distortion and its consequences.

We are going to tell you about the cognitive biases that are most common and have the greatest impact on the work of software testers, as well as methods to mitigate the negative effects of their manifestations.

Based on their nature, the following groups of distortion can be distinguished:

• template thinking;
• error/conflict (cognitive dissonance, cognitive illusion, fallacy of conjunction, illusory correlation);
• subjectivity (confirmation bias);
• negativity;
• elevated expectations;
• emotionality.

Template Thinking

The concept of a template can be defined as the accumulated life experience of many generations. A person who uses another person's experience and learns from their mistakes will know the result of their action before they take it. 

Template thinking is a characteristic of a behavioral style and is often an indicator of low effectiveness. 

The negative impact of such behavior is that a person tries to fit reality into frames, ineffective behavioral models, and stereotypes. 

Anyone who thinks in templates attempts to solve problems and achieve goals using only a small set of techniques and methods that are familiar to them. They may struggle to come up with new problem-solving methods and break free from stereotypes, thereby limiting their own potential.

In testing, invented dependencies hinder the reproduction of a bug. This is especially dangerous in exploratory testing, where attempts to validate a template divert attention from the main task.

Testing is a creative process, so it is crucial to eliminate templates and take a broader view of the process.

Cognitive Dissonance

Cognitive dissonance is a state of psychological discomfort caused by the clash of conflicting beliefs, ideas, values, or emotional reactions in a person's consciousness. Human beings are naturally lazy creatures who tend to avoid internal conflicts. Therefore, it is easier for them to adjust their thinking rather than delve into the causes of the conflict.

For example, during testing, cognitive dissonance can occur when a tester receives several conflicting answers to their question. It is very easy to choose the answer from the most authoritative source. It is more challenging to invest time and effort into thorough investigation. The danger is that if there are contradictions in a certain area, it is likely that there will be bugs there.

Testers need to recognize when they are in a state of cognitive dissonance and learn to ask questions before making decisions.

Cognitive Illusion

The simplest example of cognitive illusion is a visual illusion.

This common trap can explain why intelligent people sometimes say foolish things: their brains may see something different from what is actually happening. A person, when looking at an object, sees it as one thing, while in reality, it is something entirely different. To avoid this trap, a tester can try to look at the situation from a different angle. However, this is difficult and requires experience and time.

Conjunction Fallacy 

The simultaneous occurrence of two independent events cannot be more probable than the probability of each event occurring separately. People often forget this and attribute a higher possibility of the combined events, thus incorrectly linking the number of events to the likelihood.

In testing, this can affect the interpretation of documentation. When creating test cases, it is important to understand how often a specific scenario will be used and how critical it is. 

To avoid the risk of the conjunction fallacy, discussing details in advance and reviewing statistics on the use of specific functionality can be helpful.

Illusory Correlation

Illusory correlation is the perception of a stronger connection between variables than there actually is or a connection that is significantly weaker than expected. 

People often make illusory correlations in various areas of life. This can be seen in the life stories of Bill Gates and Mark Zuckerberg, who dropped out of education and started billion-dollar businesses. We remember these cases and use them as examples. On the other hand, we don't hear stories about students who did not study and did not create world-famous companies. In the flow of information, people pay attention to extraordinary cases and overlook others. 

The most common example of illusory correlation in testing is including unnecessary steps in a bug report. 

Reducing the likelihood of falling into this trap can be achieved through pair testing and involving others in test planning.

Confirmation Bias 

One of the most dangerous cognitive biases is confirmation bias. It is the tendency for a person to seek, interpret, and focus on facts that confirm their preconceived beliefs while ignoring data that suggests the opposite.

To overcome this distortion, it is important to draw conclusions based on real statistics rather than just relying on information that confirms one's subjective bias. 

A quick and simple way to reduce the risk of confirmation bias in testing is to involve another person in the testing, such as testing the same functionality. There is a high probability that the second person will have their own confirmation bias, but it is likely to differ from the bias of the first tester.

Negativity 

The human brain is wired to pay more attention to negativity in the short term than positivity. By focusing on the negative, a person anticipates its manifestation. 

In testing, this distortion can manifest in two ways:

• fear of change – it leads to saying «no» to any changes. This fear has both positive and negative aspects: on one hand, it forces one to think about arguments and reconsider the necessity of the change; on the other hand, it can hinder overall progress;
• distrust of developers – on one hand, a tester's job is to «trust but verify.» On the other hand, excessive verification consumes time that may not be available. For example, a lot of testers find it challenging to complete testing tasks if no defects are found, even minor ones.

Elevated Expectations 

High and elevated expectations are the flip side of negativity. 

For example, when buying a more expensive item, a person expects its quality to be better than that of a cheaper alternative. 

In testing, this can manifest as more thorough testing of a task performed by a junior developer compared to a task done by a more experienced developer. However, in general, more experienced developers work on more complex tasks, and the likelihood of errors arising is approximately the same.

Emotions 

Emotions are not a cognitive distortion, but they also influence a person's perception. Depending on their emotional state, a person can see the same thing in two different ways – positively and negatively. Moreover, emotions affect focus and concentration. A happy person experiencing joy and an emotional uplift will be less inclined to delve into a complex and tedious algorithm. Usually, developers test their code while writing it or shortly after. In this case, the code writing and testing happen in the same emotional state, leading to similar blind spots. Therefore, it is beneficial when code is tested by someone who did not write it.

It is crucial to maintain a balance between optimism and pessimism. Setting a bar too high can have equally dangerous consequences.

In conclusion, communication, discussion, and critical thinking skills can help mitigate the impact of cognitive biases. It is important to be cautious of assumptions, beliefs, and categorical statements, both one's own and those of team members.