Pairwise tools. Creating test plans by using test design techniques

Today there is a lot of information regarding pairwise testing. This technique is known to be one of the most powerful testing methods according to the principles of resource-saving. Pairwise testing is grounded on the principles of combinatorics and is based on the «test each of the input parameters at least once» method. 

It is known that the most valuable resource is time, so the pairwise testing method allows you to cover the largest number of data values with the minimum number of checks. The most common errors in a program are usually caused by a single input parameter or by the interaction between pairs of parameters. Errors related to interactions between three or more parameters are less common and such testing is resource-consuming, cumbersome and, of course, more expensive. Therefore, the combinatorial methodology of test case selection is a good compromise between the cost and benefits, which allows to significantly reduce the number of test cases without drastic deterioration of functional coverage.

Currently, there are many desktop and online means for pairwise testing. They can help to significantly facilitate the process of preparing the test table and organizing test data and the testing process in general. Let's consider the most common ones (the full list of available tools can be found here https://www.pairwise.org/tools.html).

Desktop tools

Allpairs – https://www.satisfice.com/download/allpairs 

Allpairs is an execution file of the command line based on the Perl scenario.In order to work with the application, you need basic knowledge of console commands: how to navigate between directories, and the ability to run the program from the directory in which the test files are located.

First, it is necessary to create a table of data that will be processed by Allpairs. In Excel, you should make a table with a list of data and parameters, then export this table to a text file with column separation by «TAB» and provide the file for processing to Allpairs.  

In the directory with the allpairs.exe program, create a subdirectory (for example, «Test») in which we store text files (with Excel tables). Find a file with the results of allpairs.exe in this subdirectory. Go to the «Test» subdirectory and from it access allpairs.exe located one level above.

Specify the file that needs to be processed (in the example it is input.txt) and set the command to output the result to a file which does not yet exist on the disk (in the example it is named output.txt). This will result in a ready list of test checks with the specified parameters and their values.

PICT (Pairwise Independent Combinatorial Tool) – https://github.com/microsoft/pict

The application is similar to the previous one, as it also runs from the command line. The program takes as input a simple text file with parameters and their values, and as an output, it provides the generated test scripts.

Online tools

Pairwiser – https://inductive.no/pairwiser-tool/ 

The Pairwiser online service allows downloading or creating a test plan, defining the parameters of the system to be tested, and generating a list of tests and test scripts. The service is free but requires registration.

To generate tests, you must first define the parameters of the tested system. The parameters are specified on the «Define Parameters» tab.

Constraints should be used if there is a need to exclude any combination from the test plan. If there are mandatory tests or parameter settings that must be included in the test suite, they can be specified on the «Required Tests» tab.

Test variations are created on the «Generate Tests» tab. You can generate 1-wise, 2-wise, and 3-wise tests (tests with 1/2/3 parameters) and mixed tests.  It is also possible to include pre-decided required checks in the list. If the goal is to create a checklist, then this can be the last step in this tool. 

Select the required degree of test thoroughness and create the tests – as a result, you will get a table with combinations of parameters that exclude the constraints specified in the first tab. The results from this tab can be exported to Excel or Jira.

After the parameters of the tested system are defined and tests are created, you can write test scripts with the necessary parameters: preconditions, steps, expected results and postconditions by using the «Test Script Template» tab. If there is already a template, test scripts can be created by using the «Test Scripts» tab.

PairwiseTool – https://pairwise.teremokgames.com/

One of the simplest online systems for pairwise testing. The main actions in it include the following:

1. In order to apply pairwise testing to the tested system (SUT), one needs to create a list of input variables. In the first example, it looks this way:

2. Click the «Generate Pairwise»  button to see the number of combinations that must be tested.
3. Click «Generate All Combinations» in order to get all possible combinations.

As a result, in a few clicks, we receive a ready list of test checks according to the specified input parameters.

Pairwise Pict Online – https://pairwise.yuuniworks.com/ 

The free online variation of Microsoft Pict is quite easy to use and does not require registration. Specify the list of parameters and their test values, and, if required, the prerequisites for test variations in the field on the left. Then click on the «Generate» button and in the right field you will get a table with a list of checks. This table, as well as the list of input parameters and values, can be downloaded to the device as a txt file.

CTWedge: Combinatorial Testing Web-based Editor and Generator – https://foselab.unibg.it/ctwedge/

The free service for creating a list of test checks. It has ready-made examples that you can edit by substituting your own data. Moreover, you can use different algorithms for generating. Calculation of the entered parameters is done by clicking the «Generate Test Suite» button. An Excel file with the generated list of combinations will be automatically downloaded after clicking the «Download CSV» button. There is also a possibility to select characters that will separate the data in the table (comma, semicolon, slash or space).

Pairwise testing is a powerful but easy-to-use test development technique that helps the testers significantly reduce the number of tests while maintaining reasonable coverage and frequency of detecting problems. For those who want to deepen their knowledge in this topic, there are many studies on the minor differences between pairwise testing, allpairs algorithm, the orthogonal array strategy, their various implementations and optimal use. Pairwise testing may lack clear scientific evidence of its effectiveness, but it is widespread and recognized among the testing community for one simple reason: it actually works.