Performance testing in games. Concept of the Frame rate

The concept of performance 

Performance is the computer's speed of performing tasks.

We can highlight two main tasks, which should be performed by a computer, console, or mobile phone during the game:

• rendering and maintenance of the game world where the user is;
• reaction to the player's actions.

The performance factor is significant in games; if the productivity rate is low, a computer cannot process all tasks quickly and on time. As a result, we will get freezes, lagging, slowdown, etc.

On the other hand, if the performance is high, the game runs smoothly, and nothing prevents our immersion in it. But, when the device performance is not good enough for the game's stability, we can reduce its load by lowering the graphics quality.

The performance itself is a collective value as it depends on the following factors:

Processor:

• Processor (CPU). The higher the processor's clock rate and the more parallel working cores, the better the speed of operation.

• Graphics processor (GPU). Graphics cards with higher characteristics allow processing and displaying the image faster, which improves computer performance.

• Random-access memory (RAM). RAM stores executable program code and some processor data during device operation. More RAM stores more data and increases performance.

Types of productivity defects

Freeze is a temporary computer freezing in the game.

There are two types of freezing:

• The system freezes all the time. As a result, the image is static, and the game stops reacting to the player's actions. After the freezing, the game starts from the moment it stops working. This defect is typical for one-user or single-player games. 
• Freeze of the specific player. It resembles the previous defect in all aspects, but after this freeze, a game is renewed later than it stops working. That is, only a particular player freezes, and the game world continues its activity. This defect is more typical for online games, where other players keep exchanging data with the server while the freezing player cannot do it.

Сrash is an emergency game completion with switching to the main home screen or reloading the device due to the defects in the code. 

FPS (Frames per Second) or Frame rate is the number of frames per unit time in computer games, television, and cinema. In computers, it is the frequency with which the processor updates the image in the framebuffer.

We can turn on the FPS display in the game settings, and if no option is provided, we can use additional software such as Fraps.

This defect is due to a temporary (for example, the explosion animation) or permanent low level of frame changes. The minimum valid FPS value in games is 30 frames per second. 

A sharp decline in this indicator below 30 frames shows a defect and a signal that we need to optimize the code in this place.

Memory leak is the reduced amount of the device's free memory. It is due to errors in the program code when allocated RAM or virtual memory is not cleared after using it. 

After a certain period of non stop operation, the amount of uncleared memory could exceed the available one in the device. In this case, the device might freeze, or the application might crash without saving the data.

All the abovementioned defects negatively affect the player's gaming experience. Furthermore, excess of such defects could lead to a player wanting a refund for the game and giving a bad review.

In order to protect the user from a negative experience, we run performance testing.

How can we test performance?

Testing without tools. It is the most unreliable way of performance testing. Of course, we can find freezes and crashes in games even without special tools, but low FPC bugs are solely noticed when things are bad, and only, in that case, it is possible to see the low FPC with our own eyes and catch it on video. We might not notice any memory leaks and miss them during testing.

Testing with tools. This type of testing is divided into 2 stages: using testing tools and analyzing received data. During the analysis, the tester pays attention to the following parameters:

• Frames per Second, FPS;
• Battery consumption. At times of high loads, a phone starts consuming more energy. At high long-term loads, the device temperature can rise and reach the critical values, in which further device operation can lead to a data loss or its complete failure.
• Memory consumption. It is tracked to detect a memory leak in applications.
• CPU and GPU loads. Observing what is going on in the app and the CPU and GPU charts, we can conclude what processes and functions consume many resources and need optimization.

Based on this data, the tester can locate potentially problematic places, which should be double-checked, and places that might lead to a crash during long-term work.

Therefore, GameBench is an ideal program for this testing on Android and IOS. It allows recording the screen in parallel with charts, and it helps to understand what happens on the screen at any load moment.

Testing with an FPS counter. Crashes and freezes are detected in the same way as when we test without tools, but here the FPS counter is used to monitor in which places FPS drops below the allowed level.

The FRAPS program is perfect for tracking FPS on a PC.

It allows tracking FPS in real-time and recording data in the file for further analysis. In addition, we can make screenshots and record videos with this program.

In the following screenshot in the upper left corner, we see the FPS counter. However, we can move it to another place for convenience.

Performance testing is crucial at the time of a game's release. Crashes because of poor performance makes a lasting negative impression on the end-user. For this reason, QA engineers focus on performance testing to check if specific indicators are okay for all devices, and importantly, end-users are satisfied with the game without freezes or crashes.