Services for beta testing

Beta testing

Beta testing is the intensive exploitation of an almost finished version of a product in order to identify the maximum number of errors in its operation for their further fix before the final release of the product to the market, i.e. to the mass consumer. Beta testing is a real working version of the program with full functionality. The purpose of beta tests is to evaluate the capabilities and stability of the application from the point of view of its future users.  

Beta testing is performed by interested people, in other words, a group of testers who sign up to test the company's applications and are given access to the application. Access to testing is provided by the company for free, and the company may also give testers some bonuses. Those interested in participating in beta testing are called beta testers.

Beta testers can use the documentation for the tested application. Suppose beta testers find any errors, failures, or crashes. In that case, they notify the company's employees as soon as possible and provide a detailed description of the bug and the actions that led to the defect. This is called feedback between the beta testing group and the company. Beta testing can help to detect new bugs and provide feedback on how the program works in general.

Various beta testing services have been developed to make beta testing more convenient, for example, to find beta testers, get feedback from them, information about bugs and their environment, screenshots and videos of bugs on testers' devices, and other information about the testing process and results, some of which will be briefly discussed in this article.

Test Flight

Test Flight is a service created to simplify application testing on iOS devices. Test Flight makes it easier to collect test device codes (UDIDs) and helps testers access application builds. The service is free.

Service features: 

• providing information about how many times the application was launched;
• providing information about how many times the application has crashed;
• providing information on debugging;
• the function of managing project lists;
• the possibility of tracking the testers' actions (receiving an invitation letter, reading, installing the service, etc;)
• the tester can leave feedback on the work of the test program;
• instructions on how to work with the service.

The disadvantages include the fact that new versions of apps for testing need to be downloaded manually and wait for Apple to review them. Also, there are no notifications about new crash reports and detailed information about the conditions of the crash in the standard support for crash reports from iTunes Connect. The service is supported only on iOS devices.  

TestFairy

TestFairy is an application testing platform developed by the Israeli company TestFairy. The TestFairy service is free. The service is supported on Android and iOS platforms.

One of the advantages and main differences between TestFairy and other services is that there is no need for the developer to integrate an API or SDK into the application. It is enough to upload an APK (Android application file) to the system and it will send out invitations for beta testing.

Service features: 

• collecting key metrics from devices (memory, CPU data, network resource usage, GPS, device signal strength);
• video – the feature provides the ability to video record the test from the client side (memory, CPU, network monitoring, GPS, bug reports, registration). You can get screen recordings or screenshots from other devices to see how the application looks and works;
• support for crash reports. Crash reports are sent automatically and you can view standard iOS NSLog and Android Logcat logs;
• various analytical information and statistics are provided (for example, crash report and platform statistics, download, and session statistics, etc;)
• upload API support (automated download of new versions of the program);
• the possibility of creating a user group;
• testers can leave feedback on the work of the application;
• there is a system for notifying users about the availability of new builds.

HockeyApp

HockeyApp is a service with a full package of beta testing programs that provides information about the behavior of mobile and desktop applications during their beta testing and distribution in the store. The service is supported on Android, iOS, and Windows Phone platforms. The service is paid, with a one-month trial period without restrictions. It is suitable for testing mobile and desktop applications.

Service features: 

• bug reports are sent automatically;
• grouping of similar crash reports;
• the possibility of working together with different access levels, creating user teams;
• the ability to administer API keys;
• the possibility of downloading programs;
• providing analyzed data on the type of device and the time allocated for testing by the tester;
• the ability to leave feedback regarding the program operation and test results;
• the possibility to automate the download of new program versions;
• providing analytical data for each version of the program by type of device, number of installations and crashes, operating system versions, and the amount of time used by the program.

Crashlytics

Crashlytics is a service for collecting and analyzing crash reports. The service is supported on Android and iOS platforms and is free for users.

Service features:

• the possibility to automate the download of new program versions;
• it has a user-friendly interface;
• clear presentation of information. Statistical and analytical data are displayed in the form of charts and graphs. Graphs and charts are updated in real-time;
• flexible system for setting up notifications. Notifications can be set up in such a way that team members always receive relevant information;
• a large database of frequently asked questions for quick and easy search for the necessary information;
• real-time statistics on crashes (the number of crashes, the most frequent crashes, the number of users with crash problems, changes in the status of crashes);
• providing a variety of analytical information (active users per day and month, new users per day, sessions and their length, users without crashes, application versions, active devices per day).

Android Alpha/Beta testing

Android Alpha/Beta testing is a service for publishing Alpha/Beta versions of applications from Google. The service is free and supported on Android platforms.

Service features:

• there is no automation for downloading new versions of programs. You need to manually download the APK file. There is also no procedure for reviewing the application;
• possibility to create closed groups of testers. To do this, Google Groups is used;
• there is no possibility for testers to leave feedback about the program;
• crash and other analytical information is provided through the Google Play Developer Console.

These beta testing services make it easier to find beta testers, control the workflow, communicate with them, provide testers with new versions of the program, monitor test results, and provide analytical information about test results. They are designed to facilitate and optimize the beta testing process and, consequently, improve the quality of the developed product and the quality of the user experience.