Testing mobile devices, one can face the problem when there are simply no devices required by the client to test on. Evidently, the first thing to be done – is to buy a device. But what if there is a big scope of devices to be tested on? Or, for instance, there is an urgency to check something on some precise device or platform, but there’s no device like that available? Each model of the device can differ and has its own peculiarities that need to be considered in the development stage. And this is without mentioning the big differences between the operating systems of mobile devices. There are several variants on how to deal with such a situation and still perform testing: using emulators and simulators of mobile OS or usage of special services (also called «mobile device farms»). These are the subjects of this article.
Emulators, simulators and farming devices for mobile testing
Emulators and Simulators of the mobile devices
Emulators of mobile devices help to test the functionality of the application on different mobile platforms as well as emulate the software and allow it to perform certain checks.
In the case of iOS devices, everything is quite complicated. There are no high quality emulators of this exact platform due to its closed architecture.
However, there are a few ways to emulate Apple devices (iPhone, iPad). First of all, it is an official Apple iOS Simulator that is part of Xcode. It allows us to test different combinations of OS and devices, not only Mac. And that is, at the same time, a great disadvantage. If you don’t have access to the MacBook, it won’t be possible to use such a simulator. Xcode is also needed to install and launch this application.
In case of a need to test Android devices, the situation is much better. There is a wide range of emulators for this platform. Let us look through some examples.
MEmu
MEmu – a free emulator of Android devices for Windows that is specialized in video games. The emulator is compatible with games from the Play Market as well as APK. It provides fast performance speed, access to the computer folders, GPS change, shortcuts from the keyboard buttons to the screen areas, flexible settings and other advanced features.
Remix OS Player
Remix OS Player emulator is based on Remix OS, which is an Android x86 modification developed specifically for computer and laptop launch (the «Start» button and a taskbar are present). The main disadvantage is that this emulator works only on Intel processors.
Nox App Player
Nox App Player demonstrates high levels of performance.
There is a built-in Play Market, where one can download different applications and games as well as launch apps from apk files from the computer. The emulator window can be maximized without losing the performance level. It also has the following options:
- ability to change the screen resolution and orientation in the basic mode ;
- location change;
- downloading and exporting the file (an easy exchange of files between the PC and emulator);
- taking screenshots.
Leapdroid emulator
The advantages of this emulator are speed, ability to work without the hardware visualization. Also, there is a limited number of options to configure, as well the option of changing the screen resolution and configuring any other parameter except those that are present on the Android setting panel.
BlueStacks
BlueStacks – a free program for launching the Android-games on Windows or Mac OS, it has a multi language interface. The program supports a full screen mode, 3D games and also a large amount of applications and has a pack of already installed applications.
Koplayer
Koplayer – is a free emulator that allows to easily launch Android-based games and applications on a computer or laptop with Windows OS. It has a fast level of performance on relatively old systems and has similar settings, including assigning the amount of RAM to the emulator. The program has very convenient keyboard settings for each game separately and for the keys you can assign gestures on the Android screen, accelerometer actions, pressing individual areas of the screen.
Android emulators for the developers
If the main task of all the emulators above is to run Android games and applications on Windows 10, 8 and Windows 7 by ordinary users, then the following are intended primarily for application developers and allow you to debug, support ADB (respectively, connect to Android Studio)
Creating emulators in Android Virtual Device Manager
On the Android applications developers site you can download Android Studio and everything that is needed for Android programming (Android SDK). This set includes the tools for testing as well as for debugging applications on virtual devices. An emulator can be created and launched without even opening the Android Studio.
Genymotion
Genymotion emulator is simple in installing, it allows to emulate a wide range of real devices with different versions of Android OS. It works fast and supports a hardware boost of the graphics.
It emulates the main and front camera and supports screencast function as well as remote control. Among the advantages – ADB emulation.
There is also an integration with popular IDEs (Android Studio, Eclipse) and simulation of incoming calls, SMS, battery discharge and many other features.
Conclusion
At first, it seems that you can safely replace testing on real devices with emulators. However, in practice, it is not so simple.
The main advantages of testing with mobile emulators:
- most of them are free;
- it is much easier to test with the help of them;
- it is easier to find and screenshot a defect;
- it can be used to study the behavior of the program.
However, there are some disadvantages:
- emulator testing may give a false impression of the application;
- since mobile emulators run on a PC, they have a completely different network environment;
- most emulators cannot simulate incoming SMS and calls;
- emulators do not support compatibility testing because they are not connected to a mobile network;
- emulators can not simulate the operation of different battery status of the mobile device, the operation of the camera of the mobile device, etc.
Mobile device farms
The problem with the diversity of the device fleet arose long ago. Back then, Nokia began offering their developers a free smartphone rental service. It was possible to install the application on the device and go through the key scenarios in the remote control mode. Though some devices had to be waited a long time for, such testing gave the developer the opportunity to find out whether his program will work correctly on different smartphones or not. The beginning was good and the device farms received their further development - the OS developers themselves began to provide tools to simulate user actions.
The most popular cloud farm devices are Firebase Test Lab, Samsung Remote Test Lab, AWS Device Farm, Sauce Labs, Xamarin Test Cloud, Perfecto.
Samsung Remote Test Lab
One of the advantages of the Samsung Remote Test Lab is that it provides access to Samsung's new products and you can test applications on devices that have not yet gone on sale. Plus, there's access to Tizen-based devices: the Z smartphone line and the Gear smartwatch.
The process of work with the service looks as follows: the device is reserved and a special Java-application is launched, which provides remote access to the screen and input devices (touchscreen, buttons). Currently there are 25 absolutely free models of smartphones and tablets, each of the models in several copies and modifications. Automation is done at the level of manual recording of the sequence of events, and the application must be installed manually. Samsung Remote Test Lab supports convenient remote debugging mode.
Firebase Test Lab for Android
Firebase Test Lab service for Android is developed by Google. Overall, Firebase is well suited for teams specializing in Android development. The farm has a wide range of devices and their configurations, as well as the ability to run on emulators. Test Lab is included in a single subscription to the Firebase service and can be completely free to get started.
Special farms
Let us consider larger farms that support iOS, have a large fleet of models and are paid.
AWS Device Farm
AWS Device Farm has almost 400 devices (around 100 models), priced from $ 0.17 per minute. They are unlimited and the first 1,000 minutes are free. Among the advantages – high quality service and the ability to integrate into the DevOps-pipeline. The service allows you to test simultaneously on several devices. You can use Appium (iOS + Android), Calabash (iOS + Android), Espresso (Android), Robotium (Android), UI Automation (iOS), XCTest (iOS) and a number of others.
Xamarin Test Cloud
Xamarin Test Cloud offers more than 2,500 real devices. IOS and Android are supported. The following features are provided: screenshots, automated scripts, videos, remote debugging and future recording. The cost of services – from $ 99 per month. The service is ideal for developers of cross-platform solutions (Xamarin, React Native) and projects with a wide user audience. It supports automated scripts based on Calabash and Xamarin.UITest.
Sauce Labs
Sauce Labs has been in the automated testing market for a long time. Jason Huggins is one of the company's founders and developer of Selenium. Sauce Labs is a serious service for adult teams. Prices come from $ 149 per month, there are native and hybrid apps for iOS and Android, as well as the ability to organize a private cloud or conduct manual testing. The support for integration with DevOps pipelines and running on emulators/simulators is provided, but no more than twenty device models have been announced.
Perfecto
Back in the days of Symbian and Windows Mobile, the company offered its devices for rent, now Perfecto provides more than 3,000 real devices. Services are paid, but on trial you can quickly check the application for performance. You can test manually for free, but automation will cost from $ 299 per month.
The main disadvantages of using farms are:
- price: due to low competition, prices remain not very affordable for ordinary users;
- performance: commands are transmitted over the Internet, which can lead to delays in the response of the device to any action, as a consequence – dependence on the Internet connection;
- location: since most farms are located in the US or the EU, testing applications for operators from other countries is problematic.
Conclusions
Judging by the information mentioned above, we can conclude that you can still test the program even without the necessary real devices. But you need to understand that no emulator or farm will give a full experience with the device. However, these approaches have their drawbacks that need to be considered when conducting tests. But in the case when you need to make a quick run of the program for performance on a particular device, such options are quite workable and allow you to perform the necessary tasks. The choice depends solely on the goals and available resources.
