Chatbot testing

What is Chatbot?

Chatbot is a program for sending automatic responses to users' questions. Replies are close to the real-life dialogs and correspond to the essence of the question. Chatbots like this are used for giving feedback and performing different functions for the customer on websites, social networks, messengers, call centers, etc. Sometimes they also play the role of personal assistants (for example, Siri, Cortana and Alice).

Chatbots can be built on the base of artificial intelligence using machine learning (improving its understanding of human language in the communication process) or on programmed scenarios with the possibility to choose different options for responses or questions (algorithm).

How smart a chatbot is, depends on how it was designed. Chatbots with machine learning handle tasks better, all because they can understand human language. It learns to communicate with the user, gains experience and can solve new problems similar to the previous ones. In comparison to chatbots on programmed scenarios, the customer does not have to write specific phrases to receive a relevant reply.

The popularity of chatbots is caused by their economic profitability, the possibility to provide feedback 24/7 and the effectiveness of communication with customers, in particular, due to the possibility to provide quality service in two languages.

How to test chatbots effectively:

1. Check the communication process with a chatbot. Start a conversation with the chatbot and go through different communication scenarios, from the most used or important phrases to exceptional situations.

2. Use specific terms to verify the provision of accurate information about the product or service.

3. Engage developers in chatbot testing. They will test the chatbot questions and answers at the next build. Checks like this will be considered as verification and validation, for example, whether the chatbot provides precise and actual answers to hypothetical user's questions.

4. Check situations when the user enters a meaningless sentence or rarely used expressions. Of course, it is impossible to predict all of the irrelevant and invalid information that users might enter. In other words, all exceptions the chatbot may deal with are unreal to check. Yet developers must come up with «emergency answers» for those exceptions that can be foreseen. This is the testing process phase when checking whether the chatbot gives clear answers in such «exceptional» scenarios is required.

There is also a list of words that a chatbot must recognize so that users do not consider it absurd. For example, there are basic words such as: hello (and all the analogues), new, start, start again, menu, such specific words as buy, order, assortment, services, delivery, payment, conditions, etc.

5. Use tools for chatbot testing, for example, Chatbottest, Botanalytics, Dimon.

Chatbottest  is an open-source guide containing 120 questions to evaluate the user experience provided by a chatbot.

It works on three levels:

• possible scenarios of testing chatbot;
• expected scenarios;
• almost impossible scenarios.

It gives 7 metrics to access chatbot performance:

• Understanding. Does the chatbot recognize user messages – likes, small talks, idioms, emoticons?
• Responses. Are the answers relevant and accurate enough?
• Navigation. Is the conversation intuitively understandable enough?
• Personality. Does the chatbot's answer tone match the user audience and the conversation character?
• User-friendliness. Do users understand how the chatbot functions during the first use? Is it intuitive enough how to interact with the chatbot? Each chatbot response should contain a call to the next step: the need to write something or a button to advance the conversation further?
• Intelligence. Does the chatbot remember certain details, key information provided by a user throughout the conversation?
• Error management. How does the chatbot handle errors and exceptions?

Botanalytics is a custom service that allows testing every key aspect of the chatbot from usability to conversation. 

Dimon, the distinguishing feature of which is a chatbot testing tool is that it is easily integrated with basic platforms such as Telegram, Slack, WeChat, Facebook and Messenger. It can be used to identify any problems in the chatbot dialog.

6. Automate the chatbot testing. You can use a chatbot which will interact with the chatbot tested without the involvement of a tester. This way it is possible to start recording conversations automatically. The tester's work will be reduced to one key task: evaluating and analysing the chatbot's responses.

7. Make the chatbot as close to a real person as possible. The chatbot can have everything to make communication more comfortable and real: name, photo or logo.

8. Check the pause interval between messages. Users should have enough time to read the message before receiving another one.

9. Check the work of the buttons and quick replies. In order to facilitate the user's responses, a chatbot can send options for pre-defined responses. Buttons that are needed for navigation can be used to get a reply from the chatbot.

Quick replies usually disappear after clicking them, however the buttons on the contrary shouldn't disappear. Predefined responses used for refusal or objection should be polite and tactful so users do not feel uncomfortable using them.

10. Check that after the end of the conversation with the chatbot there is a possibility to come back to it again. There must be a response that will pull the chatbot back into the conversation as well as the starting message from the chatbot, opening which users understand what to do next.

11. Check for the presence of deadlocks. Users must understand what every testing stage means and what to do next.

Chatbot testing is an ongoing process. Any chatbot checks before the release are beta testing no matter whether the tester uses manual or automated testing. Online feedback provided by users can become a true challenge. You need to be prepared to always adjust the chatbot according to reviews after its release.