Overview of ChatBot Testing

What is chatbot? A software or computer program that simulates human conversation.

Chatbot testing is necessary to make sure that an effective and efficient chatbot is deployed. Testing experts analyze if all the necessary features are implemented correctly or not, and make sure they can appropriately answer each user's queries.

Below Fig1 is the process which shows steps to follow to do testing, step1 says 'Identify the Use cases' which are highlighted in Fig 2, step4 the functionalities are highlighted in fig3.

Tools, Techniques, and Metrics to consider:

Techniques to Consider

1. Test the Chatbot's AI Conversational Flow

Take it as a form of UI/UX testing, where the interface's given by the questions and replies that your bot serves up.

Start the conversation with your chatbot-

Begin with the broad, user-greeting questions and critical use cases (or chatbot testing scenarios), then gradually tackle the edge cases..

• Does my chatbot understand the user's questions?
• Does it respond promptly to them?
• Are its responses accurate enough and relevant?
• Are the conversation steps too many or too less?
• Does it keep the user engaged?

These questions will help you analyze your bot with regard to the flow of conversation, language, tone, and style.

2. Run a domain-specific test

This is that stage of the conversation where your chatbot provides credible information on your products/services.

Imagine that including every single expression specific to your field and every possible question related to your services/product is... mission impossible.

And yet, you can still provide broad classes when setting your context-specific questions for the bot to answer to. Do pay great attention to how you formulate them and to what niche terms you include there. In the same light, you should ask your team members to cumulatively assess each domain well before its launch.

3. Run a developer test

Check whether the chatbot serves up accurate and relevant answers to a hypothetical user's questions.

This one is pretty straightforward and we are sure that your team of developers will certainly run a test or two while creating your chatbot. When the developers are performing a verification and validation test, they can also predefine the bot’s replies and set user inquiries too. So they will simply check if the bot is giving accurate answers to a hypothetical user question or not!

4. Run a Chatbot AI-Error Handling Test

What if a user enters a meaningless sentence or a not so commonly used expression? How would my chatbot respond to that?

It is not possible to accommodate all inquiries that the users may have. However, in a broader sense, your chatbot needs to be ready to seamlessly handle unexpected messages. For instance, if you have an appointment booking chatbot and the customer asks “Can you tell me about the weather today?” then it’s definitely an unrelated question. To such queries, you can perhaps answer with “Sorry but I might not be able to help you with that. However, if you do wish to book an appointment, I can certainly help you out!”. This way you don’t let the customers go unanswered even though they might not be a prospect. Not being able to answer at all can portray your brand in a bad light.

Chatbot Testing Tools to Consider:

1. Chatbottest  ‘ https://chatbottest.com ’

Chatbottest is an open-source guide that has 120 questions that help you assess the user experience that your chatbot provides. It assesses the chatbot on three levels: possible testing scenarios, scenarios that are expected, and scenarios that are impossible. The guide uses 7 different metrics for evaluating the performance of a bot.

• Personality: Does the chatbot have a clear voice and tone suitable for the audience and conversations?
• Intelligence: Is the chatbot intelligent? Can it remember things? Does it manage to read the context?
• Navigation: Is the chatbot conversation easy to go through? Does the conversation lead somewhere or do you feel lost?
• Error Management: Is the chatbot equipped to deal with errors well and can it recover from them?
• Onboarding: Do users understand what the chatbot is all about? and how does it interact with the user right from the start?
• Answering: Are the answers relevant to the questions of the customers? Do they fit right for the moment and context?
• Understanding: Does the chatbot understand different emojis, idioms, and requests? What is it able to grasp?

2. Botanalytics

Botanalytics is an actionable AI-powered conversational analytics platform that helps you drive engagement across your user lifecycle. With the advanced AI of the platform, you can improve conversational success with the help of A/B testing and sentiment analysis, and much more! This tool will help you assess every key aspect of your chatbot conversation.

3. Dimon

Dimon is a tool that you can use to test the conversational flow of your chatbot and the overall user experience that it delivers. You can rest assured because Dimon integrates easily with platforms like Slack, Telegram, Facebook Messenger, and WeChat!

It can be your development team that tests the bot or automated testing. Any chatbot tested before its release is beta testing! Moreover, you are bound to get more feedback from users once the chatbot is out and about! So be ready to constantly make adjustments to your chatbot even after it is released. Needless to say, ongoing chatbot testing is a prerequisite for keeping your chatbot relevant.

A few more techniques that you should consider while testing a bot:

Chatbot testing is a crucial aspect of the entire chatbot lifecycle. Lastly, to make the chatbot more conversational, initiate small talk, look for intent matching, and define a fallback, and great navigation too.

1. The tone of your chatbot

Understanding intent is pivotal for building a relevant and interactive chatbot experience! The intent is nothing but the goal or purpose of the user’s input. The chatbot voice often has several different intents which define the scope of the conversation. Let’s take a simple example to understand intent. If a customer wishes to order a coffee, we can give a command to the bot to make a coffee. However, there will be different ways of saying the same thing. The training phrases could be something like:

“I would like to order a coffee”

“May I have coffee”

“May I have a long black”

Thus, the bot should respond with a confirmation of the order like “Which coffee would you like to have?” or “Your order is confirmed!”, or “Thank you for your order”. You can see here how important it is to identify the intent.

2. Small talk

Small talks are nothing but a casual conversation between the user and the bot. It sets a good rapport with the customers and the conversation flows in a lighter and informal way. For instance;

User: How are you?

Bot: Awesome as always, thanks for asking!

You can add more and more small talks to the conversation and assess how the users are responding to each conversation. These casual talks either before a chat or in between will engage customers more and they are less likely to drop from the conversation. Greetings and jokes might not be the first thing that comes to your mind while crafting a script but it certainly plays a significant role.

3. Fallback

Fallback is a process that essentially measures the bot's response to inputs that don’t match. For instance, while placing a coffee order a user asks an unmatched query to the chatbot like “Which color shirt are you wearing?”. If they were talking to a human the response will be “I’m sorry I didn’t get you!”. Likewise, you need to train your bot to send such fallbacks when an unexpected query is entered. Write different test cases and then create a list of fallbacks.

4. Navigation

Navigation is a very important aspect of making your chatbot conversational. A chatbot doesn’t have back buttons or search boxes to help customers skip to a certain part of the interaction. Nevertheless, users do wish to go back and skip steps when conversing with your bot. For the same reason, chatbot testing should include good navigation to handle such scenarios appropriately.

For instance, while ordering a coffee if the user wishes to cancel or even change their order, then they will expect steps for the same in the chat flow. Hybrid Chat keeps the navigational flow in mind and incorporates buttons in the conversation. Let’s say a customer wants to order coffee, then our chatbot will provide them with buttons like. “Place an order for coffee”, “Go back”, “Explore today’s specials”, and “Avail an offer” etc. The best part is that you can customize the buttons and add a variety of options to provide smooth navigation for the customer.

5. Emotions

Emotion is the tone of the language which can include anger, joy, disgust, and sadness. Some of the popular chatbot platforms are using this in their chatbot messenger to gauge the user’s mood and then add text to the chat window. Test cases have to be written for grasping the emotions of users and responding appropriately. A great way of responding to emotions is to use emojis and Gifs. During your chatbot testing, you can add emojis to your bot to make them more engaging and truly conversational.

Define Metrics

Chatbot testing should be done after defining metrics properly. Even in UX, which is highly subjective, one must have clear KPIs to speed up the development exponentially. Other possible parameters include the number of steps for performing a request, the percentage of returning visitors, and the average time spent by users in a single session. Retention rates, click-through rates, and confusion handling are also important aspects to look at. These targets trigger optimizations and include quality indicators for the website. Currently, there are no specific chatbot metrics but we can expect QA companies to define them too.

Define your testing methods

Any chatbot tested before its release is beta testing! Moreover, you are bound to get more feedback from users once the chatbot is out and about! So be ready to constantly make adjustments to your chatbot even after it is released. Needless to say, ongoing chatbot testing is a prerequisite for keeping your chatbot relevant.

References:

1. https://hybrid.chat/blog/how-to-do-chatbot-testing-tools-and-metrics-to-consider/
2. https://www.optasy.com/blog/chatbot-testing-checklist-tools-techniques-and-metrics-include-your-testing-strategy

Also Checkout:

1. One such magical product that offers explainability is AIEnsured by testAIng. Do check this link.