What to do if your chatbot doesn’t know the answer
Many companies have been using chatbots to provide automatic, scalable, and personalized customer care services. You may think building a chatbot to this end is an easy task, just defining a sequence of messages according to the standard conversation flow, right? No, there is a lot more to it than that!
In order to deliver a pleasant experience to end-users (or simply users) and to keep them engaged, the Conversational Designers need to put in quite a bit of effort. Usually, they deep-dive into diverse articles and studies to better understand the business. Then, they assume the role of the user, empathizing with the major pain they could face while using the service and/or product.
As result, many artifacts are created, such as experience and empathy maps, CSD (Certainties, Suppositions, and Doubts) Matrix, the definition of persona, and so on. Those materials are essential to build a good chatbot. Do you want to go deeper into this? Great! I can suggest to you the materials provided by Chatlayer experts available on this page.
The Problem
So, after that work, can we guarantee that chatbot will be able to answer all questions that users may have? Unfortunately, even when the work was very well done, unexpected conversations are very likely to happen. This means that the bot will not understand what the user is trying to say and the classical “Not Understood” messages or any of its variations will be sent to the user.
For example, our team recently developed a chatbot to help users of a big Online Tickets company, which sells tickets for movies and shows in Brazil. The chatbot’s goal was answering frequent questions users can have, such as information of event (address, date, time, etc.), payment methods, cancellation and refund policies, and so on. In general, the chatbot served the thousands of customers who interacted with the bot monthly very well, however, we were surprised when we read the messages that resulted in a “Not Understood”.
A customer asked if he could wear shorts at a concert, while another asked what the shortest route was to go to the event. Despite that those messages are related to the chatbot’s context, i.e., tickets and events, they were completely outside of the “knowledge” of the chatbot.
As you can imagine, this is not trivial, and it might not even be feasible to map and prepare the chatbot for all situations. But we should avoid the chatbot answering “Not understood” as much as possible, because it can easily degrade the users’ experience. Once they might doubt chatbot’s ability, they might stop interacting with it.
Ways to Solve the Problem
So, it seems we have a hard problem to solve. Let’s introduce some ways to deal with the “Not Understood” problem, from the simplest to the most powerful one, which will enhance the users’ experience.
Pareto Principle (80:20 rule)
The well-known Pareto Principle applied to chatbots says “80% of user expressions will match 20% of chatbot intents”, as widely mentioned in blog posts [1, 2] and scientific literature [3, 4].
In this way, you can significantly reduce the work of your team, by identifying the main intents (the 20%), training the NLP engine, and preparing the conversational flows that should be triggered by each intent. Thus, your chatbot should be able to answer satisfactorily most of the times (80%).
Great! So, we can expect that 20% or less of the interactions will fall into the “Not understood” case. When this happens, we can transfer to a human agent, or to a support ticketing system, without degrading the user experience, right? Maybe! Consider the following scenarios:
- A small or medium business, where the daily volume of interactions with the chatbot is below 1K.
- A big company with 100K or more interactions with the chatbot daily.
In the first scenario, at most 200 interactions daily fall into “Not understood” and, even if all of them want to be directed to second-line support, it should not be a big issue for a small support team to handle them quickly.
On the other hand, for big companies, it seems not that reasonable to let 20% of the daily volume (20K or more conversations) go to second-line support without trying to reduce it. After all, high volumes like this would require many human agents (consequently, increasing the cost), and might increase the response time from hours to days. This will have a negative impact on the user experience. And, as you can imagine, the higher the volume the worse the problem will be.
Move up the Pareto Curve
One obvious way to improve the experience is to move up the Pareto curve and cover more than 80% of the use cases. You can do this by adding more intents and their corresponding expressions and conversational flows. But this would mean that you still have 80% of the work ahead of you to cover the last 20%. This also means that going from 80% coverage to 85% is less work than going from 85% to 90% coverage. You have run up to the law of diminishing returns [5].
But, keep calm, and let’s explore how we can solve this problem by using search engines.
Search Engine Lookup
Search engines, like Google, are automated processes to fetch web pages that are similar to a given question. In this way, we can consider that all expressions falling into the “Not understand” case are the input queries to a search engine.
As described in this Chatlayer’s tutorial, we can create a search engine lookup as a fallback for a “Not understood”. Using a list of predefined websites, where it could find answers to these questions, for example, your FAQ page or your documentation. In this tutorial, we show how you can use Google’s ‘Programmable Search’ to perform a lookup search on the website and return the first five results. But you can obviously use other search engines as well.
So, now your chatbot will be able to provide further information to users, even when no intents match. Awesome! But it can be even better?
Most search engines available on the market only support keyword-based search. Given a user expression those engines search on the website for words that also occur in the question. But this doesn’t really look for answers to your question and sometimes users express themselves in different words that have the same meaning. A keyword-based search engine will not be able to deal with that. It will return the wrong pages or no results at all.
Chatbot users tend to send non-formal expressions, containing slang, misspellings, contractions, and so on. Which makes matters even worse. To overcome that, ML & AI models come into play to provide more robust solutions.
Search Engine Powered by AI
Recently, Sinch launched a product called AskFrank (see more on AskFrank website), which is an intelligent question-answering search engine able to understand natural language expressions. It can understand what you are asking even if you don’t use the right keywords.
AskFrank can index several sources of content, for example, FAQ websites, Confluence pages, PDF and Word Documents, into a knowledge base and make them searchable. In other words, most of the materials that your company already has can be used to search and find meaningful answers.
Are you curious to see AskFrank in action? Great! You only need to access the Sinch Community Support website and interact with the chatbot embedded on the page. As you can see on this website, there are many articles organized into categories, such as Messaging, Chatlayer, Contact, and so on, where the article name is a common question, and its content is the answer to the question.
That is all AskFrank needs to work properly. For instance, send the question “How askfrank work?” and you will receive an answer that refers to the question “How does AskFrank work?”, which is available on Sinch Community > More topics > AskFrank. Below, the figure illustrates this example:
Impressive, right? Besides Sinch Community, some customers from industries of the different sectors, such as retailers, fintech, and pharmaceutical, are already using AskFrank to provide better experiences to their customers.
Final Remarks
In this blog post, we briefly discussed a common problem for chatbots, handling unexpected expressions, without degrading users’ experience by answering “Not understood” many times.
We presented the Pareto Principle, or 80:20 rule, as a good practice widely applied by chatbot designers to avoid many cases of “Not Understood”.
Beyond that, we also present as search engines could be a good solution to fetch meaningful answers, creating delightful experiences for end-users.
Finally, we introduce AskFrank by Sinch, an intelligent search engine, which fills out gaps of keyword-based search engines and brings new powerful features, such as supporting over 100 languages and the ability to index content from many sources. Besides that, AskFrank doesn’t need any training, you just connect it to your data sources, and it will do the rest, helping chatbots deal with unexpected questions without increasing their complexity.
To know more or request AskFrank demo, please, get in touch. Just send us an email to info@askfrank.ai. We will be happy to help you solve your problem.
References
[1] CABOT, Jordi. The Pareto Principle applied to the design of chatbots. Xatkit, available in: https://xatkit.com/pareto-principle-chatbot-intent-design/, 2020. Last access on March 23, 2022.
[2] KeyReply Team. The Conversation Design Process: How to Design Chatbot Dialogue. KeyReply, available in: https://keyreply.com/blog/chatbots-conversational-design/. Last access on March 23, 2022.
[3] CHAIDRATA, Alvin et al. Intent Matching based Customer Services Chatbot with Natural Language Understanding. In: 2021 5th International Conference on Communication and Information Systems (ICCIS). IEEE, 2021. p. 129–133.
[4] FREED, Andrew. Conversational AI. Manning Publications, 2021, p. 129.
[5] HAYES, Adam et al. Law of Diminishing Marginal Returns. Investopedia, available in: https://www.investopedia.com/terms/l/lawofdiminishingmarginalreturn.asp. Last access on April 6, 2022.
