Today, more and more companies are investing in chatbots. However, not all chatbots are the same. One of the reasons? The components of their engine. Some are based on NLP bricks while others, more elaborate, are based on NLU bricks. But what exactly is the difference between NLP and NLU?
Natural Language Processing (NLP) is what allows humans and machines to communicate in natural language, natural language being the syntax and vocabulary of humans. In short, NLP consists of automatically analyzing in a literal way the sentences formulated orally or in writing by a human being. The machine will not try to understand the meaning of what is said or written: it will analyze and recognize the lemmas used, i.e. the words used, without interpreting them, and respond in the best possible way.
Natural Language Understanding (NLU) is a sub-branch of NLP which is the combination of NLP techniques with Machine Learning (ML) or Artificial Intelligence (AI) technologies to understand natural language. NLU is therefore an advanced form of NLP which, this time, will allow machines and humans to communicate in natural language while improving the understanding of natural language by machines. In short, the NLU will automatically analyze the semantics of sentences formulated by a human being in order to understand the meaning of what is said or written: the machine will analyze the lemmas used to understand the meaning of the utterance and respond to it in the best possible way.
To go further, the “meaning” of a statement is identified by the intent and the entities. Intent corresponds to the overall meaning, for example, the action the user wishes to perform. The entities are the other strong elements contained in the statement. Let’s take the example “I would like to go swimming on Thursday in Toulouse“:
– “Go swimming” corresponds to the user’s intention, here to the action the user wishes to perform. The chatbot can translate the user’s intention by “search pool”.
– “Thursday” and “Toulouse” are two entities of the statement which, respectively, correspond to the date (when) and the city (place).
The user is looking for a place where he will be able to swim on Thursday in the city of Toulouse: this is what the chatbot will understand thanks to the NLU.
To understand the difference between NLP and NLU, I often use this parallel to machine translation: NLP would be word-by-word machine translation while NLU would be neural machine translation, i.e. translation based on neural network technology. NLP will simply transcribe the form of the sentence word by word without considering the meaning of the words, if they have several possible meanings, while NLU will try to transcribe the meaning of the sentence as best as possible without worrying about the form. NLP deals with the form while NLU deals with the meaning of the statement.
For example, for the sentence “It rings me a bell” in English, the word-by-word translation would be “Ça me sonne une cloche” while the neural translation would be “Ça me rappelle quelque chose” in French. In metropolitan France, it is the neural translation that is correct, while in Quebec, it is the word-by-word translation that is correct. The context of application of NLP and NLU is therefore very important.
Always keep in mind that a blog or article that deals with the meaning of the statement will always refer to NLU bricks, even if the author uses the name “NLP”. This is not a mistake in itself since NLU is a sub-branch of NLP, but it is a lack of precision that can lead to confusion.
The main objective of a chatbot is to understand the request expressed by a user in natural language and to extract the right information from it in order to be able to react or answer in the most coherent way possible. If you don’t know which brick to choose between NLP and NLU for your chatbot, your budget may give you the answer. NLU technologies are, by experience, the most powerful technologies available today. Unfortunately, performance rhymes with high cost. However, depending on your chatbot’s field of application, NLP bricks could be more than enough to obtain excellent results. Studying the different cases of use and application of your chatbot will help you make your choice. For example, if your chatbot has to create answers from scratch to users’ questions, NLU bricks will be necessary because creating an answer in natural language is much more complex for a machine than finding the answer to a question in a database, like a FAQ, with NLP.
After reading many blogs and articles, it also seems to me necessary to point out that it is more than complex to classify the chatbot engines currently on the market because each of them uses very different algorithms. The performance of these engines varies according to the scenario chosen: for a given scenario, the first engine will be the most efficient, while for another scenario, one or more other engines will be more efficient than the first one. You must therefore always keep a critical mind when browsing the rankings, but also when using these different chatbot engines!
Don’t forget that your choice between NLP and NLU depends on the desired application areas for your chatbot and your budget, and not on the performance of other chatbots.
What about Hubi?
When the Hubi architecture was developed, its creators first tested several approaches and asked themselves many questions: which branch to use, NLP or NLU? Which platform to choose? Should we develop our own platform? Answering these questions could be the subject of a whole blog, but the answers are already all found concerning Hubi!
