What is Natural Language Understanding (NLU)?

What is Natural Language Understanding (NLU)?

AI Natural Language Understanding (NLU) enables a computer to understand human language in a more semantic way than simple keyword analysis. NLU makes use of both semantic and syntactic analysis to understand full sentences. It determines the intent behind a message, identifies the entities mentioned within it and understands the context in which it is used.

Note that we distinguish here between NLU and other applications for Natural Language Processing. With a simple keyword-matching system you could for instance set up a system that cancels orders, subscriptions or even meetings. However, it would not know the difference between these scenarios and therefore would not provide adequate solutions. The system powered by NLU reads the entire sentence and takes into account all the information available to it about the current conversation. It then determines which action is intended by the user and routes the request to the relevant team member.

You can see NLU in action if you have ever asked a voice assistant any questions, tried to have a conversation with a chatbot to get a billing issue sorted out without it getting escalated to a manager, or searched for something on the web using natural language instead of exact keywords and had the results show up with relevant information to what you actually meant.

How NLU actually works

NLU programs process text in different stages using supervised and unsupervised learning. Three types of methods are generally used:

–   Tokenization and embedding: This is used to split up a piece of text into the individual words, and convert each word into a numerical vector that can be compared with other words to see how similar they are to each other.

–   Named entity recognition (NER): This recognizes certain words within text as being specific entities, such as names, dates, order numbers, locations, amounts of money etc. It also recognizes what type of entity each of these words are.

–   Intent recognition: This recognizes the intent behind human language and what action the user wants to be carried out as a result.

More recently, NLU has been extended to use transformer-based models of NLP to deal with long contexts and track entities over several sentences. This has enabled models to correctly follow conversations, for example, understanding that “it” in a sentence three sentences after the relevant order has been mentioned refers back to that order.

Where NLU is used: industry use cases

This is what people are really looking for when they search for “NLU” – practical examples of where it’s used and what it does in real life scenarios.

Customer support and contact centers. The NLU algorithms enable chatbots and virtual assistants to read and understand customer messages and voice commands. It also enables real time sentiment analysis during customer interactions with human agents to intervene early and turn negative conversations around.

Banking and financial services. NLU is used to power personal banking assistants that enable customers to carry out balance checks and card blocks, report incidents of fraud, and make loan inquiries via chat. The technology is also used by banks to power their compliance tools which analyze and search call recordings and live chat conversations for signs of risky behavior by staff and to check that the correct processes and required disclosures have been observed and communicated.

Healthcare: There are many applications of NLU in healthcare including the extraction of structured information from unstructured clinical notes to aid population of the electronic patient record by clinicians as well as patient facing chatbots to aid triage and direct patients to appropriate care.

E-commerce and Retail. Search bars are one of the most common applications of NLU. As described above, a search for a warm jacket under 3000 can return the most relevant results. Shopping assistants can also answer questions such as “does this come in a smaller size?” in the context of the previous conversation.

Travel and hospitality. When customers search for travel itineraries, for example, booking assistants use NLU to understand search modifications such as “cheaper” or “direct flight” in order to modify an existing search rather than launching an entirely new one.

HR and internal enterprise tools. This functionality enables HR to provide employees with natural language interfaces to answer typical HR questions such as “How many days leave do I have left?” or “What is the company’s policy with respect to parental leave?”

Legal and compliance: NLU helps scan through contracts and identify specific clauses, obligations or even risks within large volumes of documents, enabling legal teams to prioritize their work and filter through information as needed.

Search engines. All search engines, even the simple ones, are powered by NLU as they try to understand the user’s search intent. The system can return the same results even if you type similar search terms in a different order, or even use completely different wording to ask the same question.

Benefits of NLU for businesses

Faster, more accurate resolution. Systems that use NLU can resolve a much greater percentage of incoming requests on the first pass, reducing the average handling time (AHT) per call and issue and thus lowering the overall amount of human involvement required to service customers.

Reduced support and operational costs. Helpdesk, HR and IT questions can be answered quickly and accurately by NLU powered chatbots and virtual assistants. The majority of issues can be resolved without human intervention saving businesses a great deal of money at scale.

Improved self-service. Users don’t want to fill out long forms of questions and answers. They want to search for information or ask questions in natural language. NLU-driven interfaces enable users to search for, and find, information in a way that is natural to them.

Insight from unstructured data. Unstructured text from reviews, support tickets, call transcripts, emails, and surveys such as customer feedback is very difficult to analyze at scale. NLU helps to create structured information from this type of data, such as sentiment, categories of repeating complaints, and potential issues with products and services.

Scalability across languages and volume. One of the biggest advantages of using NLU is that it can scale to process huge volumes of language-based requests from customers or employees. Once an NLU model has been trained on a language, it can be extended to support other languages as well, without requiring a complete re-write of the model’s logic.

Foundation for agentic AI. Agents can use NLU to correctly interpret user requests before delegating to other agents or systems. Misunderstanding the user’s intent at the first step in a chain of interactions can have far reaching consequences down the line.

What is a good NLU vs. What is not good NLU

It’s worth noting that there are many “AI powered chatbots” in the market that are not utilizing strong NLU. We can typically tell after the first real example conversation that it’s not utilizing a strong NLU.

Weak NLU implementations typically require users to interact with the system in exactly the same way that the developer interacted with it while creating examples. For example, “What is the status of my order?” may work perfectly, but “Cancel my order” and “I ordered something last week, where is it?” will not. Typing a question with misspellings or incorrect capitalization can also cause problems. Furthermore, NLU systems that have weak NLU often forget the context of a conversation after a single turn of interaction, requiring the user to restate information that the user previously provided.

The strong NLU system is able to hold all of the context of a conversation and still handle multiple intents in one message. The system will also be able to degrade gracefully when it is unsure of what to do next, asking a targeted question in order to attempt to come to a resolution. This is in contrast to the weak NLU system that is unable to handle these types of complex scenarios.

For enterprises looking to use NLU in their chatbots, conduct the following simple test: run messy, real-life language through the system (do not use clean example sentences from a demo). You will typically find that the system breaks on unusual language very quickly.

NLU vs. NLP vs. NLG

These three concepts are often used as if they were interchangeables, but they represent distinct layers within the same large architecture.

NLP (Natural Language Processing) refers to all aspects of processing human language, including grammar and syntax, for the purpose of enabling computers to understand human language and to generate human language for use by computers.

NLU ( Natural Language Understanding). This is the half of NLP that is able to understand an input, with the goal of doing something with that input.

NLG (Natural Language Generation). NLG is the reverse process of NLU. It is used to automatically generate text from information stored in databases or created by applications. For example, it can be used in chatbots to automatically respond to user input.

The architecture for a typical conversational interface would therefore comprise input from users which would be analyzed by NLU for intent and meaning. NLG would then be used to create appropriate human language for output from the system. So a system which appears to listen and then talk back to users would have both NLU and NLG active.

Where NLU still runs into trouble

It’s worth noting the boundaries of what NLU can do. While a system can be taught to recognize many nuances in human language such as sarcasm, code-switching and idioms, there are areas in which it can struggle, including recognizing heavy dialect use and industry-specific jargon not present in the NLU model’s general training data (e.g. web text). Such models typically require fine-tuning on domain-specific data to achieve strong results on enterprise use cases.

The gap between generated text appearing to be correct and actually being correct also needs to be addressed. Even with high-quality generated text, systems are frequently misinterpreted because the underlying intent was incorrectly understood by the NLU.