What Is Retrieval-Augmented Generation (RAG)?

What Is Retrieval-Augmented Generation (RAG)?

What Is Retrieval-Augmented Generation (RAG)? RAG (Retrieval-Augmented Generation) is a combination of information retrieval and generative AI. It consists of two main components: information retrieval (retrieval) and generative AI (generation). First, useful information is retrieved and then passed to the generative model.

Large language models can generate lots of useful answers. The answers are based on what the model learned from the data that it was trained on. That data has a lot of limitations. For example, it won’t include a company’s most up-to-date documents, or information that is specific to a project, such as knowledge that is unique to a particular group of people.

This search space can include a company’s internal documents, such as contracts and meeting notes, product information, academic research, websites, support articles, databases and more.

As the first step, RAG searches for information on the information that the large language model can use as input to the generator. The relevant material is then added to the input of the generator as additional context, such that the generator can then use the original input query in conjunction with the retrieved material to generate an optimal answer.

Thus RAG is particularly suitable for use cases in which an AI application is working with information that changes often or is organization-specific.

How Does RAG Work?

A RAG system typically consists of two steps: retrieval and generation.

This flow starts with analyzing the user’s question or instruction in order to search the connected knowledge base for relevant information.

For the retrieved information, an integration layer is then used to merge this with the original prompt. The result is an augmented prompt which includes additional context. This prompt is then fed to the generator in the form of an input.

This information is then used by the generative AI model in the RAG system to generate an answer to the user’s question. This answer can then be cross referenced against the original sources that were used in the answering process, in order to provide complete transparency around the answers that have been generated.

In simple terms, the workflow is:

User query → Retrieve relevant information → Add context → Generate response.

Because the quality of the generated answer crucially depends on the quality of the retrieved information, retrieval is one of the most important components of a RAG system.

What Are the Main Components of a RAG System?

A typical RAG architecture consists of four main components: Knowledge Base, Retriever, Integration Layer, and Generator.

In the architecture of the RAG system, there is a knowledge base consisting of external information (documents, PDFs, web pages, user manuals, reports, databases etc.) that the system can retrieve.

Here, the retriever searches through all the information stored in the knowledge base. RAG systems typically use information such as embeddings to search for information that is semantically similar to the input query. So, the information retrieved may not contain exactly the same keywords that the user used in their question.

The integration layer: In this layer all the input from the retrieval step is processed and used to construct a suitable context for the generator.

The generator in a RAG system is the language model which generates output based on the input query and the retrieved information from the Knowledge Base.

In addition to the main components that are required to implement a RAG system, additional components such as ranking, input processing and validation, and monitoring may be implemented.

How Does Retrieval Work in RAG?

The retrieval step within RAG is used to retrieve relevant information from the connected knowledge source. Before you can start searching for answers using a RAG system, the source information needs to be prepared. Large documents need to be chunked (i.e. split up into smaller sections) and then converted into embeddings, which are numerical vectors that represent the meaning of the chunks.

As the question will most likely not look the same as the content in the knowledge base, this allows for very powerful retrieval of relevant content. This content is then used as additional context for the model.

This is important, because the user question and the source information will often not have used exactly the same words. The algorithm should be able to identify related semantic concepts.

For example, the user’s question “How do I change my account password?” could retrieve documentation with the heading “Credential Reset Procedure” and still be very relevant.

This information is then also added as additional context to the generative model.

Why Is Chunking Important in RAG?

The way information from a knowledge base is chunked (i.e. split up into smaller parts) before it is retrieved is an important factor for RAG systems.

However, large chunks of source information can result in the retrieved information containing too much irrelevant information, which in turn can be difficult for the system to pinpoint the relevant part for the query.

Similarly, small chunks of content lead to poorly integrated context from retrieved sections. A single retrieved section typically only contains a portion of the necessary information for answering a user’s question.

Therefore, the design of a RAG system has to consider the chunk size and the document structure in order to process the appropriate information for answering the questions.

We also require knowledge sources to be regularly maintained in order to ensure RAG’s performance, as out-of-date or incomplete information may lead to low quality of information retrieval, and consequently of generated responses as well.

Where Is RAG Used?

RAG can be used in a wide variety of cases where a generative AI needs information that goes above and beyond its general knowledge.

Customer Support

Customer service assistants can connect to product documentation, support articles, policies and account information to answer customer queries. The system retrieves the relevant information before answering a question.

In a similar manner, support teams can also search through large knowledge bases. Instead of manually looking for individual documents, they can be retrieved in seconds.

Enterprise Knowledge Management

Information in an organization is often distributed over documents, policies, manuals, reports, and systems.

A RAG-based assistant can then be used to search a company’s knowledge base of internal documents, policies, and more. Employees can ask questions in natural language instead of having to search for individual documents in different folders.

Research

Researchers can also use RAG systems to look for relevant papers, documents, reports, etc. that the model can then use to create summaries or answers.

This is particularly useful in situations in which you are dealing with large amounts of information, possibly even changing frequently. The retrieved sources can then be used to verify the model’s generated answers.

Content Generation

RAG can be used to support AI content generation by providing it with the authoritative information it can use to generate content and then verify against.

An example of such an application could be the generation of a product explanation based on previously approved product documentation.

Market Analysis

To perform a Market Analysis a RAG system can retrieve market reports, customer feedback, news, reviews and many more.

This large collection of information can be searched using natural language questions.

Recommendation Services

RAG supports recommendation systems as well. As with summarization, a retrieval process first retrieves relevant information for products, content, customer preferences or past user interactions, and then the model generates recommendations based on this information.

How Does RAG Reduce AI Hallucinations?

RAG can also help reduce AI hallucinations because it gives a generative model relevant information to work with.

Note that while a model operating only from its learned parameters can generate answers based on all sorts of patterns that do not actually correspond to the information requested, a RAG system grounds the response in the retrieved information.

RAG by itself doesn’t guarantee the generative model doesn’t hallucinate, either, because the RAG-retrieved information itself could be wrong in a variety of ways (e.g., outdated, missing a crucial aspect, entirely irrelevant).

Even if the source information is accurate, there is the potential that it is out of date, incomplete, or not relevant. Additionally, the model may misinterpret the retrieved information and use it to create output that is not relevant, or generate information that the retrieved information did not support. But note that reliable source data, effective retrieval, proper prompting, evaluation and monitoring of a RAG system is important for avoiding hallucinations.

RAG vs. Fine-Tuning

This section compares RAG with fine-tuning of the language model for a given domain.

The RAG architecture can be used in conjunction with fine-tuning to improve the model’s behavior and to allow it to use the current, domain-specific information that is stored in the external knowledge sources