What is Embedding?

What Is an Embedding?

An embedding is a way to represent information (text, images, audio, etc.) as a numerical vector that an AI program can understand and compare. These vectors are typically high-dimensional, meaning they have many numbers. But this creates a problem: how can one compare two vectors? The typical method for comparing two high-dimensional vectors is to use a measure of similarity called the cosine of the angle between the two vectors.

The numbers within a vector are often irrelevant on their own, and are typically given meaning when compared with the numbers in other vectors. By placing vectors with similar characteristics close together within a high-dimensional space, such as a 300-dimensional vector space, an embedding attempts to capture meaningful relationships between pieces of information.

As an example, consider the two following sentences: “How do I reset my password?” and “I forgot my login password”. These two sentences are very different, when looking at the surface level of their word choice. However, an embedding model would map both of these sentences to semantically similar locations in the vector space, because they describe very similar intent.

Therefore, the same technique can be applied to other types of information in order to enable comparison based on meaning. This is the core purpose of embeddings. These representations can be not only text, but also images, audio, products, users, documents and more. Any type of data can be embedded.

How Do Embeddings Work?

An embedding model maps input(s) to vector(s) in the embedding space. Typically, each embedding model is trained on a large corpus of data and thus learns a lot of pattern information within that data.

When you input text in an embedding model it is processed by the model and an output vector of a fixed number of dimensions is generated where each dimension contains a number. This resulting vector can then be compared to other vectors in the embedding space. Most typically, cosine similarity is used to compare vectors. But there are also other distance measures that can be used.

A higher similarity score means that the two pieces of information represented by the two vectors are more similar according to the embedding model.

How Text Is Converted Into Vectors

The input to the text embedding model can be text of any length, including words, short sentences, paragraphs, documents, questions, and more. When converting text to a vector, the Model can output a number of dimensions, and each dimension can have a value which characterizes the input text.

For longer documents it’s very common to embed sections of that document. Thus you can retrieve the relevant parts of longer content based on search queries. This vector can be stored in a database and compared to other document vectors to find similar documents.

How Embeddings Capture Meaning

As opposed to simply labeling information with words, the relationships in the information are captured by the embeddings. Words with similar meanings are located near each other in the vector space. For example, words like “doctor” and “physician” 

Embeddings allow words with different spellings to have related meanings embedded near each other in vector space. For example the words ‘doctor’ and ‘physician’ have different spellings but are related so an embedding model would map them to nearby points in vector space.

This is equally relevant for longer structures like sentences or even documents as well. In these cases, similar content relating to similar subjects will result in similar vectors as well.

Note that the way AI systems “understand” meaning is quite different from the human understanding of meaning and therefore their vector representations will also be different. The quality of the vector representations also depends on the model that created them and on the data with which this model was trained.

What Are Embeddings Used For?

When an AI application needs to compare, search, organize or even retrieve information that has similarities with other pieces of information, then embeddings can be of great value.

The most common use case for semantic search is converting search queries into embeddings, which are then compared with the embeddings of stored content. Semantic search systems are able to identify information that is semantically related to a user’s query. Recommendation systems also use this type of representation. Users, user behavior, products and content can all be mapped into a vector space in such a way that the proximity of items to a user indicates relevance to that user.

As well as semantic search, embeddings can be used for classification and clustering of data, for example data with similar representations can be grouped together to form clusters. Also these vectors can be used as features for a machine learning model for classification.

Semantic Search

By building a semantic search function into your application, users can search for information in natural language. Thus, for example, a search for “ways to reduce cloud expenses” can yield content that discusses “cloud cost optimization”

Most keyword-based search approaches will only retrieve results with exactly the same word order as the query. Embeddings enable search approaches that actually understand the semantic meaning of search queries and documents.

For instance, a search query such as “ways to reduce cloud expenses” might return results like “cloud cost optimization”. Semantic search is a particularly powerful tool for Knowledge Bases, Enterprise Search, Documentation and Q&A Systems.

Recommendation Systems

Embeddings can represent users and items within the same or related vector spaces. In a recommendation system, users and items can be embedded in the same or similar vector spaces. Thus, representations of products can be compared with the representation of a user’s interests or past behavior.

Relationships between products can also be represented. Two different products are recommended to the user because their characteristics or usage behavior are similar, even though they are from different categories.

Document Retrieval

Embeddings can also be used for document retrieval in large collections of information.

This information can then be organized within a document. A document can be split up into sections. Each section can then be embedded. The user’s question can then be embedded as well. This embedded question can then be compared against the document’s embedded sections. The model then searches in the mass of stored documents for the most relevant parts and hands them over to the next AI component.

Embeddings are widely applied in many areas of Enterprise Search, Knowledge Management, Document Analysis and AI Assistants.

Classification and Clustering

This also enables the classification of data and the clustering of similar samples. In the context of classification tasks the embedding vectors can be assigned to categories. For clustering tasks similar vectors are grouped together. There is no need for a label for every item.

For instance, customer feedback can be embedded and then clustered to similar issues in the feedback. Recurring themes can then be found in large volumes of unstructured feedback.

What Are the Different Types of Embeddings?

It’s worth noting that all types of data can be converted into embeddings, depending on how a particular embedding model was trained and used in applications.

Text Embeddings

Text embeddings represent written language as numerical vectors.

Text embeddings can represent words, sentences, paragraphs or even whole documents. They are used in many AI applications like search (semantic search), document retrieval, content recommendations, text classification and many more. Recent architectures like RAG models are heavily based on text embeddings as well.

Embeddings – as any other method – can be of poor quality. The quality of text embeddings greatly depends on the model, the training data, the language, the domain and on several other factors that can affect the input text.

Image Embeddings

Image embeddings represent visual information as numerical vectors.

An image embedding is used to represent visual information, which can then be compared with other images to identify similar images, perform image classification, find similar items, etc.

An example of using image embeddings for image search would be a user uploading an image and then comparing the embedding for that image against the embeddings for all stored images with similarity to that image being returned to the user.

Audio Embeddings

Audio embeddings represent information contained in sound or speech as numerical vectors.

Audio can be used for speech, music, sound classification and audio similarity. Raw audio signals can’t be compared but the learned numerical representations of them can be.

Audio embeddings can also be used for organizing and searching large collections of audio data.

Why Are Embeddings Important in AI?

An embedding is a mathematical way to represent complex information in a useful way.

Information in AI applications today mostly comes in the form of text documents (e.g. customer feedback, documents), customer or user conversations, product or service descriptions, images and even audio files. All this information needs to be processed by computers, too.

Information is converted into numerical representations, and then measured and compared in order to search for related content, identify similarity, group information, and connect relevant information.

In particular, embeddings are incredibly valuable in AI working with large amounts of unstructured data. Instead of relying on the exact words in a document or manually created categories, embedding-based similarity search allows you to search for similar information within a massive corpus of information.

Embeddings and Semantic Similarity

For this purpose semantic similarity – i.e. how similar in meaning two pieces of information are – is an important measure.

These representations enable the measuring of semantic similarity by comparing the distance between the vectors. Generally, two vectors with sufficient similarity are considered to represent semantically similar information, even if they were generated from vastly different sources of information.

By their nature, semantic similarity detection systems can typically identify similar semantic concepts regardless of word choice.

In short: nothing is guaranteed. Bad input, a bad embedding model or a bad method to measure similarity can all lead to poor results when it comes to retrieval.

Embeddings and Unstructured Data

A lot of unstructured information is being produced by organizations in the form of documents, emails, customer service conversations and other unstructured data.

By embedding information such as content into vectors, AI applications are able to search for and compare them in a powerful way. Information that consists of unstructured data can be analyzed using the embedded information in a wide variety of ways.

The primary reason for the increasing use of embeddings in modern architectures for handling unstructured information is the function that embeddings perform – that of bridging the two types of information and enabling their processing by machine learning models.

How Are Embeddings Used in RAG?

A common application for embeddings in AI applications is to enable retrieval of relevant content for the generation of final answers, such as in **retrieval-augmented generation (RAG)**.

Note that in a typical RAG architecture, a document (like a long piece of text) is split up into smaller pieces or sections of text, and then embedded, one section at a time. Once a question has been converted into a query vector, that query vector is compared against the stored vectors of all sections of documents, and the most relevant document section is selected as context for the generator. This generated information is then used as context by a generative AI model to produce answers.

In general, an AI model’s ability to generate answers depends on the information with which it has been trained. In cases in which there is a wealth of knowledge within an organization that has not been captured in a structured form, the information can be embedded in documents, internal policies and procedures, product documentation, technical manuals, knowledge bases and more.

The quality of generated answers also crucially depends on the quality of the retrieval of relevant information for a given query. If a model retrieves too little or even irrelevant information for a question, it will also fail to generate good answers, since it can only work with the information that it has found.

Real-World Examples of Embeddings

Embeddings are used in many AI applications today, but usually you don’t see them at all.

Search and Information Retrieval

In Search Systems like Search within an Enterprise Knowledge Base, Technical Documentation, Customer Service Systems, etc. semantic retrieval can be used to retrieve documents and information that match best the meaning of a search query.

Unlike search that requires users to memorize exact wordings in documents, semantic retrieval systems using embeddings enable users to search for information using related terms and even concepts.

Product Recommendations

For an e-commerce website with products of different descriptions, interests of customers can also be represented as embeddings and similar products as well as products of similar interests of a customer can be suggested to the customer.

For example, a movie about a wizard, stored with its description as walking the magical path, would likely be closer to a movie about an awesome superhero, stored with its description as having extraordinary powers and saving the world, than to a romantic comedy about friendship.

Customer Support

The embeddings can also be used in customer support. In such a system, the system will match a customer’s question with relevant articles from the knowledge base or previous correspondences with the customer.

The same is true for the support of a customer. Here the same semantic similarity is used to search for corresponding articles even if the question does not match the article 1:1.

For customer support issues, also customer conversations (e.g. via chat) can be embedded into an appropriate structure. Similarly, for similar questions, complaints, requests, etc., it can be useful to cluster them.

Content and Document Analysis

You can also use embeddings for analyzing big volumes of documents. In such a case you will get a vector representation of all documents in your knowledge base, which you can compare with each other.

All types of documents (reports, contracts, technical documents, research papers etc.) can be represented as vectors and compared for semantic similarity in order to perform a search or to support document discovery within Knowledge Management systems.

This also enables document discovery, knowledge management, duplicates detection, classification and retrieval within large collections of documents and correspondence.