What Is a Vector Embedding?

What Is a Vector Embedding?

A vector embedding is a form of information representation for use by a machine learning model. The embedding of information, such as words, text, images, audio, is typically a vector of numbers of hundreds or thousands of dimensions.

The values in the vector (in an ideal embedding) are not arbitrary labels, but similar data points are close in the embedding space, whereas dissimilar data points are far from each other.

The words “doctor” and “physician” are similar in meaning so their word vectors would be close in the vector space. This then allows the AI to do the matching in terms of the relationships between concepts. 

Note that the terms vector and embedding are sometimes used synonymously. However, vector is generally more correct since it refers to a simple list of numbers. In machine learning, the term embedding is more specific since it refers to a mapping that has certain properties, such as capturing meaningful structure in a high dimensional space. Most modern embeddings are vectors, however.

How Does Vector Embedding Work?

Vector embedding models are typically trained on a large dataset of the type you want to embed. After training the model on a huge dataset of words, it can take in a word you type and map it to a vector. For example, an image embedding model might take in an image of a dog and map it to a vector in space. The vector has a long list of numbers that represent different features of the image such as color, shape, size, etc.

In particular, the embedding model has learned to represent good features of the input to the model to map them to a vector space. The number of values in the embedding vector determines its dimensionality. While usually in the hundreds or thousands, this dimensionality is a parameter that the modeler can set for the particular embedding that is used for a given application.

Converting Data Into Vectors

First, some data (e.g. text, images, etc.) is fed into an embedding model. That data could be individual words, full sentences, even entire documents. For images, the entire image is fed in, and for audio, characteristics of sound are used.

A model makes a numerical array to represent the input. Typically the output of an embedding model is a fixed size vector of numbers. Each number in the vector corresponds to a dimension or feature of the input.

It’s also important to note that the meaning of each of the numbers in an embedding is generally not obvious and are distributed across the vector dimensions. This means that understanding an individual dimension of an embedding is generally not possible without additional information.

Representing Meaning as Numbers

A key aspect of the meaning in an embedding is how similar other inputs map to nearby points in vector space. The relationships in the data between the individual input cases are captured in the embedding. For text the relationships capture semantic meaning. For images the relationships capture local structure of the objects in the images.

For text the relationships between the input tokens (words and their orthographic neighbors) can capture semantic meaning, context and more. Images can be represented in an embedding space that captures relevant parts of the image and their spatial relations. Audio can be represented in an embedding space that captures features of sound such as pitch, timbre, and the phases of different frequencies.

This then allows information to be compared mathematically to determine whether similar information exists within other data sets. Rather than simply determining whether two pieces of text contain the exact same words, for example, information embedded within vectors allows AI systems to find very similar information by comparing the vector representations of the two pieces of information.

Measuring Similarity Between Vectors

For similar reasons the number of dimensions of an embedding vector can also be referred to as the dimensionality of the data after it has gone through an embedding process. It is then possible to compare two vectors in the resulting space of data by using various measures of similarity of vectors, like cosine similarity, Euclidean distance, or dot product.

Cosine similarity is the measurement of the angle between two vectors, and is most frequently applied to text (natural language) data. Euclidean distance measures the straight-line distance between two or more vectors in space. A dot product measures the amount of change in each of the corresponding components of two input vectors, for example [a, b] . [c, d] = ac + ad.

There are several measures for similarity of two vectors in a vector space. A fundamental property for all of them is that vectors representing similar information should be closer to each other in the vector space.

What Are the Applications of Vector Embeddings?

All sorts of AI tasks are powered by vector embeddings. The main thing they do is to let a model treat all sorts of data in the same way – words, images, whatever. Then it can use the same machine learning algorithms to work with that data.

Semantic Search

These search systems can be further enabled by semantic search systems, written on top of vector embeddings of all text information to search. The above image illustrates a simple example of a search for documents containing the search string “ways to keep a phone battery healthy”.

For example, a user could search for “ways to keep a phone battery healthy” while the relevant document contains the phrase “improving smartphone battery life”. A keyword-based search engine would not consider these two search phrases to be similar, whereas semantic search using vector embeddings would.

Many search engines, enterprise knowledge systems, and question-answering systems today also use embeddings to index their knowledge bases.

Recommendation Systems

To make embeddings useful for recommendation systems, items can be represented as vectors too, which can then be compared in order to find similar items. In e-commerce for example products with similar characteristics can be found, or in content platforms articles, videos or even music with similar characteristics can be found.

For example, an e-commerce platform could use product embeddings to suggest products with similar characteristics. Alternatively, a content platform could use embeddings to suggest articles, videos or music that are similar to other articles, videos or music based on the content itself.

Retrieval-Augmented Generation (RAG)

A Retrieval-Augmented Generation (RAG) system is also utilizing vector embeddings of documents (or other knowledge sources) that are stored in a database for vector search.

A user asks a question, and his question is converted to a vector as well. We then search for the most semantically similar vectors, and use the information associated with these vectors to generate answers. Such information can be retrieved from large external knowledge bases such as documents, web pages, databases, etc. This information is then used as context to answer user questions.

Using Retrieval-Augmented Generation (RAG) enables AI applications to use a vast external knowledge source to complement the information a model was trained with.

Classification and Clustering

Embeddings are also used for classification and clustering. Classification uses vector representations of data to predict the category of new data points. For example, sentence embeddings can be used as input for a sentiment classification model. Clustering groups similar data points in vector space.

Similar to classification, data points that are similar to each other can be grouped into clusters of information. The clustering process groups similar information without the need for human intervention to categorize each piece of data.

Similarity Search

Similarity search can be viewed as a broader application of vector embeddings. The main goal is to find data points in space that are most similar to a given query.

Similarity search can be applied to all kinds of data such as text, images, audio, products etc. A user can search for images similar to an example image he provides. He can also search for documents containing text similar to a given query.

To perform efficient similarity search on top of large collections of embeddings, vector databases are used.

What Types of Data Can Be Represented as Vector Embeddings?

This, of course, is not limited to text. Other data types are also able to be embedded into vector spaces, when appropriate embedding models are used.

Text

Text embeddings capture information about words (e.g., the word “hello”), sentences (e.g., “hello, how are you?”), paragraphs (e.g., a long piece of text) and even entire documents (e.g., books). In general, text embeddings are very good at capturing semantic information about the text (i.e., the meaning of the words and how they relate to each other in context).

The same applies to text: we can embed single words (word-level), short texts like questions or statements (sentence-level), and longer texts (document-level).

Images

Image embeddings: This type of embedding maps the visual information (such as shapes, objects, and patterns) found in images into numerical vectors. Images are then searched for in a similar manner to how text is searched in order to find the most similar images.

Images: Similarity search, classification, Recommendation Systems, Multimodal search (e.g. text2image search). Images can be embedded to connect text to images.

Audio

Audio embeddings can represent the characteristics of sound. There are a number of applications of audio embeddings including voice assistants (e.g. Siri), music recommender systems, audio recognition and many more.

These embeddings are typically generated using neural networks that are task-specific, i.e. that have been designed to perform a specific task. For audio embeddings, these could be convolutional (CNNs), recurrent (RNNs) or even transformer-based models.

Other Multimodal Data

Embeddings can represent product information as well as information in the form of graphs (e.g. social networks, proteins etc.).

Product information, for example, can be represented by embeddings that encode the relationships of products to each other. These embeddings can then support recommendation systems. Graphs and their nodes can also be embedded in vector spaces, for example to represent social networks or other complex biological networks.

This concept is further extended by Multimodal AI to embed different data types in a single embedding space, where an application (e.g. text search + image search) can then compare and associate information of different formats (e.g. text descriptions of images).

Real-World Applications of Vector Embeddings

Vector embeddings are ubiquitous in many current applications of AI, because they allow for efficient comparison of complex information.

Search platforms: By making sense of input search queries in relation to larger pools of data (like search results, content or information), they return relevant results, even where the search terms do not appear exactly in the content being searched.

Enterprise knowledge systems: convert all the documents, policies, manuals etc. an organization has produced into embeddings and enable search by question, i.e. employees search for information by asking questions instead of searching for keywords.

Customer support systems: can identify relevant knowledge articles, like FAQs or help articles, and previous support issues, based on the meaning in a customer’s question. This information is then used to assist the customer interacting with the AI system, or to assist a human customer support agent handling the issue.

Recommendation Systems By analyzing relationships between products, contents, users, and other data entities, their vector representations help in deciding relevant information or products for a user.

Visual Search and Recognition Enables the search for similar images. Facilitates the recognition of images within text. Excellent for image discovery as well as for product search.

Generative AI applications can use embeddings to retrieve information from external sources, such as knowledge bases. For multimodal systems, embeddings can also be used to connect different types of content. In such a system, for example, images of food can be embedded in the same space as recipes.