What Is a Foundation Model?

What Is a Foundation Model?

A foundation model is a type of AI model. A large amount of data, of varying types, has been used for training a foundation model. It is a model which can be used for a number of different tasks and therefore forms the basis for a number of applications.

The adapted model is then used for the intended application. Methods for adapting a model include fine-tuning, prompting and further training.

In terms of AI system development, the use of foundation models changes things in that as opposed to having to create a new model from scratch for each use case, developers can instead use a pre-existing model and ‘tune’ it to the needs of their specific application.

Since the input data can be of very different types, a foundation model can for example work with text, images, audio, video, and even code. The variety of tasks for which the model can be used also varies greatly.

Note, however, that large language models are an important category of foundation models, but by no means do they exhaust the space of foundation models. While the former are, by and large, limited to language, the latter can deal with a wide array of different data types and tasks.

How Does a Foundation Model Work?

The foundation model first goes through a large-scale training of a broad dataset. During the training, the model learns all the relationships, patterns, structures, etc. that exist in the data.

The training of a foundation model can follow different paths, depending on the chosen architecture of the model as well as the type of data that the model is supposed to process.

For language models, the pretraining can involve tasks to predict missing or subsequent tokens in sequences of text. For image models, the pretraining can involve tasks to predict outputs for images and to learn the visual patterns and relationships between objects in images.

After initial training of a model has been completed, the model can be adapted to specific tasks which require fine-tuning of the model that was already pre-trained.

For example, a general-purpose model can be fine-tuned for a company’s terminology or for document classification, for report summary or for assistance with software development.

The model can then be used for a number of different tasks, and the customization is what determines the task the model can complete.

Pretraining

The pretraining phase includes learning general patterns in large amounts of data.

This process involves the model going through enormous amounts of information and adjusting the internal parameters based on the training objective.

This stage requires a large amount of computing power and typically occurs before models are released for specific use cases.

Fine-Tuning

Fine-tuning uses a pretrained model and a specific dataset to adapt the model to specific tasks, such as for example domain-specific language understanding.

In order to further specialize a general language model in a certain domain or to improve its performance on specific tasks it can be fine-tuned with domain specific data.

Note that we are not re-building the whole model during fine-tuning. We are actually building on top of what the model has learned during pretraining.

Prompting

Prompting is providing a model with a prompt. A prompt is typically a piece of text that instructs the model on how to produce output.

A well designed prompt can instruct the model to produce the desired output, or even to behave in a desired manner.

Prompting can be much easier than fine-tuning and can result in very flexible applications without having to set up a whole new model.

Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) – connecting a foundation model to information sources.

Instead of depending on what it learned when it was trained, RAG retrieval-augmented generation systems retrieve relevant information from documents, databases and knowledge bases. The foundation model is then used to generate a response to a user query based on that retrieved information.

This can be particularly useful when you need to return information that is currently accurate, e.g. organizational information.

What Are the Main Characteristics of Foundation Models?

Several characteristics of Foundation Models can be distinguished from those of traditional task-specific AI models.

Broad Training

They are trained on large, diverse datasets as opposed to small datasets for specific tasks.

Reusability

A single foundation model can support multiple applications. Such a model can be used for different purposes after appropriate adaptation.

Transferability

The knowledge and patterns learned by a foundation model in its pretraining phase are also applicable to subsequent tasks.

Adaptability

Foundation models can be customized through prompting, fine-tuning, retrieval, or other techniques.

Multimodal Capabilities

Some foundation models are multimodal and can process different types of information, e.g. text, images, audio, and video.

The multimodal capability however can vary greatly from one foundation model to another and does not necessarily include all types of data (e.g. images, audio, video).

What Are the Types of Foundation Models?

The different types of foundation models (Language, Vision, Multimodal, Audio & Speech) that process different types of information and can support different types of tasks.

Language Foundation Models

These models focus on understanding and generating text.

For example, GPT-style large language models can write, summarize, answer questions, translate, code and more.

Vision Foundation Models

Vision models are trained to understand visual information.

Applications that use a foundation model for vision, for example image classification, object detection, image segmentation, visual search, etc.

Multimodal Foundation Models

Multimodal models work across more than one type of data.

A typical multimodal model will take in text and images, and then produce a text output. However, there are many more advanced models that can take in multiple types of data such as text, images, audio and video.

Audio and Speech Foundation Models

These models are trained to process speech and/or audio.

These types of models can support applications and services such as speech recognition, speech and language translation, automatic transcription of audio and video, audio classification and many voice related applications and services.

Code Foundation Models

Code-focused models are trained on programming languages and software-related data.

Foundation models for Code can assist with Code Generation, Code Explanation, Bug Detection, Code Completion and Code Transformation.

Where Are Foundation Models Used?

A large number of industries can use Foundation Models, because the basic model can be used for many different purposes.

Customer Service

Conversational assistants that answer questions and support service agents in customer interactions and beyond.

When connected to business systems, the conversational assistant can use customer information and knowledge sources relevant to the case at hand.

Healthcare

The model can assist with document processing, medical research, clinical information retrieval, summarization, etc. in the health care domain.

In the Healthcare industry, where outputs can impact decisions and therefore require validation by humans, these models can only be used with careful oversight and in compliance with privacy laws.

Finance

Financial models can be used for financial document analysis, customer support, report summarization, research assistance and information retrieval.

Sensitive financial data requires strong access controls and appropriate security measures.

Software Development

Code generation: assists developers in generating code, explains existing code and can even identify errors and generate relevant documentation.

Although the generated code can save a developer from lots of repetitive work, the code needs to be reviewed and tested as well by the developer.

Marketing and Content

For instance, in Marketing and Content, it can support content to be generated, to be summarized, translated, classified, support with research, and support with personalization.

Human review of generated content is still required to check for accuracy, for adherence to a brand’s tone and style, and to verify the factual content of generated material.

Enterprise Knowledge Management

Organizations often have information distributed across documents, databases, wikis, and internal applications.

A foundation model can be connected to an enterprise’s documents, databases, wikis, and internal applications to act as a natural-language interface to find and summarize information of relevance.

What Are the Benefits of Foundation Models?

Foundation models offer several advantages over building separate AI models for every task.

Faster AI Development

This means that instead of having to build a complete AI application from scratch, a developer can use a pre-trained foundation model as a starting point.

This can significantly reduce the work required to create an AI application.

Support for Multiple Applications

One foundation model can support multiple applications.

Support for Multiple Applications. One foundation model can serve as the underlying technology for several different applications. For example, a chatbot in one system and document analysis in another.

Reduced Need for Task-Specific Training

Applications do not always require a completely new model.

For particular use cases of applications Prompting, retrieval or fine-tuning of the Foundation Model can be sufficient.

Ability to Transfer Learned Knowledge

The model’s broad training enables it to apply the patterns that it learned to tasks that were not explicitly represented as separate applications during the model’s development.

Support for Generative AI

A Foundation model is typically the underlying technology to support applications of generative AI.

Support for Generative AI. The model generates text, images, code, audio, etc. depending on the design of the model and its training.

Foundation Model vs. Large Language Model

Foundation models and traditional task-specific models differ mainly in their purpose and flexibility. 

The main distinction is flexibility. A traditional model might be built specifically to classify fraudulent transactions, while a foundation model can serve as the starting point for many different AI applications.

What Are Some Examples of Foundation Models?

There are several models which could be classified as a Foundation Model. They include: GPT models (GPT-3 for example), Llama, Gemini, Claude, BERT, and many of the multimodal models such as vision.

For example, GPT models, Llama, Gemini, Claude, BERT, and many more models for vision, and multimodal models.

However, each model has its own set of capabilities, licensing, training and application support. It is best to read up on each individual model rather than assume that all foundation models are created equal.