What Is a Small Language Model (SLM)?

What Is a Small Language Model (SLM)?

Small Language Models, SLMs for short, are language models. That is to say they are AI programs designed to process human language in some way (to understand it or to generate it, for example) and that differ from the Large Language Models, LLMs for short that we described earlier in terms of the number of parameters that the two types of models have. While LLMs have hundreds of billions of parameters (to give some idea of the scale), SLMs have just a few million (to give some other idea of the scale).

These smaller models can be millions or even a few billion parameters in size, versus the hundreds of billions of parameters in some of the very largest Large Language Models.

This in no way means that smaller models are less useful. In fact, there are a lot of scenarios in which focused models are much more effective than larger, more general models. Many use cases, environments, or even business requirements demand special tailored solutions that, although smaller in scope, offer more than sufficient power for the specific tasks at hand.

These smaller models require less memory and also are more suitable for mobile, edge or local applications on desktops and servers running on premises. In addition, these models are private by design, meaning all processing and data generation occurs on a user’s own devices and in the user’s control.

I used the word ‘key’ here as it seems the core message of the entire document was lost in the detail.

How Does a Small Language Model Work?

All Small Language Models are based on the same core principles to process and generate human language. Most models today are based on the neural network architecture of the transformer model for natural language processing.

Small Language Models process human language, whether entered by a user or read from a file, one token or word at a time. Each token in turn is represented numerically by the model to process.

The transformer’s attention mechanism helps in identifying relationships between all tokens in a given input. It considers a context when processing an input, and then it generates an appropriate output.

The model will generate an output based on patterns in the input it was trained on.

The main difference between a Small Language Model and a Large Language Model is the scale. As described above, a Small Language Model contains fewer parameters than a Large Language Model. Consequently, the Small Language Model requires less memory and it is more efficient for inference.

How Are Small Language Models Created?

These small models can be trained from scratch to form a very compact model or even compressed from a larger model (this is called model compression).

Reduce the model size, and keep as much performance as possible.

Pruning

Pruning: This approach to compressing a neural network removes parameters that have the least impact on the model’s performance, for example, the weights of certain neurons, whole neurons or even parts of a neural network.

Fine-tuning is typically required after a model has been pruned in order to regain the lost accuracy. However, too much pruning can result in a model with reduced performance.

Quantization

Quantization reduces a model’s memory usage by quantizing its floating-point numbers to integers with lower bit-widths (e.g., 8-bit integers instead of 32-bit floating-point numbers).

Low numerical precision, such as 8-bit integers instead of 32-bit floating-point numbers, for model values, like input and weights. This reduces the memory for a model and can even increase the speed for inference. This can be done after training of a model (so-called post-training quantization) or even during training (so-called quantization-aware training).

Knowledge Distillation

Knowledge Distillation. This method transfers knowledge from a larger Teacher Model to a smaller Student Model.

The student learns from the teacher model’s output and behavior, and this leads to compact SLMs that, even though they are much smaller than their teacher model, still manage to retain the relevant knowledge and be as accurate or even more accurate.

Low-Rank Factorization

Low-Rank Factorization. Factorizing large weight matrices into low rank factorized forms. This can reduce the number of parameters as well as required number of computations to produce model outputs. Note that subsequent fine-tuning may be required to restore original model performance.

The approach can also create a smaller model that still has acceptable performance and requires fine-tuning.

What Are the Main Characteristics of an SLM?

Several characteristics distinguish SLMs from larger language models.

Smaller model size is the most obvious difference. Fewer parameters generally mean lower memory and computing requirements.

Lower latency is another important characteristic. With less computation involved, an SLM can often produce results quickly.

Deployment flexibility also matters. Smaller models are easier to deploy on local machines, mobile devices, edge hardware, or private infrastructure.

SLMs can also be task-specific. Instead of attempting to understand every possible topic, a model can be adapted for a particular domain or application.

This combination of size, speed, and specialization makes SLMs useful when an AI application has clear and manageable requirements.

Where Are Small Language Models Used?

Small Language Models are used in applications where efficient language processing is more important than maximum model scale.

Mobile Applications

For mobile applications the small language model is run locally on the mobile device. The local running is possible because of the limited memory, the low processing power and the low battery capacity of the mobile device.

Text Assistance, Summarization, Translation and Voice related functionality can all be powered by SLMs on a mobile device.

Local processing can also reduce dependence on a continuous internet connection.

Customer Service

SLMs can be used to power customer service chatbots, to classify customer requests, to create summaries of conversations with customers and to handle a large volume of routine customer inquiries.

This allows the required amount of processing power to be used for the specific support task that is required, without the costs of sending every request to a larger language model.

Document Summarization

SLMs can summarize emails, meeting discussions, transcripts, reports, and other text.

This is particularly useful when summarization needs to happen locally, quickly, or within an application with limited computing resources.

Sentiment Analysis

Businesses can use SLMs to classify customer feedback and identify sentiment in large volumes of text.

A model can be trained or adapted to recognize patterns associated with positive, negative, or other predefined categories.

Code Assistance

Code Assistance for Small Language Models. Some small language models can generate, explain, summarize or translate code.

In addition, for focused developer tools, a small language model can also be used to generate, explain, summarize, or even translate code.

Translation

Translation – The Multilingual SLM can also translate between languages. Its small size makes it perfect for local translation.

Predictive Maintenance

SLMs can be deployed alongside edge and IoT systems to process information from connected equipment.

Local AI processing of information from connected equipment can identify patterns in the data and support the maintenance decision. It does not have to send all the information to a central system.

What Are the Benefits of Small Language Models?

Smaller models like SLMs are very efficient. They allow for useful language capabilities without the need for as much infrastructure as larger models.

Lower Computing Requirements

SLMs generally require less memory and processing power. Therefore, they can run on hardware that would not be able to support larger models.

Faster Response Times

A smaller computational workload results in faster inference and hence in faster responses, which are very important for interactive applications and for real-time systems in general.

Lower Operating Costs

Lower Operating Costs. Smaller models can save organizations money on computing resources to train, run, and make predictions with their models.

Easier Local Deployment

An SLM can be deployed on local devices or private infrastructure when the model and application requirements allow it.

This gives organizations more flexibility in deciding where AI processing takes place.

Greater Data Control

Deploy on local devices or private infrastructure, when necessary for applications involving sensitive business or customer information.

Model size does not automatically guarantee security. There is still a need for appropriate access control and data protection in the surrounding application.

Lower Energy Requirements

Smaller models require fewer computational resources for the inference process. Therefore, SLMs require less energy during inference than larger models.

Finally, another very practical benefit of SLMs: Lower Energy Requirements for Running Them.

Easier Customization

The small size of an SLM can be an advantage as these models are very easy to adjust to a specific task or domain and are therefore very suitable for organizations that only need focused AI.

What Are the Limitations of Small Language Models?

Another important consideration is the limitations involved in using SLMs. These models have a small size, which is very beneficial for organizations that need focused AI solutions for specific tasks or domains.

Limited General Knowledge

A small model does not have the same amount of general knowledge as a large language model. A small model is perfect for a certain domain but not for completely different topics.

Performance on Complex Tasks

Some tasks like for example broad knowledge, complex reasoning or even longer sequences of steps can be problematic for smaller models.

Note that even smaller models can handle more complex tasks than larger models, depending on the training data, architecture, etc.

Hallucinations

The model can generate lots of information that sounds perfect but actually is incorrect.

For tasks where accuracy is important, SLMs can output lots of information that sounds correct but is in fact incorrect. Therefore, SLMs need to be validated against reliable information sources in order to ‘ground’ generated responses.

Bias

A Small Language Model can also contain biases from its training data or from the other models that it has been trained on in order to develop into the SLM.

Testing would need to be done for relevant user groups, languages, tasks and input types, and validated against reliable information sources to “ground” the SLM’s responses.

Small Language Model vs. Large Language Model

As explained earlier, Small Language Models and Large Language Models perform similar language tasks but differ in terms of scale, computing resources, and typical use cases.

The important point is that SLM and LLM are not competing choices in every situation. Both can be useful within the same AI system.

SLM vs. LLM: Which One Should You Choose?

Small Language Models are great for clearly defined tasks and want low latency, low resources or even local processing.

For example: sentiment classification, document summarization, simple customer inquiries etc. are not necessarily to be processed with very large language models.

LLMs are the better option for applications that require a broad knowledge of topics, that involve flexible and/or long conversations, and that require complex levels of reasoning.

Select the model that fits the application’s needs, not solely based on the size of the model. There are various factors that organizations can consider when choosing a model, including accuracy, latency, cost, privacy, hardware, integration requirements, and expected workload.

How Do SLMs and LLMs Work Together?

SLMs and LLMs can be used in a hybrid architecture to improve performance for complex tasks. For instance, a simple classification request could remain with an SLM. A complex research question could then be routed to an LLM. Intelligent routing between the different models is another option. This involves a routing system that determines which model is the best to handle a given request.

For example, simple classification of requests (e.g. sentiment of a customer message) can be served by an SLM. A complex research question can then be routed to an LLM.

This way expensive computing resources are not used for every request. However, the organization still has access to stronger models, if required.

What Are Some Examples of Small Language Models?

There are several families of models, including some compact language models and smaller variants to the normal models.

DistilBERT is a smaller BERT model that has been created using knowledge distillation.

Gemma also contains compact language models to deploy efficiently.

Llama also has smaller models, to use when more resource efficient use of the AI is required.

Ministral: a library of compact models. The models in this library have been designed to be as efficient as possible and to run as fast as possible during inference.

Phi is a family of small language models created by Microsoft for applications that need compact AI models.

Other models are emerging. The capabilities of these models are as yet unknown.