What is a Small Language Model?
The characteristics and capabilities of Small Language Models (SLMs).
Size and Architecture
Small language models are typically smaller in size compared to larger language models. This can be measured in several ways, including:
- Number of parameters: SLMs usually have between 10 million to 100 million parameters, whereas larger models can have billions of parameters.
- Model size: SLMs are often represented as a smaller number of layers, fewer attention heads, and smaller hidden dimensions.
- Model architecture: SLMs may employ simpler architectures, such as a smaller number of transformer layers, fewer layers overall, or different types of transformer layers (e.g., smaller attention heads or fewer feed-forward layers).