Today, I’m sharing some thoughts on the differences between large and small models.
First, let's consider why Qwen2 is currently the most popular open-source model.
To be honest, compared to the detailed reports from DeepSeek, LLaMA, and MiniCPM, Qwen2's report feels a bit lacking, as it doesn't cover key technical details.
However, the comprehensive "all-in-one" package Qwen2 offers to the open-source community is something no lengthy report can match.
For LLM researchers, the value of a cluster of smaller LLMs, derived from the same tokenizer and 7T pretraining data, far exceeds that of Qwen2-72B itself!
Now, let's move forward with two key concepts:
Homologous small models: These are smaller-sized LLMs trained with the same tokenizer and data.
Small models: The focus here is on their size—they are fast to infer or are purely classification models, regardless of the training method. Examples include small-sized LLMs, BERT, RoBERTa, XGBoost, and LR.
