Google Releases T5Gemma, Reigniting the Architecture War!
Google releases T5Gemma, reviving encoder-decoder architecture with superior performance over decoder-only models in AI reasoning tasks.
"AI Disruption" Publication 7100 Subscriptions 20% Discount Offer Link.
Since 2023, the battlefield of large models has been unified under the decoder-only architecture. From the GPT family to LLaMA, Gemma, Mistral, and then to Claude, Command-R, and the Yi series, almost all mainstream LLMs that can be named are uniformly "pure decoders" (decoder-only).
But today, Google has made a comeback with T5Gemma—
Not only has it restarted the encoder-decoder technical route, but it has also made it take off immediately with simple techniques, outperforming the original Gemma 2.
T5Gemma itself is based on the decoder-only Gemma 2 framework. Remarkably, through a simple "adaptation" conversion to an encoder-decoder architecture, T5Gemma achieved a performance leap in one fell swoop.
T5Gemma 9B-9B scored 9 points higher than the original Gemma 2 9B on GSM8K (mathematical reasoning) and 4 points higher on DROP (reading comprehension).
When further reducing the parameter count, the results became even more stunning! T5Gemma 2B-2B IT's MMLU score improved by nearly 12 points compared to Gemma 2 2B, with GSM8K accuracy surging to 70.7%.
T5Gemma is primarily aimed at text generation tasks, including question-answering systems, mathematical reasoning, reading comprehension, and more. The encoder-decoder architecture supports "unbalanced" configurations. For example, a 9B encoder paired with a 2B decoder can strike a perfect balance between quality and efficiency.