Google BERT’s Successor After 6 Years: Faster, Smarter, Longer!

Discover ModernBERT: the faster, smarter, and longer-context successor to BERT, delivering state-of-the-art performance for encoding tasks without GenAI hype.

Truly Useful Mainstream Model

BERT was released in 2018, which, in AI’s timeline, feels like it was a millennium ago! Despite its age, BERT remains widely used to this day. In fact, it is currently the second most downloaded model on HuggingFace Hub, with over 68 million downloads per month.

The good news is that six years later, we finally have a replacement!

Recently, the new AI research lab Answer.AI, along with NVIDIA and others, introduced ModernBERT.

ModernBERT is a new series of models available in two variants: a base version with 139M parameters and a larger version with 395M. It significantly improves on BERT and similar models in both speed and accuracy.

The model incorporates numerous advancements in large language models (LLMs) over the past years, including updates to architecture and training processes.

In addition to being faster and more accurate, ModernBERT increases the context length to 8k tokens, compared to just 512 tokens for most encoders. It is also the first encoder-only model trained on a large amount of code data.