DeepSeek’s New-Year Paper: A New Architecture Era
DeepSeek’s mHC paper unlocks stable, scalable multi-stream Transformers with 6.7% overhead and big gains
“AI Disruption” Publication 8500 Subscriptions 20% Discount Offer Link.
On the first day of the new year, DeepSeek published a new paper proposing an architecture called mHC (Manifold-Constrained Hyper-Connections).
The research aims to address the instability issues of traditional hyper-connections in large-scale model training while maintaining their significant performance gains.
Simply put, DeepSeek’s proposed mHC extends the single residual stream of traditional Transformers into a multi-stream parallel architecture and uses the Sinkhorn-Knopp algorithm to constrain the connection matrix on a doubly stochastic matrix manifold, successfully solving the numerical instability and signal explosion problems caused by hyper-connections (HC) breaking the identity mapping property during large-scale training.
The paper has three first authors: Zhenda Xie, Yixuan Wei, and Huanqi Cao. Notably, DeepSeek founder & CEO Liang Wenfeng is also listed among the authors.
Traditional residual connections (the x + F(x) structure in Transformers) guarantee lossless signal transmission and training stability through “identity mapping.” However, their bottleneck lies in the information channel width being limited by the hidden layer dimension C.
Recently, research represented by Hyper-Connections (HC) proposed by ByteDance’s Seed team has expanded the residual connection paradigm widely applied over the past decade by extending residual stream width and diversifying connection patterns.
Although these methods brought significant performance improvements, they also introduced two serious problems:
