73 Years Ago, Shannon Planted the Seed for Large Model Development

Did Shannon Predict the Future? Discover the Roots of LLMs

Tracing AI's roots back to its founding fathers.

Is the principle behind today's booming Large Language Models (LLM) proposed by Claude Shannon?

Today, Princeton University Professor Sebastian Seung shared this perspective:

In 1951, Claude Shannon, working at Bell Labs in Murray Hill, New Jersey, posed the problem of predicting the next word, laying the seed for current LLMs.

Many seemingly cutting-edge concepts were actually proposed decades ago, even in new fields like computer science. This notion and the mention of Murray Hill sparked discussions, with Turing Award winner and Meta Chief Scientist Yann LeCun noting that influential work originated from Murray Hill, Florham Park, and Princeton.

LeCun listed several notable research outcomes:

• Hopfield Networks (by Hopfield, who worked at both Bell Labs and Princeton University)

• ConvNets

• Boosting/Adaboost

• Non-negative Matrix Factorization

• Support Vector Machines (SVM) and Kernel Methods

• Structured Prediction

• Computational Learning Theory/VC Theory

So, how did Shannon contribute to today's path toward AGI with his early work?