AI Disruption

AI Disruption

Share this post

AI Disruption
AI Disruption
73 Years Ago, Shannon Planted the Seed for Large Model Development

73 Years Ago, Shannon Planted the Seed for Large Model Development

Did Shannon Predict the Future? Discover the Roots of LLMs

Meng Li's avatar
Meng Li
Jul 06, 2024
∙ Paid

Share this post

AI Disruption
AI Disruption
73 Years Ago, Shannon Planted the Seed for Large Model Development
1
Share

Tracing AI's roots back to its founding fathers.

Is the principle behind today's booming Large Language Models (LLM) proposed by Claude Shannon?

Today, Princeton University Professor Sebastian Seung shared this perspective:

In 1951, Claude Shannon, working at Bell Labs in Murray Hill, New Jersey, posed the problem of predicting the next word, laying the seed for current LLMs.

Many seemingly cutting-edge concepts were actually proposed decades ago, even in new fields like computer science. This notion and the mention of Murray Hill sparked discussions, with Turing Award winner and Meta Chief Scientist Yann LeCun noting that influential work originated from Murray Hill, Florham Park, and Princeton.

LeCun listed several notable research outcomes:

  • Hopfield Networks (by Hopfield, who worked at both Bell Labs and Princeton University)

  • ConvNets

  • Boosting/Adaboost

  • Non-negative Matrix Factorization

  • Support Vector Machines (SVM) and Kernel Methods

  • Structured Prediction

  • Computational Learning Theory/VC Theory

So, how did Shannon contribute to today's path toward AGI with his early work?

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Meng Li
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share