Ilya Sutskever's Top 30
‘If you really learn all of these, you’ll know 90% of what matters today."
"I rather expected @ilyasut to have made a public post by now after all the discussion of the AI reading list he gave me. A canonical list of references from a leading figure would be appreciated by many. I would be curious myself about what he would add from the last three years." - John Carmack (Feb 2023)
Tap to Preview

Framing Intelligence and Complexity

  • The First Law of Complexodynamics

  • Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton

    • Recurrent Foundations of Sequence Modeling

  • The Unreasonable Effectiveness of Recurrent Neural Networks

  • Understanding LSTM Networks

  • Recurrent Neural Network Regularization

  • Relational Recurrent Neural Networks

    • Neural Memory, Alignment and Structured Sequence Learning

  • Pointer Networks

  • Order Matters: Sequence to Sequences for Sets

  • Neural Machine Translation by Jointly Learning to Align and Translate

  • Neural Turing Machines

    • Attention, Transformers and Scale

  • The Annotated Transformer

  • Attention Is All You Need

  • Scaling Laws for Neural Language Models

    • Deep Vision and Representation Hierarchies

  • ImageNet Classification with Deep Convolutional Neural Networks

  • Deep Residual Learning for Image Recognition

  • Multi-Scale Context Aggregation by Dilated Convolutions

  • Identity Mapings in Deep Residual Networks

    • Relational And Graph-Based Reasoning

  • Neural Message Passing for Quantum Chemistry

  • A Simple Neural Network Module for Relational Reasoning

    • Generative Learning, Speech and Systems Scaling

  • GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

  • Variational Lossy Autoencoder

  • Deep Speech 2: End-to-End Speech Recognition in English and Mandarin