Reading List

These papers, books, and essays serve as a shared basis of knowledge for the MIT Probabilistic Computing Project.

Non-Technical Background (Required)

  • The Structure of Scientific Revolutions by Thomas Kuhn. [Amazon]
  • The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business by Clayton Christensen. [Amazon]
  • Epigrams on Programming by Alan Perlis. [html] [Turing award page][Foreword to SICP]
  • The Effective Executive: The Definitive Guide to Getting the Right Things Done by Peter F. Drucker.[Amazon]
  • Technology and Courage by Ivan Sutherland. [pdf]
  • Mathematical Writing by Don Knuth. [pdf]
  • You and Your Research by Hamming [html]
  • Whirlwind I Computer [Wikipedia]
  • Worse Is Better by Richard Gabriel [html]
  • Who Inspired the Leamer-Rosenthal Prizes? Part II by Ed Leamer [html]
  • The Laws of Medicine: Field Notes From an Uncertain Science by Siddhartha Mukherjee [Interview][Amazon]
  • The War of Art: Winning the Inner Creative Battle by Steven Pressfield [Amazon]
  • Study of Telemedicine Finds Misdiagnoses of Skin Problems by Melinda Beck [html]
  • Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results by Silberzahn et al. [pdf]
  • What Is It Like To Be A Bat? by Thomas Nagel [pdf]
  • When U.S. air force discovered the flaw of averages [html]
  • The origins of SageMath – creating a viable open source alternative to Magma, Maple, Mathematica, and Matlab [pdf]
  • Politics and the English Language by George Orwell [html]
  • The Sciences of the Artificial by Herbert Simon [Amazon]
  • Building Robust Systems an essay by Gerald Jay Sussman [pdf]

Technical Background (Required)

  • Mathematics: Its Content, Methods and Meaning: Chapter 11. The Theory of Probability by A. N. Kolmogorov. [Amazon]
  • Structure and Interpretation of Computer Programs by Harold Abelson, Gerald Jay Sussman, and Julie Sussman. [Amazon] [pdf]
  • Classifier Technology and the Illusion of Progress by David J. Hand. [pdf]
  • Machine Learning: The High-Interest Credit Card of Technical Debt by Sculley et al. [pdf]
  • (Skim) On Computable Numbers and the Entscheidungsproblem by Alan Turing. [pdf]
  • (Skim) Noncomputable Conditional Distributions by Ackerman, Freer, and Roy [pdf]
  • Natively Probabilistic Computation; PhD Dissertation by Vikash Mansinghka, 2009 [pdf]
  • Design of LISP-Based Processors by Steele and Sussman, 1979 [pdf]
  • Lisp: A Language for Stratified Design by Abelson and Sussman, 1987 [pdf]
  • Steps Toward Artificial Intelligence by Minsky, 1961 [pdf]
  • On Chomsky and the Two Cultures of Statistical Learning by Peter Norvig [html]
  • Statistical Modeling: The Two Cultures by Leo Breiman [pdf]
  • Critique of Paper by “Deep Learning Conspiracy” (Nature 521 p 436) by Jürgen Schmidhuber [html]
  • Pages 3-31 (Introducton) from Artificial Intelligence A Modern Approach by Russell and Norvig [pdf]
  • Confessions of a Used Programming Language Salesman by Erik Meijer [pdf]

Probabilistic Programming (Required)

  • VentureScript Tutorial. [html]
  • BayesDB Documentation. [html]
  • Picture: an imperative probabilistic programming language for scene perception. Kulkarni, T., Kohli, P., Tenenbaum, J., Mansinghka, V. [Under review at CVPR.] [[webpage]]43
  • Bayesian Statistics Without Tears: A Sampling-Resampling Perspective by Smith and Gelfand. [pdf]
  • When are probabilistic programs probably computationally tractable? Freer, C., Mansinghka, V., Roy, D. [NIPS Workshop on Monte Carlo Methods for Modern Applications.], 2010. [pdf]
  • Bayesian Optimization as a Probabilistic Meta-Program; MEng thesis by Ben Zinberg, 2015 [pdf]

Probabilistic Programming (Optional, Outdated)

  • Venture: a general purpose probabilistic programming platform with efficient stochastic inference. Mansinghka, V., Radul, A. [pre­print available as arXiv, number 1404:0099] [[pdf]]47
  • A stochastic programming perspective on nonparametric Bayes. Roy, Mansinghka, Goodman and Tenenbaum. [ICML Nonparametric Bayes Workshop.], 2008. [Oral] [[pdf]]48

Foundations of Probabilistic Modeling (Required)

  • Bayesianism and Causality, or, Why I am Only a Half-Bayesian by Judea Pearl. [pdf]
  • Why I Am Not a Bayesian by Clark Glymour. [pdf]
  • Chapters 1-3 of Probability Theory: The Logic of Science by E. T. Jaynes. [pdf] [Amazon]
  • Chapter 28 (Model Comparison and Occam’s Razor) from Information Theory, Inference, and Learning Algorithms by David MacKay. [webpage]

Standard Library of Modeling Techniques (Required)

  • Chapter 45 (Gaussian Processes) from Information Theory, Inference, and Learning Algorithms by David MacKay. [webpage]
  • Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process by Griffiths, Canini, Sanborn, and Navarro. [pdf]
  • Discovering latent classes in relational data by C. Kemp, T. L. Griffiths, and J. B. Tenenbaum. [pdf]
  • Topics in Semantic Representation by Griffiths and Steyvers. [pdf]
  • Finding Scientific Topics by Griffiths and Steyvers. [pdf]
  • Continuous Time Dynamic Topic Models by Wang, Blei, and Heckerman. [pdf]
  • Practical Bayesian Optimization of Machine Learning Algorithms by Snoek, Larochelle, and Adams.[webpage]
  • A Unifying Review of Linear Gaussian Models by Roweis and Ghahramani. [pdf]

Standard Library of Inference Techniques (Required)

  • Composable Probabilistic Inference with Blaise by Keith A. Bonawitz. [pdf]
  • Chapters 5-8 of Venture: a general purpose probabilistic programming platform with efficient stochastic inference. Mansinghka, V., Radul, A. [pre­print available as arXiv, number 1404:0099] [[pdf]]47
  • An Introduction to MCMC for Machine Learning by Andrieu, De Freitas, Doucet, and Jordan. [pdf]
  • The Markov Chain Monte Carlo Revolution by Persi Diaconis. [pdf]
  • Convergence of Sequential Monte Carlo-based Sampling Methods by Huggins and Roy. [pdf]

Useful Examples (Optional)

  • MrBayes: Bayesian Inference of Phylogenetic Trees by Huelsenbeck and Ronquist. [pdf] [webpage]
  • FastSLAM: FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem. [pdf]
  • First few chapters of Bayesian Spectrum Analysis and Parameter Estimation by Bretthorst. [pdf]
  • Recovering Intrinsic Images from a Single Image by Tappen, Freeman, and Adelson. [pdf]
  • Bayesian Inference for PCFGs via Markov chain Monte Carlo by Johnson, Griffiths and Goldwater. [pdf]
  • Global Seismic Monitoring as Probabilistic Inference by Arora, Russell, Kidwell, and Sudderth. [pdf]

Automatic Model Discovery (Optional)

  • CrossCat: A Fully Bayesian Nonparametric Method for Analyzing Heterogeneous, High Dimensional Data by Mansinghka et al. [Journal of Machine Learning Research, accepted pending revision, available internally]
  • Structure Discovery in Nonparametric Regression through Compositional Kernel Search by Duvenaud, Lloyd, Grosse, Tenenbaum, Ghahramani. [pdf]
  • Exploiting compositionality to explore a large space of model structures by Grosse et al. [pdf]
  • Model Selection in Compositional Spaces by Grosse. PhD Thesis. [pdf]
  • The discovery of structural form by Kemp and Tenenbaum. [pdf]
  • Structured priors for structure learning by Mansinghka, Kemp, Tenenbaum, and Griffiths [pdf]
  • Chapter 28 (Model Comparison and Occam’s Razor) from Information Theory, Inference, and Learning Algorithms by David MacKay. [webpage]

Reference Material on Approximate Inference (Optional)

  • Probabilistic Inference using Markov chain Monte Carlo Methods by Neal. [pdf]
  • Hamiltonian Monte Carlo tutorial by Neal. [pdf]
  • Exact and approximate inference via systematic stochastic search by Mansinghka et al. [pdf]
  • Factor graphs and the sum-product algorithm by Kschischang, Frey, and Loeliger. [pdf]
  • An Architecture for Parallel Topic Models by Smola and Narayanamurthy. [pdf]
  • LeCun, Bottou, Orr and Muller: Efficient BackProp, in Orr, and Muller (Eds), Neural Networks: Tricks of the trade. [pdf]
  • Black Box Variational Inference by Ranganath, Gerrish and Blei. [pdf]

Selecting Actions (Optional)

  • Towards common-sense reasoning via conditional simulation: legacies of Turing in Artificial Intelligence by Freer, Roy, Tenenbaum [pdf]
  • Rational Metareasoning and Compilation for Optimizing Decisions Under Bounded Resources by Eric Horvitz [pdf]