MDL Reading

There is a large body of literature on the Minimum Description Length principle in the contexts of statistics, mathematics, machine learning, philosophy, etc. We give only a small selection of publications that we have found especially useful and important. More publications can be easily found using search engines such as Google and CiteSeer.

Articles

A.Barron, J.Rissanen, and B.Yu, The minimum description length principle in coding and modeling. IEEE Trans. Information Theory, vol. 44 (1998), no. 6, pp. 2743-2760.

Q.Gao, M.Li, and P.M.B.Vitanyi, Applying MDL to learning best model granularity, Artificial Intelligence, vol. 121 (2000), no. 1-2, pp. 1-29. (available at Prof. Vitányi's homepage)

P.Grünwald, P.Kontkanen, P.Myllymäki, T.Silander, and H.Tirri, Minimum encoding approaches for predictive modeling. Proc. 14th Int. Conf. on Uncertainty in AI (UAI'98), G.Cooper and S.Moral (eds.), 1998, pp. 183-192. (available at CoSCo homepage)

M.H.Hansen, and B.Yu, Model selection and the principle of minimum description length. J. American Statistical Association, vol. 96 (2001), pp. 746-774. (available at Dr. Hansen's homepage)

A.D.Lanterman, Schwarz, Wallace, and Rissanen: Intertwining themes in theories of model order estimation. International Statistical Review, vol. 69 (2001), no. 2, pp. 185-212. (available at CiteSeer)

I.J.Myung, V.Balasubramanian, and M.A.Pitt. Counting probability distributions: Differential geometry and model selection. Proc. National Academy of Sciences, USA, vol. 97 (2000), pp. 11170-11175. (available at Prof. Balasubramanian's homepage)

J.Rissanen, Modeling by shortest data description. Automatica, vol. 14 (1978), pp. 465-471.

J.Rissanen, Universal coding, information, prediction, and estimation, IEEE Trans. Information Theory, vol. 30 (1984), pp. 629-636.

J.Rissanen, Stochastic complexity. J. Royal Statistical Society, Series B, vol. 49 (1987), no. 3, pp. 223-239.

J.Rissanen, Stochastic complexity and modeling. Annals of Statistics, vol. 14 (1986), pp. 1080-1100.

J.Rissanen, Fisher information and stochastic complexity. IEEE Trans. Information Theory, vol. 42 (1996), pp. 40-47.

J.Rissanen, Hypothesis selection and testing by the MDL principle. The Computer Journal, vol. 42 (1999), no. 4, pp. 260-269. (available at Computer Journal)

J.Rissanen, MDL Denoising. IEEE Trans. Information Theory, vol. 46 (2000), no. 7, pp. 2537-2543.

J.Rissanen, Strong optimality of the normalized ML models as universal codes and information in data. IEEE Trans. Information Theory, vol. 47 (2001), no. 5.

J.Rissanen, Complexity of simple nonlogarithmic loss functions. To appear in IEEE Trans. Information Theory, 2003.

N.Vereshchagin, and P.M.B.Vitanyi, Kolmogorov's structure functions with an application to the foundations of model selection, Proc. 47th IEEE Symp. Found. Comput. Sci. (FOCS'02), 2002. (available at Prof. Vitanyi's homepage)

P.M.B.Vitanyi, and M.Li, Minimum description length induction, Bayesianism, and Kolmogorov complexity. IEEE Trans. Information Theory, vol. 47 (2000), pp. 446-464. (available at Prof. Vitányi's homepage)

K.Yamanishi, A Decision-theoretic extension of stochastic complexity and its applications to learning. IEEE Trans. Information Theory, vol. 44 (1998), pp. 1424-1439.

Books

Peter Grünwald, The Minimum Description Length Principle and Reasoning under Uncertainty, Ph.D. Thesis, ILLC Dissertation Series DS 1998-03, CWI, the Netherlands, 1998. (available at Dr. Grünwald's homepage)

Jorma Rissanen, Stochastic Complexity in Statistical Inquiry, World Scientific, 1989.

Lecture Material

On-line video: Jorma Rissanen, MDL theory as a foundation for statistical modeling. MSRI Workshop on Information Theory, Mathematical Sciences Research Institute, Berkeley, February-March 2002. (available at MSRI)

Lecture notes: Jorma Rissanen, Lectures on statistical modeling theory, July 2002.

Presentation slides: Peter Grünwald, Tutorial on modern MDL, NIPS 2001 Workshop on MDL: Developments in Theory and New Applications, Whistler, Canada, December, 2001. (available at NIPS 2001)

Presentation slides: Henry Tirri, On Minimum description length modeling, Three Concepts: Information, Dept. of Computer Science, University of Helsinki, January-May, 2002.

Journals

Computer Journal (Special Issue on Kolmogorov Complexity)
IEEE Transactions on Information Theory (also at IEEE Xplore)
Journal of the Royal Statistical Society: Series B

Conferences and Workshops

DIMACS Workshop on Complexity and Inference, June 2003.
IEEE Information Theory Workshop 2002, October 2002.
MSRI Workshop on Information Theory, February-March 2002.
Neural Information Processing Systems (NIPS).
    - NIPS 2001 Workshop on MDL: Developments in Theory and New Applications, December 2001.
    - NIPS 2002 Workshop on Universal Learning Algorithms and Optimal Search, December 2002.
Uncertainty in Artificial Intelligence (UAI).