Information Theory and Generative Modeling

3.0 credits · Graduate

A graduate-level introduction to information theory, data compression, and generative modeling. An introduction to information measures: entropy, mutual information, relative entropy, differential entropy, and their properties. Lossless compression and its connection to prediction and generative modeling. The Minimum Description Length (MDL) principle in model selection. Practical lossless compression using arithmetic coding. The rate-distortion theorem and its connection to lossy compression standards such as JPEG, mp3, and AAC as well as generative modeling techniques such as autoencoders and variational inference. The Nonlinear Transform Coding framework. Practical methods for lossy compression such as Trellis-Coded Quantization (TCQ) and entropy-constrained dithered quantization. ECE 7620 is intended for students intending to undertake Ph.D.-level research in information theory, statistics, or machine learning. Knowledge of probability and linear algebra at the advanced undergraduate level is required.