Information Theory: From Coding to Learning

3-0-9 units · Graduate

Introduces fundamentals of information theory and its applications to contemporary problems in statistics, machine learning, and computer science. A thorough study of information measures, including Fisher information, f-divergences, their convex duality, and variational characterizations. Covers information-theoretic treatment of inference, hypothesis testing and large deviations, universal compression, channel coding, lossy compression, and strong data-processing inequalities. Methods are applied to deriving PAC-Bayes bounds, GANs, and regret inequalities in machine learning, parametric and non-parametric estimation in statistics, communication complexity, and computation with noisy gates in computer science. Fast-paced journey through a recent textbook with the same title. For a communication-focused version, consider 6.7470 .