Information Theory

Information theory and its relationship with probability, statistics, and data compression. Entropy. The Shannon-McMillan-Breiman theorem. Shannon's source coding theorems. Statistical inference; hypothesis testing; model selection; the minimum description length principle. Information-theoretic proofs of limit theorems in probability: Law of large numbers, central limit theorem, large deviations, Markov chain convergence, Poisson approximation, Hewitt-Savage 0-1 law. Prerequisites: APMA 2630 , 1710 .