Theory of Markov Processes: Selected Topics with Applications in ML and GenAI

12 units · Graduate

Intermittent: 12 units Markov processes are a fundamental mathematical concept with broad applications, including emerging fields such as reinforcement learning and diffusion models. This course is structured into two parts. Part I covers the core theory of Markov processes, including discrete-time and continuous-time Markov chains, as well as Markov processes with continuous state space such as diffusion processes. Part II builds on the core theory and covers selected topics in the theoretical foundation of reinforcement learning and diffusion models in generative AI. Key topics: - Discrete-state Markov processes: ergodicity, Lyapunov drift analysis - Continuous-state Markov processes: Brownian motion, Markov semigroups, Itô integral, stochastic differential equations, Langevin diffusions - Mixing time analysis: coupling method, Poincar and #233; inequality, log Sobolev inequalities - Reinforcement learning: Bellman equation, value iteration, policy iteration, Q-learning, TD-learning, policy gradient - Diffusion models: sampling algorithms, DDPM, DDIM, flow matching, convergence analysis, discrete diffusion models