Discrete Stochastic Processes

Topics

• Introduction and probability review
• More review; the Bernoulli process
• Laws of large numbers, convergence
• Poisson (the perfect arrival process)
• Poisson combining and splitting
• From Poisson to Markov
• Finite-state Markov chains; the matrix approach
• Markov eigenvalues and eigenvectors
• Markov rewards and dynamic programming
• Renewals and the strong law of large numbers (SLLN)
• Renewals: strong law and rewards
• Renewal rewards, stopping trials, and Wald’s equality
• Little, M/G/1, ensemble averages
• The last renewal
• Renewals and countable state Markov
• Countable-state Markov chains
• Countable-state Markov chains and processes
• Countable-state Markov processes
• Markov processes and random walks
• Hypothesis testing and random walks
• Random walks and thresholds
• Martingales (plain, sub and super)
• Martingales: stopping and converging
• Putting it all together