everything.study
COMP 5801

Topics in Machine Learning

0.5 credit · Graduate

Study notes

Independent, AI-drafted study guide · Public sources linked below · Added November 5, 2026

Topics in Machine Learning — Multi-agent Aspects of Deep Reinforcement Learning

Fall 2026 COMP 5801G offering alignment; this guide is specifically about MARL, cooperation, communication and strategic learning. Other COMP 5801 offerings may have entirely different content.

From one learning agent to a joint interaction model

A stochastic game extends an MDP to several agents whose joint action affects transitions and rewards. Agents may have different observations and objectives. A cooperative partially observed setting often requires decentralized policies using only each agent's local history at execution. Specify whether rewards are shared, whether actions are simultaneous and what information is available during training. These modelling choices determine the actual coordination problem before a neural architecture is selected.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Why independent learning changes the environment

From one agent's perspective, another learner's changing policy alters the distribution of outcomes for the same local action. Experience collected earlier may therefore reflect a different effective environment. Independent value learners can struggle even when each would work in a stationary single-agent task. Replay buffers, exploration and policy updates must be interpreted with this nonstationarity in mind. A training failure is not automatically solved by a larger model or a longer replay history.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Solution concepts describe different goals

A best response maximizes one agent's payoff against fixed opponent policies. A Nash equilibrium permits no profitable unilateral deviation, but it need not maximize group welfare or be unique. In cooperative games, several equally good conventions can still be difficult to coordinate on. Evaluate which solution concept fits the task instead of calling every stable behaviour optimal. For mixed-motive tasks, report individual outcomes and exploitability where appropriate, rather than only an average team score.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Centralized training and decentralized execution

Centralized training may use joint observations or actions to estimate credit and reduce uncertainty, while deployed actors rely only on locally available information. A centralized critic can support decentralized actor updates, but it must not leak unavailable information into the actors' execution inputs. Value-decomposition methods impose structure on a joint value function, creating useful scalability and coordination trade-offs. State these representational restrictions when interpreting a result on tasks requiring complex joint decisions.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Communication is an action with consequences

Learned communication gives agents a channel that can affect teammates' decisions. Define channel capacity, timing, noise, cost and who can observe each message. A code that works among one trained team may fail with independently trained partners because its meanings are convention-dependent. Check whether messages add task-relevant information by ablating or perturbing them. High message entropy alone does not establish useful communication, and correlated messages can arise without causal benefit.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Federation, manipulation and machine teaching

Federated learning coordinates model updates across data holders, while MARL coordinates sequential decisions among interacting agents; the ideas can overlap but solve different problems. Strategic learners may shape another agent's future behaviour, creating incentives beyond immediate reward. Distinguish influence that improves cooperative learning from exploitative manipulation against a different objective. A research comparison should specify who adapts, whether gradients through other learners are assumed and what information about those learners is actually available.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Worked coordination game and research evaluation

Two robots choose Left or Right. Both receive 1 when their choices match and 0 otherwise. Always-Left and Always-Right are both coordinated pure equilibria; mixing two independently trained teams can yield mismatched conventions despite excellent self-play. Evaluate self-play, cross-play with new partners and robustness to observation or channel changes. For a seminar paper, separate its claimed contribution from its baseline choices, computational budget, seed variation and assumptions about centralized training information.

References: Fall 2026 COMP 5801G offering alignment · Albrecht, Christianos and Schäfer: Multi-Agent Reinforcement Learning

Practice & worked answers

Original exercises for this guide.

1. Why can a perfect self-play team perform poorly with a new partner?

Show worked answer

Its agents may rely on a private convention learned together. A new partner can use a different equally successful convention, so self-play success alone does not establish transferable coordination.

2. Does a centralized critic permit a decentralized actor to use global state at deployment?

Show worked answer

No. Training information and execution information are separate. The actor must obey the observation limits of the deployed problem even when the critic receives global state during learning.

Public notes & references

Course sources establish the scope above. Reference materials support the explanations; linked lecture notes and books remain with their authors.

Fall 2026 · Multi-agent Aspects of Deep Reinforcement Learning

Topics

  • Reinforcement learning fundamentals
  • Games, interaction models and solution concepts
  • Foundational multi-agent learning algorithms
  • Deep learning and deep reinforcement learning
  • Cooperative learning and federated learning
  • Learning to communicate and emergent communication
  • Strategy manipulation and machine teaching
  • Research paper seminars and a multi-agent learning project

Readings

2026

Topics

  • Recurrent Neural Networks (RNNs): Architecture and applications to sequential data
  • Attention Mechanisms and Self-Attention: Concepts and applications in transformers
  • Sequence-to-Sequence (S2S) Models: Encoder-decoder architectures for tasks like translation
  • Transformers: Introduction and training of transformers for NLP and other tasks
  • BERT and GPT: Understanding bidirectional and autoregressive LLMs
  • Reinforcement Learning from Human Feedback (RLHF): Alignment techniques in LLMs
  • Variational Autoencoders (VAEs): Dimensionality reduction and generative modeling
  • Generative Adversarial Networks (GANs): Adversarial training techniques
  • Diffusion Models (DDPM): Denoising diffusion probabilistic models for generative tasks
  • Ethics and Governance: Bias, fairness, and regulatory frameworks
  • Emerging Research: Multimodal models, autoformalization, and future directions
  • Advanced/Optional Topics (Subject to Available Time)
  • Efficiency in Models: Knowledge distillation, efficient transformers, and sparsity methods
Sources