The AI Compute Stack: Programs, Processors, Power Grids, and Policy

(3 Credit Hours) · Undergraduate

This course examines large-scale AI infrastructure through four interlocking lenses: - Programs — How modern AI workloads—especially large language models—actually compute. Through detailed case studies, we will analyze where the real bottlenecks arise: attention's memory bandwidth limits, KV cache growth, distributed training communication costs, long-context scaling, and inference-time compute tradeoffs. - Processors — How hardware and physics constrain and enable AI at scale. We will examine GPUs, TPUs, memory hierarchies, interconnects, and emerging accelerator designs, along with the physical realities of power density, rack design, cooling systems, and the limits imposed by thermodynamics and semiconductor fabrication. - Power Grids — The economics of AI infrastructure and the central role of electricity. We will study the cost structure of GPU clusters, cloud pricing models, capital expenditure versus operating expenditure, electricity as the dominant operating input, grid interconnection bottlenecks, and how energy markets shape where and how large-scale compute can be built. - Policy — The geopolitical, environmental, and regulatory forces that shape AI's physical footprint. We will analyze semiconductor supply chains, export controls, energy and water constraints, embodied carbon, permitting regimes, and the broader political economy of AI infrastructure. Students finish the course able to reason across all four layers simultaneously—connecting, for example, a transformer's KV cache memory bottleneck to HBM supply constraints, to rack-level power density limits, to electricity markets, to export controls and carbon accounting. PREREQUISITES: Data Structures and Computer Architecture