Neural Networks

Layers, parameters and nonlinear composition

A dense layer transforms an input vector using a matrix and bias. A network alternates such transformations with nonlinear activations. Without nonlinearities, any stack of affine layers collapses into one affine transformation, limiting what it can represent. Track tensor dimensions explicitly: a batch of n examples with d features multiplied by a d-by-h weight matrix produces n-by-h activations. Shape checks detect many errors before expensive training hides them inside a large model.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Loss functions and the meaning of predictions

Choose a loss matched to the output interpretation. Squared error is a natural starting point for real-valued regression; cross-entropy fits probabilistic classification. Logits are unconstrained scores, whereas normalized probabilities sum to one. Numerically stable fused loss implementations avoid separately exponentiating very large scores. Never use test-set performance as a training objective or a repeated hyperparameter-selection signal, because that leaks information into the reported evaluation.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Backpropagation is the chain rule organized

The forward pass builds intermediate values; reverse-mode differentiation propagates how the loss changes with each intermediate value. For y = w x and loss (y - t)^2 / 2, the derivative with respect to w is (y - t) x. Shared parameters receive the sum of gradient contributions from every use. Automatic differentiation performs this bookkeeping, but a small finite-difference check on a tiny model remains useful when implementing a custom differentiable operation.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Optimization and diagnosing training failures

Mini-batch gradient descent estimates a gradient from a subset of examples and steps opposite it. Learning rate controls step size and is not interchangeable with batch size or model capacity. If loss does not decrease, first verify labels, loss reduction, gradient flow and parameter updates on a very small batch. A model that cannot deliberately overfit a handful of examples may have an implementation problem rather than insufficient training data.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Capacity, regularization and data splits

Training performance measures fit to observed examples; generalization concerns new examples from the target setting. Weight decay, dropout, augmentation and early stopping impose different constraints and are not universal remedies. Split related examples carefully: frames from one video or records from one person can leak information across a naive random split. Compare training and validation curves, preserve a final test set and report uncertainty where data or random initialization substantially changes the result.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Convolution, recurrence and attention

Convolutions reuse local filters across positions, encoding locality and parameter sharing. Recurrent networks maintain state across a sequence, while attention combines information using query-dependent weights. These are structural choices about information flow, not a ranking in which the newest architecture always wins. Padding and masking are part of the computation: an attention model for next-token prediction must not see future target tokens, and a padded sequence should not treat padding as meaningful evidence.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Worked gradient and a reliable evaluation loop

For x = 3, w = 2 and target t = 4, the prediction is 6 and the half-squared loss is 2. The weight gradient is (6 - 4) × 3 = 6. With learning rate 0.1, update w to 1.4; the next prediction is 4.2 and the loss is 0.02. This example separates gradient calculation from parameter updating. In a real model, use the appropriate training or evaluation mode and avoid retaining gradients during pure evaluation.

References: Winter 2026 public outline alignment · Dive into Deep Learning: free author-maintained textbook

Practice & worked answers

Original exercises for this guide.

Public notes & references

Course sources establish the scope above. Reference materials support the explanations; linked lecture notes and books remain with their authors.

• Winter 2026 public outline alignment ↗Course scope
• Dive into Deep Learning: free author-maintained textbook ↗Supplementary primary reference