Introduction to Natural Language Processing

Text representation is a modelling decision

A model operates on encoded units rather than directly on meaning. Word tokenization is intuitive but struggles with unseen words and morphology; subword tokenization reuses smaller units but can split comparable text unevenly across languages. Normalize text only when the task permits it: removing case can erase useful information for names or acronyms. Keep document boundaries and train-test splits explicit so near-duplicate passages do not leak into evaluation.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Embeddings encode useful relationships

An embedding maps a discrete symbol into a learned vector. Similar contexts can produce nearby vectors, supporting generalization beyond exact word matches. Static embeddings give a word one vector, while contextual models assign representations that depend on its sentence. Cosine similarity measures vector direction rather than magnitude, but a high similarity is not proof of synonymy or truth. Representations also inherit patterns and biases from the data and objective used to learn them.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Language modelling and cross-entropy

An autoregressive language model factors sequence probability into conditional next-token probabilities. Training usually minimizes the negative log probability of observed tokens. Perplexity exponentiates average negative log probability when using natural logarithms; lower values indicate better predictive likelihood on comparable data. Perplexities are not directly comparable across different tokenizers without careful normalization. Predicting plausible text is also different from verifying factual claims or completing a task safely and accurately.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Attention, positions and causal masking

Attention compares a query with keys, normalizes the resulting scores and combines corresponding value vectors. In scaled dot-product attention, dividing by the square root of key dimension helps control score magnitude. Positional information is necessary because the attention operation alone does not encode sequence order. A causal mask blocks future tokens during next-token prediction. If the target token is accidentally visible during training, the model can appear excellent while learning a task unavailable at generation time.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Translation and question answering

Translation maps one sequence into another and must account for meaning, word order and alternatives rather than replacing words independently. Question answering may select a span from supplied context or generate an answer. Define which evidence the model may use and whether it must abstain when that evidence is insufficient. A fluent answer can be unsupported; evaluating only surface overlap can also penalize a correct paraphrase. Use metrics that match the task and inspect errors.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Pretraining, adaptation and retrieval

Pretraining learns broad statistical structure from a large corpus, while adaptation changes behaviour for a domain or task. Retrieval can supply relevant external passages at inference time, but the generator still needs to use them correctly. Keep retrieved context separate from trusted instructions, track passage provenance and test whether the answer follows the evidence. Training data, retrieval indexes and evaluation datasets can overlap; audit that overlap before attributing improvement to actual generalization.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Worked probability and evaluation checks

If a model assigns probabilities 0.5 and 0.25 to two observed target tokens, their joint conditional probability is 0.125. Average negative log probability is -(ln 0.5 + ln 0.25)/2, so perplexity is the square root of 8, approximately 2.83. This score says nothing by itself about factual reliability. For a practical system, evaluate held-out examples by domain, language and failure type, including impossible questions and examples requiring evidence rather than memorized phrasing.

References: Carleton 4115 official calendar record · Jurafsky and Martin: Speech and Language Processing, public draft

Practice & worked answers

Original exercises for this guide.

Public notes & references

Course sources establish the scope above. Reference materials support the explanations; linked lecture notes and books remain with their authors.

• Carleton 4115 official calendar record ↗Course scope · 2026–27
• Jurafsky and Martin: Speech and Language Processing, public draft ↗Supplementary primary reference