Intelligent Web-based Information Systems

From a document collection to a searchable system

A web information system collects content, represents it, retrieves candidates and ranks results for a user need. The pipeline must handle duplicate documents, failed fetches and updates as well as language processing. Treat a URL, a fetched representation and a canonical document as related but distinct objects. Preserve timestamps and provenance so users can understand whether a result reflects the current page or a historical crawl.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Inverted indexes and efficient retrieval

An inverted index maps a term to the documents containing it. For a Boolean AND query, intersect posting lists; process a shorter list first to reduce candidate work. Positional information supports phrase queries and proximity ranking. Token normalization improves matching but can also merge distinctions useful to the task. Index design therefore balances storage, update costs and the query operations the product actually needs, rather than storing every available feature indiscriminately.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Ranking needs an explicit relevance definition

Raw term frequency overweights repetitive or long documents. Weighting methods use features such as term rarity and document length to produce more useful scores. Link analysis supplies another signal, but a popular page is not automatically relevant to a particular query. Keep candidate retrieval separate from final ranking so each stage can be measured. A relevance judgement should reflect the user's task, including freshness and authority where those matter.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Recommendation and feedback bias

Content-based recommenders compare item attributes with a user's observed interests. Collaborative methods infer preferences from interaction patterns among users and items. Missing feedback does not mean dislike: an item may never have been shown. Recommendation logs are shaped by the previous recommender, producing exposure bias. Separate training and testing in time where appropriate, compare against simple popularity baselines and evaluate whether the system provides useful diversity instead of repeatedly amplifying already visible items.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Link analysis, influence and social structure

Model documents or users as vertices and relationships as edges, preserving direction and weight where meaningful. Degree counts local connections, whereas paths and centrality measures capture different structural roles. Influence is causal, while network correlation is observational; connected users may behave similarly because of shared interests rather than persuasion. A link-ranking algorithm can propagate importance, but its output depends on graph construction, damping assumptions and how disconnected or manipulated regions are handled.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Trust, reputation and adversarial data

A reputation score aggregates observations under assumptions about identity, incentives and evidence quality. A system vulnerable to cheap account creation may allow one actor to manufacture apparent agreement. Weighting by historical reliability can help but can also disadvantage new participants. Design mechanisms for contesting errors and separating verified outcomes from subjective ratings. Audit whether feedback is missing unevenly across groups, because the absence of a rating is not neutral evidence of poor quality.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Worked precision and recall trade-off

Suppose a query has ten relevant documents in the collection. A system returns five results, of which four are relevant. Precision is 4/5 = 0.8 and recall is 4/10 = 0.4. Expanding the result set can improve recall while reducing precision. For recommendation or search, also consider ordering: placing the useful item first is usually better than burying it at rank fifty, even when both systems retrieve the same set.

References: Carleton 4601 official calendar record · Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page)

Practice & worked answers

Original exercises for this guide.

Public notes & references

Course sources establish the scope above. Reference materials support the explanations; linked lecture notes and books remain with their authors.

• Carleton 4601 official calendar record ↗Course scope · 2026–27
• Manning, Raghavan and Schütze: Introduction to Information Retrieval (coauthor university page) ↗Supplementary primary reference