Big Data Analytics

3 Credits · Level: Undergraduate

The growth of the Web and the improvements in data creation, collection, and use have lead to tremendous increase in the amount and complexity of the data that a search engine needs to handle. "Big data" presents challenges to search engines from three perspectives: bigger data volume, higher data complexity, and faster data change rate. The increase of the magnitude and complexity of the data has become a major drive for new data analytics algorithms and technologies that are scalable, highly interactive, and able to handle complex and dynamic information seeking tasks in the big data era. In this class, we will focus on text retrieval algorithms and programming based on Big Data. We will cover programming models that allow us to easily distribute computations across large computer clusters. In particular, we will teach Apache Spark, which is an open-source cluster computing framework that has soon become the state-of-the-art for big data programming. In contrast to Hadoop's MapReduce paradigm, Spark's in-memory primitives provide performance up to 100 times faster for certain applications. Spark provides clean API in JAVA, Scala, Python and R. This course will provide an introduction to Spark, focusing specifically on search engine design upon Spark and "thinking at scale".