Data-Intensive Information Processing Applications (Spring 2010)

What's the course about?

This course is about scalable approaches to processing large amounts of information (terabytes and even petabytes). We focus mostly on MapReduce, which is presently the most accessible and practical means of computing at this scale, but will discuss other approaches as well.

MapReduce is a programming model for expressing distributed computations on massive amounts of information and an execution framework for large-scale information processing on clusters of commodity servers. It was originally developed by Google and built on well-known principles in parallel and distributed processing dating back several decades. MapReduce has since enjoyed widespread adoption via an open-source implementation called Hadoop, whose development was led by Yahoo.

This is the "version 3.0" of a course that was previously offered in Spring 2008 and Fall 2008. Core content is similar, but organization and scope have been overhauled for the third time.

Course readings will be drawn from research papers available online, as well as the following two textbooks:

• Tom White, Hadoop: The Definitive Guide, O'Reilly, 2009. (The full text of this book is available to UMD students via Safari Books Online, but you may choose to purchase a paper copy for handy reference)
• Jimmy Lin and Chris Dyer, Data-Intensive Text Processing with MapReduce, forthcoming, 2010.

In addition to the standard distributions of Hadoop, this course uses Cloud⁹, a MapReduce library for Hadoop developed at the University of Maryland.

Pre-requisites

No previous experience with MapReduce or parallel and distributed programming is necessary. However, students taking this course should be competent Java programmers, since concepts taught in class will be reinforced through extensive programming exercises in the Hadoop implementation of MapReduce (which is in Java). Note that this is a course on algorithms and "thinking at scale"—not about Hadoop programming. Therefore, we expect you to "pick up" the details of the Hadoop API without explicit instruction from us. Of course, we will assist you by providing resources and a reasonable amount of guidance.

In addition, students are assumed to have knowledge of basic probability and statistics (e.g., axioms of probability, Bayes' Theorem, relative frequency estimation, etc.) and also a solid understanding of basic computer architecture (e.g., microprocessor architectures, memory hierarchies, cache coherence protocols, etc.).

Course Grade

• Exams (50%): There will be a midterm exam (25%) and a final exam (25%). CS MS comp grade (software engineering area) will be based entirely on the average of the two exams.
• Final project (30%): There will be group final project (~3 students) on a topic of your own choosing. More details on this will be discussed later on in the semester.
• Class assignments (15%): There will be six assignments, but we will drop your lowest grade and take the average of five assignment grades. Each assignment grade is worth 3% of your course grade.
• Class participation (5%): Showing up for class, demonstrating preparedness (i.e., doing the readings), and contributing to class discussions.

See syllabus for assignment due dates and exams dates.

Homework Assignments

The homework assignments are designed to help you learn the material, and will not primarily be used as an evaluative tool. As a result, they are not worth a large percentage of your final course grade, relative to the effort that should be devoted to them.

Assignments will be graded on a five-point scale, described below.

• Score = 0: no credit (e.g., you failed to turn in the assignment).
• Score = 1: serious deficiencies (i.e., you missed significant portions of the assignment or a significant number of the answers were incorrect.
• Score = 2: adequate completion (i.e., most of the answers were correct, but there were some missing or incorrect answers).
• Score = 3: satisfactory completion (i.e., everything or nearly everything was correct).
• Score = 4: impressive mastery (i.e., you went above and beyond what was asked for in the assignment).

A score of 4 will only be given out in extraordinary cases. A score of 3 indicates "A" quality work.

Policy for Late and Incomplete Work

• Assignment deadlines: Assignments are due at the beginning of class unless otherwise noted.
• Late assignments: If an assignment is late by up to 24 hours, the grade will be reduced by 20%. By 48 hours, 40%. And so forth. Exceptions can be discussed in cases of medical excuses, family emergencies, etc., but being busy is not a valid excuse. Exceptions will only be considered prior to the deadline, and the sooner you talk to us about a problem the more likely we will be sympathetic. There are several common problems that we are unlikely to consider as valid reasons for failing to get work in on time. These include (a) failure to manage your time properly, (b) discovering an assignment is harder or takes longer than you expected it to be (see item a), (c) having competing priorities such as other classes, job interviews, etc., and (d) losing code or data that should have been backed up, unless it's someone else's fault.
• 'Incomplete' as a grade: We will not issue an 'incomplete' as a grade except for serious, valid reasons, generally in the category of emergencies. See above for some reasons unlikely to be considered valid. If you are having problems of any kind, please talk to us as soon as possible.

Academic Integrity

The University of Maryland, College Park has a nationally recognized Code of Academic Integrity, administered by the Student Honor Council. This Code sets standards for academic integrity at Maryland for all undergraduate and graduate students. As a student you are responsible for upholding these standards for this course. It is very important for you to be aware of the consequences of cheating, fabrication, facilitation, and plagiarism. Please visit the Code of Academic Integrity or the Student Honor Council for more information.

Students with Disabilities

Students with disabilities needing academic accommodation should: (1) register with and provide documentation to the Disability Support Services office, and (2) discuss any necessary academic accommodation with me. This should be done at the beginning of the semester, within the first three class sessions.

Emergency Preparedness

Information about the status of the campus is available at the campus emergency preparedness website. If the campus is closed, please make sure to stay safe. Information about possible rescheduling of course activities will be provided via e-mail once the campus has reopened.

Back to main page

This page, first created: 07 Jan 2010; last updated: