Cloud9: A Hadoop toolkit for working with big data

Cloud⁹ is a collection of Hadoop tools that tries to make working with big data a bit easier.

Warning

As of December 2015, this library is no longer being actively developed or maintained. Please see Bespin for its replacement. Why? The Cloud⁹ codebase dates back to 2007 and has accumulated a lot of cruft; it's time to start over with a blank slate.

This software was designed with two goals in mind: First, to serve as a teaching tool for MapReduce and MapReduce algorithm design. Second, to provide a collection of useful tools on which to build other "big data" systems. Here are just a few features:

API for working with various text collections, including Wikipedia, TREC document collections for information retrieval research, and the ClueWeb09 web crawl.
Reference implementations of a few common MapReduce algorithm, including PageRank, bread-first search, co-occurrence matrix computation.
Implementations of various useful Hadoop data types, including primitive collections, via the lintools-datatypes package available here.

Getting Started

Exercises

Reference Implementations

Cloud⁹ provides reference implementations of many design patterns and algorithms introduced in the book Data-Intensive Text Processing with MapReduce by Lin and Dyer. Some of these examples are also solutions to exercises included with the library, which have been previously used in MapReduce courses at the University of Maryland.

Order inversion design pattern for computing relative frequencies (Chapter 3)
Pairs and stripes design pattern for computing term co-occurrences (Chapter 3)
Inverted indexing is covered in this exercise (Chapter 4)
Parallel breadth-first search (Chapter 5)
PageRank and design patterns for efficient graph algorithms (partially covered in Chapter 5)