Cloud⁹

The ClueWeb09 collection consists of one billion web pages (5 TB compressed, 25 TB uncompressed), in ten languages, crawled in January and February 2009. Its creation, supported by U.S. National Science Foundation (NSF), was led by Jamie Callan of the Language Technologies Institute at Carnegie Mellon University to support research on information retrieval and related human language technologies. The entire collection is available for research purposes. This guide provides instructions on processing the English portion of the Clue Web collection using Hadoop with Cloud⁹, building on code developed by Mark Hoy at CMU. For now, this page deal exclusively with the English portion of the collection.

Topics that this page covers:

• Counting the records
• Repacking the records
• Sequentially-numbered docnos
• Malformed records

See the "official" dataset information page for the definitive description of the collection. Some content on this page simply mirrors information on that page for convenience.

In total, there are 503,903,810 pages in the English portion of the ClueWeb09 collection (2.08 TB compressed, 13.4 TB uncompressed). The English data is distributed in ten parts (called segments), each corresponding to a directory. Here are the page counts for each segment:

ClueWeb09_English_1    50,220,423 pages
ClueWeb09_English_2    51,577,077 pages
ClueWeb09_English_3    50,547,493 pages
ClueWeb09_English_4    52,311,060 pages
ClueWeb09_English_5    50,756,858 pages
ClueWeb09_English_6    50,559,093 pages
ClueWeb09_English_7    52,472,358 pages
ClueWeb09_English_8    49,545,346 pages
ClueWeb09_English_9    50,738,874 pages
ClueWeb09_English_10   45,175,228 pages

Each segment contains a number of sub-directories, each of which contains a number of compressed WARC files. There is no official name for these sub-directories, so to be precise I'll call them sections. They are numbered from en0000 all the way through en0133. For example, en0000 to en0011 belong to segment 1. That is, the first segment contains the following sub-directories:

ClueWeb09_English_1/en0000
ClueWeb09_English_1/en0001
...
ClueWeb09_English_1/en0011

In addition, the first segment contains three special sections, enwp00, enwp01, enwp02, enwp03. These sections hold a version of English Wikipedia. Also, sections en0083, en0084, and en0097 do not exist (or are empty, depending on the distribution).

Once you've gotten your hands on the collection, the first thing you might want to do is run some sanity checks. The simplest sanity check is to rea through all the records and count them. This functionality is provided by a demo program in Cloud⁹. Here's the command-line invocation:

hadoop jar cloud9.jar edu.umd.cloud9.collection.clue.DemoCountClueWarcRecords \
original /shared/ClueWeb09/collection.raw 1 /shared/ClueWeb09/docno-mapping.dat

The first command-line argument indicates whether your counting records in the original distribution ("original") or repacked SequenceFiles ("repacked"); for the second condition, see details below. The second argument is the base path of your ClueWeb09 distribution. The third command-line argument is the segment number (1 through 10). The final argument is the location of the docno mapping (see details below). If you run the demo program on all ten segments, you should get the following results:

Description of the columns:

• segment: the segment in question.
• # files: number files (compressed WARC files) in the segment.
• bytes off disk: compressed size of the compressed WARC files.
• # of records: number of WARC records.
• # of pages: number of actual HTML pages.
• total size: uncompressed size of all records.

As a note, each compressed WARC file has a header followed by the actual HTML pages, so number of records should be equal to number of files plus number of pages.

In some ways, the original WARC files are awkward to work with. There is, for example, no simple way to quickly access an individual record that lies in the middle of a gzipped file. A good solution is to repack the collection into block-compressed SequenceFiles. This is described in a separate page on providing random access to the WARC records.

Many information retrieval and other text processing tasks require that all documents in the collection be sequentially numbered, from 1 ... n. Typically, you'll want to start with document 1 as opposed to 0 because it is not possible to represent 0 with many standard compression schemes used in information retrieval (i.e., Golomb codes). For clarity, I call these sequentially-numbered document ids docnos, whereas I call the original ids docids. (This is a bit confusing as in previous TREC collections alphanumeric document ids are tagged as DOCNOs.)

The format of a docid (WARC-TREC-ID) in the collection is clueweb09-enXXXX-YY-ZZZZZ. Due to this regular format, it is very easy to algorithmically map between docnos and docids. In Cloud⁹, the ClueWarcDocnoMapping class in the edu.umd.cloud9.collection.clue package provides an API for you.

Even if you don't want to use the Cloud⁹ API, this mappings data file should be useful. Here are the first few lines:

en0000,0,35582,1
en0000,1,28413,35583
en0000,2,36053,63996
en0000,3,36260,100049
en0000,4,34786,136309
en0000,5,33015,171095
...

It's a CSV data file, where the first column represents the enXXXX portion of the docid, the second column represents the YY portion, and the third column lists the number of pages with the enXXXX-YY prefix. Since ZZZZZ starts at zero, the last docid is the third column minus one. The fourth column tracks the cumulative count in the number of documents. With this information, mapping between docnos and docids is a simple matter of arithmetic.

There are a number of malformed WARC records in the English portion of the collection (there may be malformed records in the other languages also, but I haven't analyzed them yet). The most prevalent problem is an extra newline in the WARC header. There are a few cases of other malformed headers also. See this list of docids: each docid refers to a WARC record that immediately precedes a malformed WARC record. For example, the first docid in the list is clueweb09-en0001-41-14941, which means that clueweb09-en0001-41-14942 is malformed.

Of all the malformed WARC records referenced in the file above, all except for the following are malformed in having an extra newline in the WARC header. The following docids are malformed in other ways (all cases of garbled URL):

clueweb09-en0044-01-04501
clueweb09-en0059-46-06368
clueweb09-en0117-48-12547
clueweb09-en0112-59-06118
clueweb09-en0126-33-37391
clueweb09-en0126-88-10049

This errata is provided primarily for reference. The API in Cloud⁹ transparently handles these malformed WARC records (thanks to code originally written by Mark Hoy from CMU).