ivory.lsh.eval

Class SampleTermDocVectors

java.lang.Object

org.apache.hadoop.conf.Configured

ivory.lsh.eval.SampleTermDocVectors

All Implemented Interfaces:

Configurable, Tool

public class SampleTermDocVectors
extends Configured
implements Tool

A program that samples from a collection of key,value pairs according to a given frequency.

• [input] path to the collection
• [output-dir] path of the output file containing sample
• [num-mappers] number of mappers to run
• [sample-frequency] if entered N, then every Nth pair is sampled. N=1 is equivalent to sampling everything.

User needs to modify the source file to change the key and value class type. Change input and output class type of the mapper, and modify the 3 static fields accordingly.

Here's a sample invocation:

 hadoop jar ivory.jar ivory.util.SampleDocVectors 
 /umd-lin/fture/pwsim/medline/wt-term-doc-vectors 
 /umd-lin/fture/pwsim/medline/wt-term-doc-vectors-sample 
 100
 

If there is a text file containing docnos to be sampled (one docno per line), this should be specified as the fifth and last argument. In this case, the sample frequency argument can be anything since it will be ignored.

usage: [input] [output-dir] [number-of-mappers] [sample-freq] ([sample-docnos-path])

Author:

ferhanture

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SampleTermDocVectors.MyReducer

Constructor Summary

Constructors

Constructor and Description
SampleTermDocVectors()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

SampleTermDocVectors

public SampleTermDocVectors()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface Tool

Throws:

Exception