ivory.core.tokenize

Class GalagoTokenizer

java.lang.Object

ivory.core.tokenize.Tokenizer

ivory.core.tokenize.GalagoTokenizer

public class GalagoTokenizer
extends Tokenizer

Constructor Summary

Constructors

Constructor and Description
GalagoTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration conf)
void	configure(Configuration mJobConf, FileSystem fs)
boolean	isStemming()
boolean	isStopWord(String word) Overrided by applicable implementing classes.
boolean	isStopwordRemoval()
static void	main(String[] args)
String[]	processContent(String text)

Methods inherited from class ivory.core.tokenize.Tokenizer

getNumberTokens, getOOVRate, getStem2NonStemMapping, getUTF8, getVocab, isDiscard, isDiscard, isStopWord, normalizeFrench, removeBorderStopWords, removeNonUnicodeChars, setVocab, stem

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GalagoTokenizer

public GalagoTokenizer()

Method Detail

configure

public void configure(Configuration conf)

Specified by:

configure in class Tokenizer

configure

public void configure(Configuration mJobConf,
             FileSystem fs)

Specified by:

configure in class Tokenizer

isStemming

public boolean isStemming()

Overrides:

isStemming in class Tokenizer

isStopWord

public boolean isStopWord(String word)

Description copied from class: Tokenizer

Overrided by applicable implementing classes.

Overrides:

isStopWord in class Tokenizer

Returns:

true if parameter is a stopword, false otherwise

isStopwordRemoval

public boolean isStopwordRemoval()

Overrides:

isStopwordRemoval in class Tokenizer

main

public static void main(String[] args)

processContent

public String[] processContent(String text)

Specified by:

processContent in class Tokenizer