ivory.core.tokenize

Class LuceneAnalyzer

java.lang.Object

ivory.core.tokenize.Tokenizer

ivory.core.tokenize.LuceneAnalyzer

public class LuceneAnalyzer
extends Tokenizer

Constructor Summary

Constructors

Constructor and Description
LuceneAnalyzer()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration conf)
void	configure(Configuration conf, FileSystem fs)
float	getOOVRate(String text, VocabularyWritable vocab)
Map<String,String>	getStem2NonStemMapping(String text)
String[]	processContent(String text)
void	setLanguage(String l)
void	setLanguageAndStemmer(String l)
String	stem(String token)

Methods inherited from class ivory.core.tokenize.Tokenizer

getNumberTokens, getUTF8, getVocab, isDiscard, isDiscard, isStemming, isStopWord, isStopWord, isStopwordRemoval, main, normalizeFrench, removeBorderStopWords, removeNonUnicodeChars, setVocab

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LuceneAnalyzer

public LuceneAnalyzer()

Method Detail

configure

public void configure(Configuration conf)

Specified by:

configure in class Tokenizer

configure

public void configure(Configuration conf,
             FileSystem fs)

Specified by:

configure in class Tokenizer

getOOVRate

public float getOOVRate(String text,
               VocabularyWritable vocab)

Overrides:

getOOVRate in class Tokenizer

getStem2NonStemMapping

public Map<String,String> getStem2NonStemMapping(String text)

Overrides:

getStem2NonStemMapping in class Tokenizer

processContent

public String[] processContent(String text)

Specified by:

processContent in class Tokenizer

setLanguage

public void setLanguage(String l)

setLanguageAndStemmer

public void setLanguageAndStemmer(String l)

stem

public String stem(String token)

Overrides:

stem in class Tokenizer