ivory.core.tokenize

Class LuceneArabicAnalyzer

java.lang.Object

ivory.core.tokenize.Tokenizer

ivory.core.tokenize.LuceneArabicAnalyzer

public class LuceneArabicAnalyzer
extends Tokenizer

Constructor Summary

Constructors

Constructor and Description
LuceneArabicAnalyzer()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration conf)
void	configure(Configuration conf, FileSystem fs)
float	getOOVRate(String text, VocabularyWritable vocab)
String[]	processContent(String text)
String	stem(String token)

Methods inherited from class ivory.core.tokenize.Tokenizer

getNumberTokens, getStem2NonStemMapping, getUTF8, getVocab, isDiscard, isDiscard, isStemming, isStopWord, isStopWord, isStopwordRemoval, main, normalizeFrench, removeBorderStopWords, removeNonUnicodeChars, setVocab

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LuceneArabicAnalyzer

public LuceneArabicAnalyzer()

Method Detail

configure

public void configure(Configuration conf)

Specified by:

configure in class Tokenizer

configure

public void configure(Configuration conf,
             FileSystem fs)

Specified by:

configure in class Tokenizer

getOOVRate

public float getOOVRate(String text,
               VocabularyWritable vocab)

Overrides:

getOOVRate in class Tokenizer

processContent

public String[] processContent(String text)

Specified by:

processContent in class Tokenizer

stem

public String stem(String token)

Overrides:

stem in class Tokenizer