ivory.core.tokenize

Class OpenNLPTokenizer

java.lang.Object

ivory.core.tokenize.Tokenizer

ivory.core.tokenize.OpenNLPTokenizer

public class OpenNLPTokenizer
extends Tokenizer

Constructor Summary

Constructors

Constructor and Description
OpenNLPTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration conf)
void	configure(Configuration conf, FileSystem fs)
int	getNumberTokens(String string) Method to return number of tokens in text.
float	getOOVRate(String text, VocabularyWritable vocab)
Map<String,String>	getStem2NonStemMapping(String text)
String[]	processContent(String text)
void	setLanguage(String l)
void	setLanguageAndStemmer(String l)
void	setTokenizer(FileSystem fs, Path p)
String	stem(String token)

Methods inherited from class ivory.core.tokenize.Tokenizer

getUTF8, getVocab, isDiscard, isDiscard, isStemming, isStopWord, isStopWord, isStopwordRemoval, main, normalizeFrench, removeBorderStopWords, removeNonUnicodeChars, setVocab

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OpenNLPTokenizer

public OpenNLPTokenizer()

Method Detail

configure

public void configure(Configuration conf)

Specified by:

configure in class Tokenizer

configure

public void configure(Configuration conf,
             FileSystem fs)

Specified by:

configure in class Tokenizer

getNumberTokens

public int getNumberTokens(String string)

Description copied from class: Tokenizer

Method to return number of tokens in text. Subclasses may override for more efficient implementations.

Overrides:

getNumberTokens in class Tokenizer

Parameters:

string - text to be processed.

Returns:

number of tokens in text.

getOOVRate

public float getOOVRate(String text,
               VocabularyWritable vocab)

Overrides:

getOOVRate in class Tokenizer

getStem2NonStemMapping

public Map<String,String> getStem2NonStemMapping(String text)

Overrides:

getStem2NonStemMapping in class Tokenizer

processContent

public String[] processContent(String text)

Specified by:

processContent in class Tokenizer

setLanguage

public void setLanguage(String l)

setLanguageAndStemmer

public void setLanguageAndStemmer(String l)

setTokenizer

public void setTokenizer(FileSystem fs,
                Path p)

stem

public String stem(String token)

Overrides:

stem in class Tokenizer