efxtools.sample.ae
Class EFXRBTokenizer
java.lang.Object
org.apache.uima.analysis_component.AnalysisComponent_ImplBase
org.apache.uima.analysis_component.Annotator_ImplBase
org.apache.uima.analysis_component.JCasAnnotator_ImplBase
efxtools.filtering.FilteringAnalysisEngine
efxtools.sample.ae.EFXRBTokenizer
- All Implemented Interfaces:
- org.apache.uima.analysis_component.AnalysisComponent
public class EFXRBTokenizer
- extends FilteringAnalysisEngine
This analysis engine represents a basic rule-based tokenizer
which, in general, can be used to split any German text into tokens.
However, the rules used by the splitter are specified with German news
articles given as plain text in mind.
Namely, the essential currently used rules for classifying whether to
split the tokens at the current text position are:
- IF next character is whitespace
THEN split after current character
- IF next character is end of sentence
THEN split after current character
- IF current character is a letter
AND NOT next character is a letter or digit
AND NOT current character belongs to a URL
THEN
- IF NOT next character is hyphen or period
THEN split after current character
- IF next character is hyphen
AND NOT previous character is whitespace
THEN split after current character
- IF next character is period
AND NOT previous character is whitespace
AND NOT period belongs to an abbreviation
THEN split after current character
- IF current character is a digit
THEN
- IF next character is a comma
AND NOT next but one character is a digit
THEN split after current character
- IF next character is a period
AND next but one is end of sentence
THEN split after current character
- IF NOT next character is letter, digit, comma, or period
AND NOT current character belongs to a URL
THEN split after current character
- IF current character is a hyphen
AND NOT current character belongs to a URL
AND NOT current character is first character of token
THEN split after current character
- IF current character is a period
AND NOT next character is a letter or digit
AND NOT current character belongs to a URL
THEN split after current character
- IF NOT current character is a letter, digit, comma, or period
AND NOT current character belongs to a URL
THEN split after current character
This analysis engine is dependent from a sentence splitter, in that it needs
sentence annotations as input to work.
An evaluation on a sample of 10 varying news articles of medium length yielded
an accuracy of 98% for the tokenizer.
- Author:
- Author of EfXTools
|
Method Summary |
void |
initialize(org.apache.uima.UimaContext aContext)
|
| Methods inherited from class org.apache.uima.analysis_component.JCasAnnotator_ImplBase |
getRequiredCasInterface, process |
| Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase |
getCasInstancesRequired, hasNext, next |
| Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase |
batchProcessComplete, collectionProcessComplete, destroy, reconfigure |
| Methods inherited from class java.lang.Object |
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
PARAM_LEXICON_URL
public static final java.lang.String PARAM_LEXICON_URL
- The name of the parameter for the URL prefix lexicon
- See Also:
- Constant Field Values
PARAM_LEXICON_ABBREVIATIONS
public static final java.lang.String PARAM_LEXICON_ABBREVIATIONS
- The name of the parameter for the abbreviation lexicon
- See Also:
- Constant Field Values
EFXRBTokenizer
public EFXRBTokenizer()
initialize
public void initialize(org.apache.uima.UimaContext aContext)
throws org.apache.uima.resource.ResourceInitializationException
- Specified by:
initialize in interface org.apache.uima.analysis_component.AnalysisComponent- Overrides:
initialize in class org.apache.uima.analysis_component.AnalysisComponent_ImplBase
- Throws:
org.apache.uima.resource.ResourceInitializationException