efxtools.sample.ae
Class EFXRBTokenizer

java.lang.Object
  extended by org.apache.uima.analysis_component.AnalysisComponent_ImplBase
      extended by org.apache.uima.analysis_component.Annotator_ImplBase
          extended by org.apache.uima.analysis_component.JCasAnnotator_ImplBase
              extended by efxtools.filtering.FilteringAnalysisEngine
                  extended by efxtools.sample.ae.EFXRBTokenizer
All Implemented Interfaces:
org.apache.uima.analysis_component.AnalysisComponent

public class EFXRBTokenizer
extends FilteringAnalysisEngine

This analysis engine represents a basic rule-based tokenizer which, in general, can be used to split any German text into tokens. However, the rules used by the splitter are specified with German news articles given as plain text in mind.

Namely, the essential currently used rules for classifying whether to split the tokens at the current text position are:

  1. IF next character is whitespace THEN split after current character
  2. IF next character is end of sentence THEN split after current character
  3. IF current character is a letter AND NOT next character is a letter or digit AND NOT current character belongs to a URL THEN
    1. IF NOT next character is hyphen or period THEN split after current character
    2. IF next character is hyphen AND NOT previous character is whitespace THEN split after current character
    3. IF next character is period AND NOT previous character is whitespace AND NOT period belongs to an abbreviation THEN split after current character
  4. IF current character is a digit THEN
    1. IF next character is a comma AND NOT next but one character is a digit THEN split after current character
    2. IF next character is a period AND next but one is end of sentence THEN split after current character
    3. IF NOT next character is letter, digit, comma, or period AND NOT current character belongs to a URL THEN split after current character
  5. IF current character is a hyphen AND NOT current character belongs to a URL AND NOT current character is first character of token THEN split after current character
  6. IF current character is a period AND NOT next character is a letter or digit AND NOT current character belongs to a URL THEN split after current character
  7. IF NOT current character is a letter, digit, comma, or period AND NOT current character belongs to a URL THEN split after current character

This analysis engine is dependent from a sentence splitter, in that it needs sentence annotations as input to work.

An evaluation on a sample of 10 varying news articles of medium length yielded an accuracy of 98% for the tokenizer.

Author:
Author of EfXTools

Field Summary
static java.lang.String PARAM_LEXICON_ABBREVIATIONS
          The name of the parameter for the abbreviation lexicon
static java.lang.String PARAM_LEXICON_URL
          The name of the parameter for the URL prefix lexicon
 
Constructor Summary
EFXRBTokenizer()
           
 
Method Summary
 void initialize(org.apache.uima.UimaContext aContext)
           
 
Methods inherited from class efxtools.filtering.FilteringAnalysisEngine
process, setResultSpecification
 
Methods inherited from class org.apache.uima.analysis_component.JCasAnnotator_ImplBase
getRequiredCasInterface, process
 
Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase
getCasInstancesRequired, hasNext, next
 
Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase
batchProcessComplete, collectionProcessComplete, destroy, reconfigure
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

PARAM_LEXICON_URL

public static final java.lang.String PARAM_LEXICON_URL
The name of the parameter for the URL prefix lexicon

See Also:
Constant Field Values

PARAM_LEXICON_ABBREVIATIONS

public static final java.lang.String PARAM_LEXICON_ABBREVIATIONS
The name of the parameter for the abbreviation lexicon

See Also:
Constant Field Values
Constructor Detail

EFXRBTokenizer

public EFXRBTokenizer()
Method Detail

initialize

public void initialize(org.apache.uima.UimaContext aContext)
                throws org.apache.uima.resource.ResourceInitializationException
Specified by:
initialize in interface org.apache.uima.analysis_component.AnalysisComponent
Overrides:
initialize in class org.apache.uima.analysis_component.AnalysisComponent_ImplBase
Throws:
org.apache.uima.resource.ResourceInitializationException