Tokenizing with Lucene | LucidBox

Tokenizing with Lucene | LucidBox
A really simple way of tokenizing without using Lucene would be to just split the String using a RegEx. Here is a naive one-liner:

String tokens[] = testString.split("\\W");
The regex here is “\W” which is the negated word character class. This might work well enough for what you need, but you lose positional information and might not yield expected results in other languages. 

A really simple way of tokenizing without using Lucene would be to just split the String using a RegEx. Here is a naive one-liner:String tokens[] = testString.split("\\W");


The regex here is “\W” which is the negated word character class. This might work well enough for what you need, but you lose positional information and might not yield expected results in other languages.

Lucene’s Filters and Tokenizers (which extend TokenStream) store attributes for each token depending on their functionality. From the StandardTokenizer we get attributes which contain the token itself, the token’s type and positional information. So we need to call getAttribute with the class of the Attribute we’re interested in, CharTermAttribute is the one StandardTokenizer uses to contain the actual token text, etc…See the javadoc ongetAttribute for additional usage details.

The calls to tokenizer’s reset(), end() and close() are part of the workflow prescribed by TokenStream’s javadoc, which contains additional helpful documentation.

CharTermAttribute charTermAttrib = tokenizer.getAttribute(CharTermAttribute.class);
TypeAttribute typeAtt = tokenizer.getAttribute(TypeAttribute.class);
OffsetAttribute offset = tokenizer.getAttribute(OffsetAttribute.class);

List<String> tokens = new ArrayList<String>();
tokenizer.reset();
while (tokenizer.incrementToken()) {
  tokens.add(charTermAttrib.toString());
  System.out.println(typeAtt.toString() + " " + offset.toString() +      ": " + charTermAttrib.toString());
}
tokenizer.end();
tokenizer.close();

In the above code, we get the attributes once and then loop through tokenizer’s tokens printing out the attributes for each token. Since these attributes came from the StandardTokenizer, which again is a TokenStream, you should consider that these attributes are actually attributes of the stream of tokens and hence is tied to the state of tokenizer. So when we change the state of the tokenizer by calling incrementToken() the values of the attributes change as well without requiring us to manually call getAttribute again.

We can note that the sentence delimiters are gone, the offsets properly skip over them, and the casing of the terms is preserved.

Suppose you really want those sentence delimiters to appear as tokens? Well, there are some other tokenizers we can try. Change tokenizer to a WhitespaceTokenizer to give it a shot.

Tokenizer tokenizer = new WhitespaceTokenizer(Version.LUCENE_36, reader);

If you run the code now, you’ll get an IllegalArgumentException. Oops! This happens when we try to get the TypeAttribute because WhitespaceTokenizer doesn’t supply any types in its TokenStream. If we change our getAttribute calls to addAttribute it will add them to the TokenStream if they don’t exist and return them just like getAttribute if they do.

CharTermAttribute charTermAttrib = tokenizer.addAttribute(CharTermAttribute.class);
 TypeAttribute typeAtt = tokenizer.getAttribute(TypeAttribute.class);
 OffsetAttribute offset = tokenizer.addAttribute(OffsetAttribute.class);

Of course the values of the attributes that aren’t actually set by WhitespaceTokenizer will stay at their default. You may wish to use tokenizer.hasAttribute(TypeAttribute.class) to check if that Attribute exists instead of just adding it if your code later on expects it to return actual values.

You may be able to ignore the edge cases for your application, but if you need more rigorous NLP, you can look at OpenNLP which does Sentence Detection as well as Tokenizing, Entity Extraction, etc… based on trained Maximum Entropy models (some of the provided models are even available in other languages).

Read full article from Tokenizing with Lucene | LucidBox