Lucene Highlighter HowTo

Lucene Highlighter HowTo

Last part of the indexing side is the TermVectors. From the Lucene Javadocs:
“A term vector is a list of the document’s terms and their number of occurrences in that document.”

For the Highlighter, TermVectors need to be available and you have a choice of either computing and storing them with the index at index time or computing them as you need them when the search is performed. Above, Field.TermVector.WITH_POSITIONS_OFFSETS indicates were are computing and storing them in the index at index time.

With the index ready for presenting contextual fragments, lets move on to generating them while processing a search request. Below is a typical “Hello World” type search block.

1 2 3 4 5 6 7 8 9 10 11 12 13

QueryParser qp = new QueryParser(“contents”, analyzer); Query query = qp.parse(searchInput); TopDocs hits = searcher.search(query, 10);   for (int i = 0; i < hits.scoreDocs.length; i++) {     int docId = hits.scoreDocs[i].doc;     Document doc = searcher.doc(docId);     String filename = doc.get("filename");     String contents =  doc.get(“contents”);           String[] fragments = hlu.getFragmentsWithHighlightedTerms(analyzer,                    query, “contents”, contents, 5, 100); }

First we obtain the TokenStream. The call shown above assumes term vectors were not stored in the index at index time.

Next is the SpanScorer and SimpleSpanFragmenter. These work to break the contents into 100 character fragments and rank them by relevancy. You can use SpanScorer and SimpleSpanFragmenter or QueryScorer and SimpleFragmenter. The full details can be found in the Javadocs.

Note: when indexing large files, like the full contents of PDF manuals, you might need to tell the Highlighter object to look at the full text by calling the setMaxDocCharsToAnalyze method with Integer.MAX_VALUE or a more appropriate value. In my case, the default value specified by Lucene was too small, thus Highlighter did not look at the full text to generate the fragments. This was not good because the match I was seeking was near the end of the contents.

Finally, we tell the Highlighter to return the best 5 fragments.

The full code for this example can be downloaded from my Google Code project. The source file that makes use of the Highlighter is HighligherUtil.java

You can also find examples of using Highlighter in the Lucene SVN Repository, specificallyHighlighterTest.java

As you can see, returning search results with contextual fragments containing your search terms is very easy with the Lucene Highlighter contrib library once you know the steps to follow.

Please read full article from Lucene Highlighter HowTo