Lucene 4.10.1 Package org.apache.lucene.search

Lucene 4.10.1 Package org.apache.lucene.search

Score Boosting

Lucene allows influencing search results by "boosting" at different times:

• Index-time boost by calling Field.setBoost() before a document is added to the index.
• Query-time boost by setting a boost on a query clause, calling Query.setBoost().

Indexing time boosts are pre-processed for storage efficiency and written to storage for a field as follows:

• All boosts of that field (i.e. all boosts under the same field name in that doc) are multiplied.
• The boost is then encoded into a normalization value by the Similarity object at index-time: computeNorm(). The actual encoding depends upon the Similarity implementation, but note that most use a lossy encoding (such as multiplying the boost with document length or similar, packed into a single byte!).
• Decoding of any index-time normalization values and integration into the document's score is also performed at search time by the Similarity.

Changing Scoring — Similarity

Changing Similarity is an easy way to influence scoring, this is done at index-time withIndexWriterConfig.setSimilarity(Similarity) and at query-time with IndexSearcher.setSimilarity(Similarity). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly); Lucene makes no effort to verify this.

You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can extend by plugging in a different component (e.g. term frequency normalizer).

Finally, you can extend the low level Similarity directly to implement a new retrieval model, or to use external scoring factors particular to your application. For example, a custom Similarity can access per-document values via FieldCache or NumericDocValues and integrate them into the score.

Custom queries are an expert level task, so tread carefully and be prepared to share your code if you want help.

With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by three main classes:

1. Query — The abstract object representation of the user's information need.
2. Weight — The internal interface representation of the user's Query, so that Query objects may be reused. This is global (across all segments of the index) and generally will require global statistics (such as docFreq for a given term across all segments).
3. Scorer — An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities. This is created per-segment.
4. BulkScorer — An abstract class that scores a range of documents. A default implementation simply iterates through the hits fromScorer, but some queries such as BooleanQuery have more efficient implementations.

The Query class has several methods that are important for derived classes:

1. createWeight(IndexSearcher searcher) — A Weight is the internal representation of the Query, so each Query implementation must provide an implementation of Weight

1. rewrite(IndexReader reader) — Rewrites queries into primitive queries. Primitive queries are: TermQuery, BooleanQuery, and other queries that implement createWeight(IndexSearcher searcher)

1. getValueForNormalization() — A weight can return a floating point value to indicate its magnitude for query normalization. Typically a weight such as TermWeight that scores via a Similarity will just defer to the Similarity's implementation:SimWeight#getValueForNormalization(). For example, with Lucene's classic vector-space formula, this is implemented as the sum of squared weights: (idf * boost)2
2. normalize(float norm, float topLevelBoost) — Performs query normalization:
   • topLevelBoost: A query-boost factor from any wrapping queries that should be multiplied into every document's score. For example, a TermQuery that is wrapped within a BooleanQuery with a boost of 5 would receive this value at this time. This allows the TermQuery (the leaf node in this case) to compute this up-front a single time (e.g. by multiplying into the IDF), rather than for every document.
   • norm: Passes in a a normalization factor which may allow for comparing scores between queries.
   Typically a weight such as TermWeight that scores via a Similarity will just defer to the Similarity's implementation:SimWeight#normalize(float,float).

The Scorer Class

The Scorer abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from DocIdSetIterator):

1. nextDoc() — Advances to the next document that matches this Query, returning true if and only if there is another document that matches.
2. docID() — Returns the id of the Document that contains the match.
3. score() — Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the TermScorer simply defers to the configured Similarity: SimScorer.score(int doc, float freq).
4. freq() — Returns the number of matches for the current document. This value can be determined in any appropriate way for an application. For instance, the TermScorer simply defers to the term frequency from the inverted index: DocsEnum.freq().
5. advance() — Skip ahead in the document matches to the document whose id is greater than or equal to the passed in value. In many instances, advance can be implemented more efficiently than simply looping through all the matching documents until the target document is identified.
6. getChildren() — Returns any child subscorers underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained details on the scoring process.

The BulkScorer scores a range of documents. There is only one abstract method:

1. score(Collector,int) — Score all documents up to but not including the specified max document.

Search Algorithm

This section is mostly notes on stepping through the Scoring process and serves as fertilizer for the earlier sections.

In the typical search application, a Query is passed to the IndexSearcher, beginning the scoring process.

Once inside the IndexSearcher, a Collector is used for the scoring and sorting of the search results. These important objects are involved in a search:

1. The Weight object of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher.
2. The IndexSearcher that initiated the call.
3. A Filter for limiting the result set. Note, the Filter may be null.
4. A Sort object for specifying how to sort the results if the standard score-based sort method is not desired.

Assuming we are not sorting (since sorting doesn't affect the raw Lucene score), we call one of the search methods of the IndexSearcher, passing in the Weight object created by IndexSearcher.createNormalizedWeight(Query), Filter and the number of results we want. This method returns a TopDocs object, which is an internal collection of search results. The IndexSearcher creates a TopScoreDocCollector and passes it along with the Weight, Filter to another expert search method (for more on the Collector mechanism, see IndexSearcher). The TopScoreDocCollector uses a PriorityQueue to collect the top results for the search.

If a Filter is being used, some initial setup is done to determine which docs to include. Otherwise, we ask the Weight for a Scorer for eachIndexReader segment and proceed by calling BulkScorer.score(Collector).

At last, we are actually going to score some documents. The score method takes in the Collector (most likely the TopScoreDocCollector or TopFieldCollector) and does its business.Of course, here is where things get involved. The Scorer that is returned by the Weight object depends on what type of Query was submitted. In most real world applications with multiple query terms, the Scorer is going to be aBooleanScorer2 created from BooleanWeight (see the section on custom queries for info on changing this).

Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the coord() factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query. Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the Scorer.nextDoc() method. The nextDoc() method advances to the next document matching the query. This is an abstract method in the Scorer class and is thus overridden by all derived implementations. If you have a simple OR query your internal Scorer is most likely a DisjunctionSumScorer, which essentially combines the scorers from the sub scorers of the OR'd terms.

Please read full article from Lucene 4.10.1 Package org.apache.lucene.search