the MoreLikeThis instance loops over the field names, and retrieves a term vector for each of the fields in the document we’re analyzing. A term vector is a data structure that holds a list of all the words that were in the field and the number of times each word was used.
After we’ve got those, each of the term vectors is merged into a map: the key being the term and the value being the number of times the word was used in the document. The map is then handed to a method (called createQueue) that calculates a reasonably complex score for each word in the map.
it does is create an instance of a class called FreqQ, which extends the PriorityQueue class. This object will maintain an object array whose elements are ordered according to their score.
Now the createQueue method iterates over each word in the term frequency map, throwing out words if they don’t occur enough times (see setMinTermFreq) and then testing to find out which field across the entire the Lucene index contains the term the most. Next it calculates the Inverse Document Frequency,
… a score factor based on a term’s document frequency (the number of documents which contain the term)… Terms that occur in fewer documents are better indicators of topic, so implementations of this method usually return larger values for rare terms, and smaller values for common terms.
and finally, a score, which is a product of the IDF score and the number of times the word existed in the source document.
Lucene finds the field that contains the most instances of the given term and then calculates the idf value and the score. The default implementation of the idf value looks like this:
return (float)(Math.log(numDocs/(double)(docFreq+1)) + 1.0)
Finally, after all the terms have been added to the priority queue, we create a Lucene query, looping over the first 25 terms (this is the default and can be changed via setMaxQueryTerms) in the queue
TermQuery tq = new TermQuery(new Term("body", "oracle"));
and optionally boosting each term according to the score:
tq.setBoost(51.376 / 82.978);
The resulting Lucene query (in string format) looks something like this:
body:pre body:username^.56974 body:column^.57123 body:oracle^.61915 ...
Read full article from How MoreLikeThis Works in Lucene | Aaron Johnson
After we’ve got those, each of the term vectors is merged into a map: the key being the term and the value being the number of times the word was used in the document. The map is then handed to a method (called createQueue) that calculates a reasonably complex score for each word in the map.
it does is create an instance of a class called FreqQ, which extends the PriorityQueue class. This object will maintain an object array whose elements are ordered according to their score.
Now the createQueue method iterates over each word in the term frequency map, throwing out words if they don’t occur enough times (see setMinTermFreq) and then testing to find out which field across the entire the Lucene index contains the term the most. Next it calculates the Inverse Document Frequency,
… a score factor based on a term’s document frequency (the number of documents which contain the term)… Terms that occur in fewer documents are better indicators of topic, so implementations of this method usually return larger values for rare terms, and smaller values for common terms.
and finally, a score, which is a product of the IDF score and the number of times the word existed in the source document.
Lucene finds the field that contains the most instances of the given term and then calculates the idf value and the score. The default implementation of the idf value looks like this:
return (float)(Math.log(numDocs/(double)(docFreq+1)) + 1.0)
Finally, after all the terms have been added to the priority queue, we create a Lucene query, looping over the first 25 terms (this is the default and can be changed via setMaxQueryTerms) in the queue
TermQuery tq = new TermQuery(new Term("body", "oracle"));
and optionally boosting each term according to the score:
tq.setBoost(51.376 / 82.978);
The resulting Lucene query (in string format) looks something like this:
body:pre body:username^.56974 body:column^.57123 body:oracle^.61915 ...
Read full article from How MoreLikeThis Works in Lucene | Aaron Johnson
No comments:
Post a Comment