Fun with DocValues in Solr 4.2 - Lucidworks
The “dvd” and “dvm” files are the DocValues, “tim” and “tip” are the terms index and dictionary, and “fdx” and “fdt” are the stored fields. You can look up what the rest of those files are in the Lucene documentation.
Without going any further, we can see that the DocValues are much more compact than the stored fields and the term index just by looking at the file sizes (recall that we are storing each of our fields as stored, indexed, and docValues separately). Since values for a single field are stored contiguously, very efficient packing algorithms can be used.
https://gist.github.com/mumrah/5265594
The performance differences here depend largely on the number of unique values for the field and the field type. The biggest difference in loading time is the field “word_idx”, which take twice as long to load from the inverted index and uses three times as much memory. For repeated access to the same field, the inverted index performs better due to internal Lucene caching (also the reason for higher memory consumption). In all cases, DocValues consume less memory during loading and after garbage collection.
DocValues have many potential uses. As we have seen from our little experiment, they are less memory hungry than indexed field and typically faster to load. If you are in a low-memory environment, or you don’t need to index a field, DocValues are perfect for faceting/grouping/filtering/sorting. They also have the potential for increasing the number of fields you can facet/group/filter/sort on without increasing your memory requirements.
Read full article from Fun with DocValues in Solr 4.2 - Lucidworks
The “dvd” and “dvm” files are the DocValues, “tim” and “tip” are the terms index and dictionary, and “fdx” and “fdt” are the stored fields. You can look up what the rest of those files are in the Lucene documentation.
Without going any further, we can see that the DocValues are much more compact than the stored fields and the term index just by looking at the file sizes (recall that we are storing each of our fields as stored, indexed, and docValues separately). Since values for a single field are stored contiguously, very efficient packing algorithms can be used.
https://gist.github.com/mumrah/5265594
The performance differences here depend largely on the number of unique values for the field and the field type. The biggest difference in loading time is the field “word_idx”, which take twice as long to load from the inverted index and uses three times as much memory. For repeated access to the same field, the inverted index performs better due to internal Lucene caching (also the reason for higher memory consumption). In all cases, DocValues consume less memory during loading and after garbage collection.
DocValues have many potential uses. As we have seen from our little experiment, they are less memory hungry than indexed field and typically faster to load. If you are in a low-memory environment, or you don’t need to index a field, DocValues are perfect for faceting/grouping/filtering/sorting. They also have the potential for increasing the number of fields you can facet/group/filter/sort on without increasing your memory requirements.
Read full article from Fun with DocValues in Solr 4.2 - Lucidworks
No comments:
Post a Comment