[CONF] Apache Solr Reference Guide > DocValues

["Hoss Man (Confluence)" <co...@apache.org>]

Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr)
Page: DocValues (https://cwiki.apache.org/confluence/display/solr/DocValues)

Change Comment:
---------------------------------------------------------------------
tweak wording, link to fieldType#docValuesFormat, info about upgrading

Edited by Hoss Man:
---------------------------------------------------------------------
An exciting addition to Solr functionality was introduced in Solr 4.2. This functionality has been around in Lucene for a while, but is now available to Solr users.

DocValues are a way of building the index that is more efficient for some purposes.

h2. Why DocValues?

The standard way that Solr builds the index is with an _inverted index_. This style builds a list of terms found in all the documents in the index and next to each term is a list of documents that the term appears in (as well as how many times the term appears in that document). This makes search very fast - since users search by terms, having a ready list of term-to-document values makes the query process faster.

For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this approach is not very efficient. The faceting engine, for example, must look up each term that appears in each document that will make up the result set and pull the document IDs in order to build the facet list. In Solr, this is maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).

In Lucene 4.0, a new approach was introduced. DocValue fields are now column-oriented fields with a document-to-value mapping built at index time. This approach promises to relieve some of the memory requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.

h2. How to Use DocValues

To use docValues, you only need to enable it for a field that you will use it with. As with all schema design, you need to define a field type and then define fields of that type with docValues enabled. All of these actions are done in {{schema.xml}}.

Enabling a field for docValues only requires adding {{docValues="true"}} to the field definition, as in this example (from Solr's default {{schema.xml}}):

{code:xml|borderStyle=solid|borderColor=#666666}
<field name="manu_exact" type="str" indexed="false" stored="false" docValues="true" default=""/>
{code}

{note}
If you have already indexed data into your Solr index, you will need to completely re-index your content after changing your field definitions in {{schema.xml}} in order to successfully use docValues.
{note}

DocValues are only available for specific field types. The types chosen determine the underlying Lucene docValue type that will be used. The available Solr field types are:

* String fields of type {{StrField}}. If this type is used, the field must be either required or have a default value, meaning every document must have a value for this field. 
** If the field is single-valued (i.e., multi-valued is false), Lucene will use the SORTED type.
** If the field is multi-valued, Lucene will use the SORTED_SET type.
* Any Trie* fields. If this type is used, the field must be either required or have a default value, meaning every document must have a value for this field.
** If the field is single-valued (i.e., multi-valued is false), Lucene will use the NUMERIC type.
** If the field is multi-valued, Lucene will use the SORTED_SET type.
* UUID fields

These Lucene types are related to how the values are sorted and stored. For more information, please refer to the Solr Wiki at [http://wiki.apache.org/solr/DocValues].

It's important that the fields be populated (either with values on every document or a default value that is applied if it is missing from the document) to avoid an error. Because docValue columns are stored in sorted order, with the first value being an ordinal number starting at 0, the distinction between "empty" and "first" is not possible with docValues. 

There is an additional configuration option available, which is to modify the {{docValuesFormat}} [used by the field type|Field Type Definitions and Properties#docValuesFormat]. The default implementation loads everything into memory, but in some cases you may wish to keep most data on disk. This option may be less performant, but it tries to maintain reasonable performance (and it's still better than fieldCache). You can do this by defining {{docValuesFormat="Disk"}} on the field type, as in this example:

{code:xml|borderStyle=solid|borderColor=#666666}
<fieldType name="string_ondisk" class="solr.StrField" docValuesFormat="Disk" />
{code}

Please note that the {{docValuesFormat}} option may change in future releases.

{info}Lucene index back-compatibility is only supported for the default codec. If you choose to customize the {{docValuesFormat}} in your schema.xml, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.
{info} 

h2. Related Topics

DocValues are quite new to Solr. For more background see:

* [Introducing Lucene Index Doc Values|http://www.searchworkings.org/blog/-/blogs/introducing-lucene-index-doc-values], by Simon Willnauer, at SearchWorkings.org
* [Fun with DocValues in Solr 4.2|http://searchhub.org/2013/04/02/fun-with-docvalues-in-solr-4-2/], by David Arthur, at SearchHub.org

{scrollbar}


Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr
Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action