[CONF] Apache Solr Reference Guide > IndexConfig in SolrConfig

["Hoss Man (Confluence)" <co...@apache.org>]

Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr)
Page: IndexConfig in SolrConfig (https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig)

Change Comment:
---------------------------------------------------------------------
SOLR-4977: infostream is now simple boolean and writes to logger

Edited by Hoss Man:
---------------------------------------------------------------------
{section}
{column:width=75%}
The {{<indexConfig>}} section of {{solrconfig.xml}} defines low-level behavior of the Lucene index writers. By default, the settings are commented out in the sample {{solrconfig.xml}} included with Solr, which means the defaults are used. In most cases, the defaults are fine.

{code:xml|borderStyle=solid|borderColor=#666666}
<indexConfig>
    ...
</indexConfig>
{code}

{note}
Prior to Solr 4, many of these settings were contained in sections called {{mainIndex}} and {{indexDefaults}}. In Solr 4, those sections are deprecated and removed. Any settings that used to be in those sections, now belong in {{<indexConfig>}}.
{note}
{column}
{column:width=25%}
{panel}
Parameters covered in this section:
{toc:maxLevel=2}
{panel}
{column}
{section}

h2. Sizing Index Segments

h3. {{ramBufferSizeMB}}

Once accumulated document updates exceed this much memory space (defined in megabytes), then the pending updates are flushed. This can also create new segments or trigger a merge. Using this setting is generally preferable to {{maxBufferedDocs}}. If both {{maxBufferedDocs}} and {{ramBufferSizeMB}} are set in {{solrconfig.xml}}, then a flush will occur when either limit is reached. The default is 100Mb (raised from 32Mb for Solr 4.1).

{code:xml|borderStyle=solid|borderColor=#666666}
<ramBufferSizeMB>100</ramBufferSizeMB>
{code}

h3. {{maxBufferedDocs}}

Sets the number of document updates to buffer in memory before flushed to disk and added to the current index segment. If the segment fills up, a new one may be created, or a merge may be started. The default Solr configuration leaves this value undefined. 

{code:xml|borderStyle=solid|borderColor=#666666}
<maxBufferedDocs>1000</maxBufferedDocs>
{code}

h3. {{maxIndexingThreads}}

The maximum number of simultaneous threads used to index documents. Once this threshold is reached, additional threads will wait for the others to finish. The default is 8. This parameter is new for Solr 4.1.

{code:xml|borderStyle=solid|borderColor=#666666}
<maxIndexingThreads>8</maxIndexingThreads>
{code}

h3. {{UseCompoundFile}}

Setting {{<useCompoundFile>}} to *true* combines the various files of a segment into a single file, although the default is *false*. On systems where the number of open files allowed per process is limited, setting this to *false* may avoid hitting that limit (the open files limit might also be tunable for your OS with the Linux/Unix {{ulimit}} command, or something similar for other operating systems). In some cases, other internal factors may set a segment to "compound=false", even if this is setting is explicitly set to true, so the compounding of the files in a segment may not always happen.

Updating a compound index may incur a minor performance hit for various reasons, depending on the runtime environment. For example, filesystem buffers are typically associated with open file descriptors, which may limit the total cache space available to each index.

This setting may also affect how much data needs to be transferred during index replication operations.

The default is *false*.

{code:xml|borderStyle=solid|borderColor=#666666}
<useCompoundFile>false</useCompoundFile>
{code}

h2. Merging Index Segments

h3. {{mergeFactor}}

The {{mergeFactor}} controls how many segments a Lucene index is allowed to have before it is coalesced into one segment.  When an update is made to an index, it is added to the most recently opened segment. When that segment fills up (see [{{maxBufferedDocs}} and {{ramBufferSizeMB}}|#Other Indexing Settings] in the next section), a new segment is created and subsequent updates are placed there.

If creating a new segment would cause the number of lowest-level segments to exceed the {{mergeFactor}} value, then all those segments are merged together to form a single large segment. Thus, if the merge factor is ten, each merge results in the creation of a single segment that is roughly ten times larger than each of its ten constituents. When there are {{mergeFactor}} settings for these larger segments, then they in turn are merged into an even larger single segment. This process can continue indefinitely.

Choosing the best merge factor is generally a trade-off of indexing speed vs. searching speed.  Having fewer segments in the index generally accelerates searches, because there are fewer places to look. It also can also result in fewer physical files on disk. But to keep the number of segments low, merges will occur more often, which can add load to the system and slow down updates to the index.

Conversely, keeping more segments can accelerate indexing, because merges happen less often, making an update is less likely to trigger a merge. But searches become more computationally expensive and will likely be slower, because search terms must be looked up in more index segments. Faster index updates also means shorter commit turnaround times, which means more timely search results.

The default value in the example {{solrconfig.xml}} is 10, which is a reasonable starting point.

{code:xml|borderStyle=solid|borderColor=#666666}
<mergeFactor>10</mergeFactor>
{code}

h3. {{mergePolicy}}

Defines how merging segments is done. The default in Solr is the {{TieredMergePolicy}}. This default policy merges segments of approximately equal size, subject to an allowed number of segments per tier. Other policies available are the {{LogMergePolicy}}, {{LogByteSizeMergePolicy}} and {{LogDocMergePolicy}}. For more information on these policies, please see the Lucene javadocs at [http://lucene.apache.org/core/4_0_0/core/index.html?org/apache/lucene/index/MergePolicy.html].

{code:xml|borderStyle=solid|borderColor=#666666}
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
   <int name="maxMergeAtOnce">10</int>
   <int name="segmentsPerTier">10</int>
</mergePolicy>
{code}

{note}
When using {{TieredMergePolicy}}, the setting {{maxMergeDocs}} is not needed. Since this is the default in Solr, the setting is effectively removed. However, if using another policy, this setting may be useful.
{note}

h3. {{mergeScheduler}}

The merge scheduler controls how merges are performed. The default {{ConcurrentMergeScheduler}} performs merges in the background using separate threads. The alternative, {{SerialMergeScheduler}}, does not perform merges with separate threads.

{code:xml|borderStyle=solid|borderColor=#666666}
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
{code}

h2. Index Locks

h3. {{lockType}}

The LockFactory options specify its implementation.

{{lockType=single}} uses SingleInstanceLockFactory, and is for a read-only index or when there is no possibility of another process trying to modify the index.

{{lockType=native}} uses NativeFSLockFactory to specify native OS file locking. Do not use when multiple Solr web applications in the same JVM are attempting to share a single index.

{{lockType=simple}} uses SimpleFSLockFactory to specify a plain file for locking.

{{native}} is the default for Solr3.6 and later versions; otherwise {{simple}} is the default.

For more information on the nuances of each LockFactory, see [http://wiki.apache.org/lucene-java/AvailableLockFactories].

{code:xml|borderStyle=solid|borderColor=#666666}
<lockType>native</lockType>
{code}

h3. {{unlockOnStartup}}

If *true*, any write or commit locks that have been held will be unlocked on system startup. This defeats the locking mechanism that allows mulitple processes to safely access a Lucene index. The default is *false*, and changing this should only be done with care. This parameter is not used if the {{lockType}} is "none" or "single".

{code:xml|borderStyle=solid|borderColor=#666666}
<unlockOnStartup>false</unlockOnStartup>
{code}

h3. {{writeLockTimeout}}

The maximum time to wait for a write lock on an IndexWriter. The default is 1000, expressed in milliseconds.

{code:xml|borderStyle=solid|borderColor=#666666}
<writeLockTimeout>1000</writeLockTimeout>
{code}

h2. Other Indexing Settings

There are a few other parameters that may be important to configure for your implementation. These settings affect how or when updates are made to an index.

|| Setting || Description ||
| termIndexInterval | Controls how often terms are loaded into memory. The default is 128. |
| reopenReaders | Controls if IndexReaders will be re-opened, instead of closed and then opened, which is often less efficient. The default is true. |
| deletionPolicy| Controls how commits are retained in case of rollback. The default is {{SolrDeletionPolicy}}, which has sub-parameters for the maximum number of commits to keep ({{maxCommitsToKeep}}), the maximum number of optimzed commits to keep ({{maxOptimizedCommitsToKeep}}), and the maximum age of any commit to keep ({{maxCommitAge}}), which supports {{DateMathParser}} syntax. |
| infoStream | The InfoStream setting instructs the underlying Lucene classes to write detailed debug information from the indexing process as Solr log messages. |

{code:xml|borderStyle=solid|borderColor=#666666}
<termIndexInterval>128</termIndexInterval>
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
  <str name="maxCommitsToKeep">1</str> 
  <str name="maxOptimizedCommitsToKeep">0</str> 
  <str name="maxCommitAge">1DAY</str>
</deletionPolicy>
<infoStream>false</infoStream>
{code}

{info}
The {{maxFieldLength}} parameter was removed in Solr 4. If restricting the length of fields is important to you, you can get similar behavior with the {{LimitTokenCountFactory}}, which can be defined for the fields you'd like to limit. For example, {{<filter class="solr.LimitTokenCountFilterFactory" maxTokenCount="10000"/>}} would limit the field to 10,000 characters.
{info}

{scrollbar}


Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr
Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action