You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by sh...@apache.org on 2018/03/15 17:37:31 UTC
lucene-solr:branch_7x: SOLR-12098: Document the Lucene spins
auto-detection and its effect on CMS dynamic defaults
Repository: lucene-solr
Updated Branches:
refs/heads/branch_7x 6c61b1557 -> ad82d65a7
SOLR-12098: Document the Lucene spins auto-detection and its effect on CMS dynamic defaults
(cherry picked from commit d50890e)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ad82d65a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ad82d65a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ad82d65a
Branch: refs/heads/branch_7x
Commit: ad82d65a7a9482e11bc56f18b7ec3c5833583fcf
Parents: 6c61b15
Author: Shalin Shekhar Mangar <sh...@apache.org>
Authored: Thu Mar 15 23:06:52 2018 +0530
Committer: Shalin Shekhar Mangar <sh...@apache.org>
Committed: Thu Mar 15 23:07:21 2018 +0530
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +++
.../src/indexconfig-in-solrconfig.adoc | 20 ++++++++++++++++++++
.../src/taking-solr-to-production.adoc | 14 ++++++++++++++
3 files changed, 37 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ad82d65a/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 57bcd3b..53513c5 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -427,6 +427,9 @@ Other Changes
* SOLR-12099: Remove reopenReaders attribute from 'IndexConfig in SolrConfig' page in ref guide. (shalin)
+* SOLR-12098: Document the Lucene spins auto-detection and its effect on CMS dynamic defaults.
+ (Cassandra Targett, shalin)
+
================== 7.2.1 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ad82d65a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
index e80eb07..c0c1696 100644
--- a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
@@ -112,11 +112,31 @@ The example above shows Solr's {solr-javadocs}/solr-core/org/apache/solr/index/S
The merge scheduler controls how merges are performed. The default `ConcurrentMergeScheduler` performs merges in the background using separate threads. The alternative, `SerialMergeScheduler`, does not perform merges with separate threads.
+The `ConcurrentMergeScheduler` has two configurable attributes:
+
+`maxMergeCount`::
+The maximum number of simultaneous merges that are allowed. If a merge is necessary yet we already have this many threads running, the indexing thread will block until a merge thread has completed. Note that Solr will only run the smallest `maxThreadCount` merges at a time.
+
+`maxThreadCount`::
+The maximum number of simultaneous merge threads that should be running at once. This must be less than `maxMergeCount`.
+
+The defaults for the above attributes are dynamically set based on whether the underlying disk drive is rotational disk or not. Refer to the <<taking-solr-to-production.adoc#dynamic-defaults-for-concurrentmergescheduler, Dynamic defaults for ConcurrentMergeScheduler>> section for more details.
+
+.Example: Dynamic defaults
[source,xml]
----
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
----
+.Example: Explicit defaults
+[source,xml]
+----
+<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
+ <int name="maxMergeCount">9</int>
+ <int name="maxThreadCount">4</int>
+</mergeScheduler>
+----
+
=== mergedSegmentWarmer
When using Solr in for <<near-real-time-searching.adoc#near-real-time-searching,Near Real Time Searching>> a merged segment warmer can be configured to warm the reader on the newly merged segment, before the merge commits. This is not required for near real-time search, but will reduce search latency on opening a new near real-time reader after a merge completes.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ad82d65a/solr/solr-ref-guide/src/taking-solr-to-production.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/taking-solr-to-production.adoc b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
index 88f127c..4ede6b4 100644
--- a/solr/solr-ref-guide/src/taking-solr-to-production.adoc
+++ b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
@@ -163,6 +163,20 @@ If the `status` command is not successful, look for error messages in `/var/solr
== Fine-Tune Your Production Setup
+=== Dynamic Defaults for ConcurrentMergeScheduler
+
+The Merge Scheduler is configured in `solrconfig.xml` and defaults to `ConcurrentMergeScheduler`. This scheduler uses multiple threads to merge Lucene segments in the background.
+
+By default, the `ConcurrentMergeScheduler` auto-detects whether the underlying disk drive is rotational or a SSD and sets defaults for `maxThreadCount` and `maxMergeCount` accordingly. If the disk drive is determined to be rotational then the `maxThreadCount` is set to 1 and `maxMergeCount` is set to 6. Otherwise, `maxThreadCount` is set to 4 or half the number of processors available to the JVM whichever is greater and `maxMergeCount` is set to `maxThreadCount+5`.
+
+This auto-detection works only on Linux and even then it is not guaranteed to be correct. On all other platforms, the disk is assumed to be rotational. Therefore, if the auto-detection fails or is incorrect then indexing performance can suffer badly due to the wrong defaults.
+
+The auto-detected value is exposed by the <<metrics-reporting.adoc#metrics-api, Metrics API>> with the key `solr.node:CONTAINER.fs.coreRoot.spins`. A value of `true` denotes that the disk is detected to be a rotational or spinning disk.
+
+It is safer to explicitly set values for `maxThreadCount` and `maxMergeCount` in the <<indexconfig-in-solrconfig.adoc#mergescheduler, IndexConfig section of SolrConfig.xml>> so that values appropriate to your hardware are used.
+
+Alternatively, the boolean system property `lucene.cms.override_spins` can be set in the `SOLR_OPTS` variable in the include file to override the auto-detected value. Similarily, the system property `lucene.cms.override_core_count` can be set to the number of CPU cores to override the auto-detected processor count.
+
=== Memory and GC Settings
By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 10 and 20 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_JAVA_MEM` variable in the include file, such as: