You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ja...@apache.org on 2020/02/24 08:58:06 UTC
[lucene-solr] branch branch_8x updated: SOLR-10306: Document in
Reference Guide how to disable or reduce swapping (#1256)
This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
The following commit(s) were added to refs/heads/branch_8x by this push:
new 365728f SOLR-10306: Document in Reference Guide how to disable or reduce swapping (#1256)
365728f is described below
commit 365728f7f45cd1056de4b9267bfa73fcf593bfba
Author: Jan Høydahl <ja...@apache.org>
AuthorDate: Mon Feb 24 09:50:01 2020 +0100
SOLR-10306: Document in Reference Guide how to disable or reduce swapping (#1256)
(cherry picked from commit 8b98befe9362b9810297a1edb475d9341d44b426)
---
solr/CHANGES.txt | 2 +
solr/solr-ref-guide/src/jvm-settings.adoc | 8 ++--
.../src/taking-solr-to-production.adoc | 49 ++++++++++++++++++++--
3 files changed, 52 insertions(+), 7 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 1d843da..f2800c4 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -177,6 +177,8 @@ Other Changes
* SOLR-11035: (at least) 2 distinct failures possible when clients attempt searches
during SolrCore reload. Removed Bandaid kludge. (Erick Erickson)
+* SOLR-10306: Document in Reference Guide how to disable or reduce swapping (janhoy)
+
================== 8.4.1 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
diff --git a/solr/solr-ref-guide/src/jvm-settings.adoc b/solr/solr-ref-guide/src/jvm-settings.adoc
index f3114ce..b4a6aef 100644
--- a/solr/solr-ref-guide/src/jvm-settings.adoc
+++ b/solr/solr-ref-guide/src/jvm-settings.adoc
@@ -26,7 +26,7 @@ For more general information about improving Solr performance, see https://cwiki
The most important JVM configuration settings control the heap allocated to the JVM: `-Xms`, which sets the initial size of the JVM's memory heap, and `-Xmx`, which sets the maximum size of the heap. Setting these two options to the same value is a common practice.
-Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application. The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (GCViewer and GCEasy are two). Also you can attach jconsole (distributed with most Java runtimes) to check mem [...]
+Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application. The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (GCViewer and GCEasy are two). Also you can attach jconsole (distributed with most Java runtimes) to check mem [...]
There are several points to keep in mind:
@@ -34,8 +34,8 @@ There are several points to keep in mind:
* Lucene/Solr makes extensive use of MMapDirectory, which uses RAM _not_ reserved for the JVM for most of the Lucene index. Therefore, as much memory as possible should be left for the operating system to use for this purpose.
* The heap allocated should be as small as possible while maintaining good performance. 8-16G is quite common, and larger heaps are sometimes used. When heaps grow to larger sizes, it is imperative to test extensively before going to production.
* The G1GC garbage collector is currently preferred when using a JVM that supports it (Java 9 and later)
- * Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs. It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps.
- * It's good practice to periodically re-analyze the GC logs and/or monitor with <<metrics-reporting#metrics-reporting,Metrics Reporting>> to see if the memory usage has changed due to changes in your application, number of documents, etc.
+ * Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs. It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps. One way to achieve this is to run Solr as a Docker container.
+ * It's good practice to periodically re-analyze the GC logs and/or monitor with <<metrics-reporting#metrics-reporting,Metrics Reporting>> to see if the memory usage has changed due to changes in your application, number of documents, etc.
* On *nix systems, we recommend that Solr be run with the "oom killer script" (see solr/bin/oom_solr.sh). This will forcefully stop Solr when the heap is exhausted rather than continue in an indeterminate state.
* All current (Java 11) garbage collectors can hit "stop the world" collections, which suspend the JVM until completed. If, through monitoring, these collections are frequent and greater than your application can tolerate, additional tuning should be considered. "Stop the world" pauses greater than 5 seconds are rarely acceptable, and having them be less than 1 second is desirable.
@@ -47,4 +47,4 @@ If you are using Sun's JVM, add the `-server` command-line option when you start
== Checking JVM Settings
-A great way to see what JVM settings your server is using, along with other useful information, is to use the admin RequestHandler, `solr/admin/system`. This request handler will display a wealth of server statistics and settings.
+A great way to see what JVM settings your server is using, along with other useful information, is to use the admin RequestHandler, `/solr/admin/info/system`. This request handler will display a wealth of server statistics and settings.
diff --git a/solr/solr-ref-guide/src/taking-solr-to-production.adoc b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
index 8227f0e..9f96fa4 100644
--- a/solr/solr-ref-guide/src/taking-solr-to-production.adoc
+++ b/solr/solr-ref-guide/src/taking-solr-to-production.adoc
@@ -179,13 +179,18 @@ Alternatively, the boolean system property `lucene.cms.override_spins` can be se
=== Memory and GC Settings
-By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 10 and 20 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_JAVA_MEM` variable in the include file, such as:
+By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 8 and 16 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_HEAP` variable in the include file, such as:
[source,bash]
----
-SOLR_JAVA_MEM="-Xms10g -Xmx10g"
+SOLR_HEAP="8g"
----
+[NOTE]
+====
+Do not allocate a very large Java Heap unless you know you need it. See <<jvm-settings.adoc#choosing-memory-heap-settings,Choosing Memory Heap Settings>> for details.
+====
+
Also, the <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script>> comes with a set of pre-configured Garbage First Garbage Collection settings that have shown to work well with Solr for a number of different workloads.
However, these settings may not work well for your specific use of Solr. Consequently, you may need to change the GC settings, which should also be done with the `GC_TUNE` variable in the `/etc/default/solr.in.sh` include file. For more information about garbage collection settings refer to following articles:
1. https://cwiki.apache.org/confluence/display/solr/ShawnHeisey
@@ -239,7 +244,7 @@ SOLR_HOST=solr1.example.com
Setting the hostname of the Solr server is recommended, especially when running in SolrCloud mode, as this determines the address of the node when it registers with ZooKeeper.
-=== Environment banner in Admin UI
+=== Environment Banner in Admin UI
To guard against accidentally doing changes to the wrong cluster, you may configure a visual indication in the Admin UI of whether you currently work with a production environment or not. To do this, edit your `solr.in.sh` or `solr.in.cmd` file with a `-Dsolr.environment=prod` setting, or set the cluster property named `environment`. To specify label and/or color, use a comma delimited format as below. The `+` character can be used instead of space to avoid quoting. Colors may be valid C [...]
@@ -302,6 +307,44 @@ Check these limits every time you upgrade your kernel or operating system. These
If these limits are exceeded, the problems reported by Solr vary depending on the specific operation responsible for exceeding the limit. Errors such as "too many open files", "connection error", and "max processes exceeded" have been reported, as well as SolrCloud recovery failures.
====
+=== Avoid Swapping (*nix Operating Systems)
+
+When running a Java application like Lucene/Solr, having the OS swap memory to disk is a very bad situation. We usually prefer a hard crash so other healthy Solr nodes can take over, instead of letting a Solr node swap, causing terrible performance, timeouts and an unstable system. So our recommendation is to disable swap on the host altogether or reduce the "swappiness". These instructions are valid for Linux environments. Also note that when running Solr in a Docker container, these ch [...]
+
+==== Disabling Swap
+
+To disable swap on a Linux system temporarily, you can run the `swapoff` command:
+
+[source,bash]
+----
+sudo swapoff -a
+----
+
+If you want to make this setting permanent, first make sure you have more than enough physical RAM on your host and then consult the documentation for your Linux system for the correct procedure to disable swap.
+
+==== Reduce Swappiness
+
+An alternative option is to reduce the "swappiness" of your system. A Linux system will by default be quite aggressive in swapping out memory to disk, by having a high default "swappiness" value. By reducing this to a very low value, it will have almost the same effect as using `swapoff`, but Linux will still be allowed to swap in case of emergency. To check the current swappiness setting, run:
+
+[source,bash]
+----
+cat /proc/sys/vm/swappiness
+----
+
+Next, to change the setting permanently, open `/etc/sysctl.conf` as the root user. Then, change or add this line to the file:
+
+[source,bash]
+----
+vm.swappiness = 1
+----
+
+Alternatively, you can change the setting temporarily by running this comamnd:
+
+[source,bash]
+----
+echo 1 > /proc/sys/vm/swappiness
+----
+
== Running Multiple Solr Nodes per Host
The `bin/solr` script is capable of running multiple instances on one machine, but for a *typical* installation, this is not a recommended setup. Extra CPU and memory resources are required for each additional instance. A single instance is easily capable of handling multiple indexes.