[Solr Wiki] Update of "ShawnHeisey" by ShawnHeisey

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.

The "ShawnHeisey" page has been changed by ShawnHeisey:
https://wiki.apache.org/solr/ShawnHeisey?action=diff&rev1=51&rev2=52

Comment:
Cleaning things up.

  
  == Java 8 recommendation for Solr ==
  
- I have a strong personal recommendation for everyone to use the latest Java 8 from Oracle with Solr 4.x and later.  There have been some very significant memory management improvements in each release of Java 8, particularly with garbage collection, both in general and specifically the G1 collector.  When OpenJDK 8 becomes generally available from *NIX packagers, it will probably be nearly as good as the Oracle version.
+ Java 8 is *required* for Solr 6.x.  For 4.x and 5.x releases, I strongly recommend it.  Earlier releases only required Java 5 and have not been extensively tested with 8.
  
- Don't ever use a JVM from IBM.  IBM aggressively enables HotSpot optimizations by default, and some of those optimizations cause Solr/Lucene to encounter bugs.  OpenJDK is fine, as long as it's at least version 7u1 (1.7.0_1), but newer versions are better.  OpenJDK 6 is known to have bugs.  If you can get an Oracle JVM, you should.
+ There have been some very significant memory management improvements in each release of Java 8, particularly with garbage collection, both in general and specifically the G1 collector.  OpenJDK 8 should be almost as good as Oracle Java 8.
  
- Java 8 will be required for Solr 6.0, not yet released.  At this time, there are no plans to require Java 8 with any 5.x release.
+ It's not a good idea to use a JVM from IBM.  IBM aggressively enables HotSpot optimizations by default, which helps a lot with performance, but some of those optimizations cause Solr/Lucene to encounter bugs.
+ 
+ OpenJDK should be fine, as long as it's 7 or later and meets the requirements of your Solr version.  The latest release is recommended.  OpenJDK 6 is known to have bugs.  If you can get an Oracle JVM, you should.
  
  == GC Tuning for Solr ==
  
- /!\ Starting in version 4.10, and greatly improved in 5.0, startup scripts were added that do GC tuning for you, using CMS (ConcurrentMarkSweep) settings very similar to the CMS settings that you can find further down on this page.  If you are using the bin/solr or bin\solr script to start Solr, you already have GC tuning and won't need to worry about the recommendations here.
+ /!\ Starting in version 4.10, and greatly improved in 5.0, startup scripts were added that do GC tuning for you, using CMS (ConcurrentMarkSweep) settings very similar to the CMS settings that you can find further down on this page.  If you are using the bin/solr or bin\solr script to start Solr, you already have GC tuning and probably won't need to worry about the recommendations here.
  
  The secret to good Solr GC tuning:  Eliminating full garbage collections.  A full garbage collection is slow, no matter what collector you're using.  If you can set up the options for GC such that both young and old generations are always kept below max size by their generation-specific collection algorithms, performance is almost guaranteed to be good.
  
@@ -25, +27 @@

  
  First you must understand how Java's memory model works, and what "[[http://en.wikipedia.org/wiki/Garbage_collection_%28computer_science%29|garbage collection]]" actually means.  I will not attempt to describe GC here ... click the link in the previous sentence or search the Internet if you want to know more.
  
- Java's default garbage collector is designed to do a decent when the major concern of your program is throughput -- how fast it can transfer data.  If a high-throughput program pauses for several seconds, that's no big deal, as long as it is pumping data quickly the rest of the time.
+ Java's default garbage collector is designed to do a decent job when the major concern of your program is throughput -- how fast it can transfer data.  If a high-throughput program pauses for several seconds, that's no big deal, as long as it is pumping data quickly the rest of the time.
  
  Good Solr performance requires optimizing for a different metric -- latency.  The jobs that Solr is doing must complete very quickly, because when a user is searching, they are expecting results very quickly ... usually less than a second or two.  If your website search feature sometimes takes several seconds to respond, users will find one of your competitors instead of using your website.
  
@@ -35, +37 @@

  
  /!\ It's important to point out a warning from the Lucene project about G1.  They say "Do not, under any circumstances, run Lucene with the G1 garbage collector."  Solr is a Lucene program.  I personally have never had any problems at all with the G1 collector, but if you're going to use it, it's important that you know you're going against advice from smart people.  '''If this worries you at all, use the [[#CMS (ConcurrentMarkSweep) Collector|CMS settings]] instead.'''
  
- With the newest Oracle Java versions, G1GC is looking '''extremely''' good.  Do not try these options unless you're willing to run the latest Java 7 or Java 8, preferably from Oracle.  Experiments with G1 on early Java 7 releases were very disappointing.  My most recent testing has been with Oracle 7u72, and I have been informed on multiple occasions that Oracle 8u40 will have much better G1 performance.
+ With the newest Oracle Java versions, G1GC is looking '''extremely''' good.  Do not try these options unless you're willing to run the latest Java 7 or Java 8, preferably from Oracle.  Experiments with G1 on early Java 7 releases were very disappointing.  My most recent testing has been with Oracle 7u72, and I have been informed on multiple occasions that Oracle 8u40 and later has much better G1 performance.
  
  I typically will use a Java heap of 6GB or 7GB.  The following settings were created as a result of a discussion with Oracle employees on the [[http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use|hotspot-gc-use mailing list]]:
  
@@ -58, +60 @@

  
  Garbage collection for allocations that are tagged as humongous is not fully handled in the concurrent and low-pause parts of the process.  They can only be properly handled by a full garbage collection, which will always be slow.  Depending on the size of your indexes and the size of your heap, the region size may require adjustment from the 8MB that I am using above.  If you have an index with about 100 million documents in it, you'll want to use a region size of 32MB, which is the maximum possible size.  Because of this limitation of the G1 collector, we recommend always keeping a Solr index below a maxDoc value of around 100 to 120 million.
  
- Further down on this page, I have included GC logging options that will create a log you can search for "humongous" to determine if there are a lot of these allocations.  I have been informed that with Java 8u40, the !G1HeapRegionSize parameter is not required, because Java 8 manages large allocations a lot better than Java 7 does.
+ Further down on this page, I have included GC logging options that will create a log you can search for "humongous" to determine if there are a lot of these allocations.  I have been told that with Java 8u40, the !G1HeapRegionSize parameter is not required, because Java 8 manages large allocations a lot better than Java 7 does.
  
  ===== Current experiments =====
  This is what I am currently trying out on my servers:
@@ -76, +78 @@

  "
  }}}
  
- This is a graph of a session with the settings above.  The heap size for this graph is -Xms4096M -Xmx6144M.  I am quite thrilled about this.  You may notice that the graph shows a clearly different GC pattern starting at about 8 hours ... this is because a full index rebuild started at just before 8 hours and ran for about 8 hours.  The bulk of the longer pauses occurred during the full reindex, but they are not numerous, and there are millions of documents being indexed at the time.  Before indexing started, there were no long pauses, and after it stopped, there have only been a handful.  I would like to fix all of the long pauses, but I can live with the results I am seeing, especially because the average GC pause time is only six hundredths of a second.
+ This is a graph of a session with the settings above on Java 7u72.  The heap size for this graph is -Xms4096M -Xmx6144M.  I am quite thrilled about this.  You may notice that the graph shows a clearly different GC pattern starting at about 8 hours ... this is because a full index rebuild started at just before 8 hours and ran for about 8 hours.  The bulk of the longer pauses occurred during the full reindex, but they are not numerous, and there are millions of documents being indexed at the time.  Before indexing started, there were no long pauses, and after it stopped, there have only been a handful.  I would like to fix all of the long pauses, but I can live with the results I am seeing, especially because the average GC pause time is only six hundredths of a second.
  
  [[attachment:gc-7u72-2015-03-27-001.png|{{attachment:gc-7u72-2015-03-27-001.png||width=1024}}]]