Solr tuning parameters

["Gerd W. Naschenweng" <ge...@bidorbuy.co.za>]

I am looking for feedback on your setup and current tuning parameters. Hopefully with your feedback we can enhance on the Wiki to list common
tuning parameters. If you can provide the following info, I think it would help everyone starting off on Solr.

I would be specifically interested in your solrconfig.xml, application-server and possible OS tuning.

We are currently in testing phase and still need to do quite a lot of work.

OS: CentOS 5.x / 64bit
RAM: 6GB
CPUs: XEN Server with 1 x 4-core 2.66Ghz Processor
Hardware: IBM BladeServer HS22
Application Server: Jetty 7
SOLR Stats: 1M documents in index.
SolrMeter Stats: Average faceted query response time: 5-40ms. 1200 queries/per minute
Solr Setup: 1 x Master and 1 x Slave. New documents are fed into the master every 5 minutes.

OS Tuning:
- NONE -

JVM Tuning:
# Server config - switch between 32/64 bit environment
JAVA_OPTIONS="$JAVA_OPTIONS -server -d64 -Djava.awt.headless=true"

# Log4J Config
JAVA_OPTIONS="$JAVA_OPTIONS -Dlog4j.configuration=file:${JETTY_HOME}/resources/log4j.properties"

# Solr Config
JAVA_OPTIONS="$JAVA_OPTIONS -Dfile.encoding=UTF-8"
JAVA_OPTIONS="$JAVA_OPTIONS -Dsolr.solr.home=${SOLR_HOME} -Dsolr.data.dir=${SOLR_HOME}/data"

## Logging
#JAVA_OPTIONS="$JAVA_OPTIONS -XX:+PrintCommandLineFlags"
# turn on some debug for GC only print the distribution when doing tuning
JAVA_OPTIONS="$JAVA_OPTIONS -verbose:gc"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+PrintGCTimeStamps"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+PrintGCDetails"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+PrintTenuringDistribution"
JAVA_OPTIONS="$JAVA_OPTIONS -Xloggc:${JETTY_HOME}/logs/jetty-gc.log"

## Garbage collection
# Keep each survivor space about 90% full 
JAVA_OPTIONS="$JAVA_OPTIONS -XX:TargetSurvivorRatio=90"
# A SurvivorRatio of 5 to a 128MB new space will yield two ~64MB survivor spaces
JAVA_OPTIONS="$JAVA_OPTIONS -XX:SurvivorRatio=5"
# Copy object between survivor space at most 16 times 
JAVA_OPTIONS="$JAVA_OPTIONS -XX:MaxTenuringThreshold=16"
# Force CMS for the collector
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+UseConcMarkSweepGC"
# Use incremental mode since minor CPU overhead is better than potential pauses
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+CMSIncrementalMode"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+CMSIncrementalPacing"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+CMSParallelRemarkEnabled"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+UseParNewGC"
# Enable perm-gen class unloading (needed with UseConcMarkSweepGC)
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+CMSClassUnloadingEnabled"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+UseTLAB"

## Non Heap memory
# PermSize controls area of heap for Class/Method objects; 
#   Dynamic class loading/reflection (e.g., JSP) may require more space 
#   Note that this space is ABOVE and beyond the min/max heap size 
JAVA_OPTIONS="$JAVA_OPTIONS -XX:PermSize=32m"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:MaxPermSize=128m"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:CodeCacheMinimumFreeSpace=8m"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:ReservedCodeCacheSize=128m"

## Heap memory
# We set heap min/max to same size for consistent results
JAVA_OPTIONS="$JAVA_OPTIONS -Xms512m"
JAVA_OPTIONS="$JAVA_OPTIONS -Xmx2000m"
# We set new area to 1/4 heap
JAVA_OPTIONS="$JAVA_OPTIONS -XX:NewSize=128m"
JAVA_OPTIONS="$JAVA_OPTIONS -XX:MaxNewSize=128m"

## Solr specific optimisations (especially to avoid GC's)
# Reduce the number of objects getting promoted into the Old Gen, reducing fragmentation and CMS frequency & time
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseCompressedStrings"
# Use compressed pointers on a 64-bit JVM, reducing memory & performance penality in using 64bit JVM
JAVA_OPTIONS="$JAVA_OPTIONS -XX:+UseCompressedOops"
# Play with this - perhaps the JVM defaults are to lenient here. Under high load CSM would stall
JAVA_OPTIONS="$JAVA_OPTIONS -XX:CMSInitiatingOccupancyFraction=75"

Application Server Tuning:
- NONE -

SOLR Tuning - solrconfig.xml:
  <indexDefaults>
    <mergeFactor>5</mergeFactor>
    ...
  </indexDefaults>
  
  <mainIndex>
    <mergeFactor>5</mergeFactor>
    ...
  <mainIndex>
  
  <query>
    <filterCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="4096"/>

    <queryResultCache class="solr.LRUCache" size="16384" initialSize="4096" autowarmCount="1024"/>

    <documentCache class="solr.LRUCache" size="16384" initialSize="16384" autowarmCount="0"/>
    
   <queryResultWindowSize>50</queryResultWindowSize>

   <queryResultMaxDocsCached>2000</queryResultMaxDocsCached>
    
    ...
  </query>