Crazy Solr startup params, please review

[Uwe Schindler <uw...@thetaphi.de>]

Hi,

I just reviewed the solr.sh.in and solr.cmd.in scripts in solr/bin folder. The following was committed but never discussed:

# These GC settings have shown to work well for a number of common Solr workloads
GC_TUNE="-XX:-UseSuperWord \
-XX:NewRatio=3 \
-XX:SurvivorRatio=4 \
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:+CMSScavengeBeforeRemark \
-XX:PretenureSizeThreshold=64m \
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=50 \
-XX:CMSTriggerPermRatio=80 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled \
-XX:+AggressiveOpts"

This is horrible, because of our experience with Hotspot bugs:

-XX:+AggressiveOpts
This option is veeeery risky and speed improvements are marginal. PLEASE DON'T DO THIS. If people want the new features they should wait for later Java releases and the new features are tested. See several tasks about the Java 7 disaster! In fact we had reports at Java 6 times when people had enabled this and were affected by the Java 7 GA bugs already in Java 6 and corrupted their indexes!!!

-XX:-UseSuperWord
If you have a Haswell CPU, all other improvements in these command line settings are eaten up by this flag! If you have 7u55 at minimum, you should never disable this. Things like BooleanFilter and other bitset operations are up to 2 times faster with Java 7u55 on Haswell CPUs and later!!! This setting only makes sense if you have one of those buggy JDKs (7u40 to 7u51). In all other cases this slows down enormous!
In addition, enabling this option may break JDKs before 7u40 (this option was added in 7u40), so breaks:
> Unrecognized VM option 'UseSuperWord'

It would be good, if we could fix the startup scripts not not have options, which may also break with JDK 8 or later!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Crazy Solr startup params, please review

[Jan Høydahl <ja...@cominvent.com>]

Can we in a smart way build these best practices into the script, by testing JVM version and (un)setting some of these automatically based on version? The solr.in.* config could have a new option GC_TUNE_AUTO="true" and GC_TUNE commented out. Then if auto is enabled, the bin/solr scripts will decide what flags to set. Experts can override. Also related is SOLR-6705, some options should be set automatically only for some JVM versions.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 5. nov. 2014 kl. 16.53 skrev Uwe Schindler <uw...@thetaphi.de>:
> 
> Hi,
> 
> I just reviewed the solr.sh.in and solr.cmd.in scripts in solr/bin folder. The following was committed but never discussed:
> 
> # These GC settings have shown to work well for a number of common Solr workloads
> GC_TUNE="-XX:-UseSuperWord \
> -XX:NewRatio=3 \
> -XX:SurvivorRatio=4 \
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:+UseParNewGC \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:+CMSScavengeBeforeRemark \
> -XX:PretenureSizeThreshold=64m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=50 \
> -XX:CMSTriggerPermRatio=80 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled \
> -XX:+ParallelRefProcEnabled \
> -XX:+AggressiveOpts"
> 
> This is horrible, because of our experience with Hotspot bugs:
> 
> -XX:+AggressiveOpts
> This option is veeeery risky and speed improvements are marginal. PLEASE DON'T DO THIS. If people want the new features they should wait for later Java releases and the new features are tested. See several tasks about the Java 7 disaster! In fact we had reports at Java 6 times when people had enabled this and were affected by the Java 7 GA bugs already in Java 6 and corrupted their indexes!!!
> 
> -XX:-UseSuperWord
> If you have a Haswell CPU, all other improvements in these command line settings are eaten up by this flag! If you have 7u55 at minimum, you should never disable this. Things like BooleanFilter and other bitset operations are up to 2 times faster with Java 7u55 on Haswell CPUs and later!!! This setting only makes sense if you have one of those buggy JDKs (7u40 to 7u51). In all other cases this slows down enormous!
> In addition, enabling this option may break JDKs before 7u40 (this option was added in 7u40), so breaks:
>> Unrecognized VM option 'UseSuperWord'
> 
> It would be good, if we could fix the startup scripts not not have options, which may also break with JDK 8 or later!
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org