[jira] [Comment Edited] (CASSANDRA-7432) Add new CMS GC flags to cassandra_env.sh for JVM later than 1.7.0_60

["graham sanderson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039890#comment-14039890 ] 

graham sanderson edited comment on CASSANDRA-7432 at 6/21/14 4:37 PM:
----------------------------------------------------------------------

Here are some performance numbers(we have an unusually large heap for C*: 8gig new, 16g old):

h3. +CMSParallelInitialMarkEnabled

||Initial Mark STW pauses(secs)|Min|Median|Mean|Max|StdDev||
|Without +CMSParallelInitialMarkEnabled|0.000|0.050|0.109|0.880|0.159|
|With +CMSParallelInitialMarkEnabled|0.000|0.010|0.019|0.090|0.019|

Note this flag is just the new cousin of {{-XX:*CMSParallelRemarkEnabled}}

h3. +CMSEdenChunksRecordAlways

||Remark STW pauses(secs)|Min|Median|Mean|Max|StdDev||
|Without +CMSEdenChunksRecordAlways|0.010|0.470|1.049|6.580|1.394|
|With +CMSEdenChunksRecordAlways|0.010|0.135|0.148|0.520|0.114|

Note this option addresses the issue loosely described [here|https://blogs.oracle.com/jonthecollector/entry/did_you_know] in the *CMSMaxAbortablePrecleanTime* section. The short version of the problem is that if you are unlucky (and bursty activity in your cluster will make you less lucky, then even though you have {{-XX:*CMSParallelRemarkEnabled}}, you may get very low parallelism in your remark STW *and* you may have a very full eden when you are doing it.


was (Author: graham sanderson):
Here are some performance numbers(we have an unusually large heap for C*: 8gig new, 16g old):

h3. +CMSParallelInitialMarkEnabled

||Initial Mark STW pauses(secs)|Min|Median|Mean|Max|StdDev||
|Without +CMSParallelInitialMarkEnabled|0.000|0.050|0.109|0.880|0.159|
|With +CMSParallelInitialMarkEnabled|0.000|0.010|0.019|0.090|0.019|

h3. +CMSEdenChunksRecordAlways

||Remark STW pauses(secs)|Min|Median|Mean|Max|StdDev||
|Without +CMSEdenChunksRecordAlways|0.010|0.470|1.049|6.580|1.394|
|With +CMSEdenChunksRecordAlways|0.010|0.135|0.148|0.520|0.114|

Note this option addresses the issue loosely described [here|https://blogs.oracle.com/jonthecollector/entry/did_you_know] in the *CMSMaxAbortablePrecleanTime* section. The short version of the problem is that if you are unlucky (and bursty activity in your cluster will make you less lucky, then even though you have {{-XX:*CMSParallelRemarkEnabled}}, you may get very low parallelism in your remark STW *and* you may have a very full eden when you are doing it.

> Add new CMS GC flags to cassandra_env.sh for JVM later than 1.7.0_60
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-7432
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7432
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Packaging
>            Reporter: graham sanderson
>
> The new flags in question are as follows:
> {code}
> -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSEdenChunksRecordAlways
> {code}
> Given we already have
> {code}
> JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" 
> JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" 
> JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" 
> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
> if [ "$JVM_ARCH" = "64-Bit" ] ; then
>     JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
> fi
> {code}
> The assumption would be that people are at least running on large number CPU cores/threads
> I would therefore recommend defaulting these flags if available - the only two possible downsides for {{+CMSEdenChunksRecordAlways}}:
> 1) There is a new very short (probably un-contended) lock in the "slow" (non TLAB) eden allocation path with {{+CMSEdenChunksRecordAlways}}. I haven't detected this timing wise - this is the "slow" path after all
> 2) If you are running with {{-XX:-UseCMSCompactAtFullCollection}} (not the default) *and* you call {{System.gc()}} then  {{+CMSEdenChunksRecordAlways}} will expose you to a possible seg fault: (see
> [http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021809])



--
This message was sent by Atlassian JIRA
(v6.2#6252)