[jira] [Comment Edited] (CASSANDRA-6541) New versions of Hotspot create new Class objects on every JMX connection causing the heap to fill up with them if CMSClassUnloadingEnabled isn't set.

["Rekha Joshi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341346#comment-14341346 ] 

Rekha Joshi edited comment on CASSANDRA-6541 at 2/28/15 11:08 PM:
------------------------------------------------------------------

On DSE 4.5.1, Java 1.7, linux ami on aws  encountered long running GC before dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load were run.The error log attached.

I suspect to have hit that hotspot/java GC issue.But in this case the class unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75

Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging enabled and is stable.However if there is a heap leak, it needs a better fix.
Suggestion: If the CPU utilization is going up, or heap size leak/long running GC is sensed, the nodetool -h localhost flush can auto kick in?

Thanks
Rekha



was (Author: rekhajoshm):
On DSE 4.5.1, Java 1.7, linux ami on aws  encountered long running GC before dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load were run.The error log attached.

I suspect to have hit that hotspot/java GC issue.But in this case the class unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75

Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging enabled and is stable.However if there is a heap leak, it needs a better fix.

Thanks
Rekha


> New versions of Hotspot create new Class objects on every JMX connection causing the heap to fill up with them if CMSClassUnloadingEnabled isn't set.
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6541
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6541
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Config
>            Reporter: jonathan lacefield
>            Assignee: Brandon Williams
>            Priority: Minor
>             Fix For: 1.2.16, 2.0.6, 2.1 beta2
>
>         Attachments: dse_systemlog
>
>
> Newer versions of Oracle's Hotspot JVM , post 6u43 (maybe earlier) and 7u25 (maybe earlier), are experiencing issues with GC and JMX where heap slowly fills up overtime until OOM or a full GC event occurs, specifically when CMS is leveraged.  Adding:
> {noformat}
> JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
> {noformat}
> The the options in cassandra-env.sh alleviates the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)