[jira] [Updated] (CASSANDRA-9019) GCInspector detected GC before ThreadPools are initialized

["Yuki Morishita (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuki Morishita updated CASSANDRA-9019:
--------------------------------------
    Attachment: 9019.txt

In this case, I think it's better to return string "N/A" (not available) rather than throwing RuntimeException when Stages are not initialized and JMX instances are not ready, so that we still can display it in StatusLogger output.

I cannot reproduce with GC, but I used {{nodetool tpstats}} because it uses the same method in ThreadPoolMetrics. If you do {{nodetool tpstats}} right after the node starts, you get similar error as follows:

{code}
$ bin/nodetool -p 7100 tpstats
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
error: org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=ActiveTasks
-- StackTrace --
javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=ActiveTasks
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1464)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        -- snip --
        at org.apache.cassandra.metrics.ThreadPoolMetrics.getJmxMetric(ThreadPoolMetrics.java:123)
        at org.apache.cassandra.tools.NodeProbe.getThreadPoolMetric(NodeProbe.java:1000)
        at org.apache.cassandra.tools.NodeTool$TpStats.execute(NodeTool.java:2630)
        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:256)
        at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
{code}

After the patch, it shows "N/A" for not yet ready thread pools.

> GCInspector detected GC before ThreadPools are initialized
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-9019
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9019
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Yuki Morishita
>             Fix For: 3.0
>
>         Attachments: 9019.txt
>
>
> While running the dtest {{one_all_test (consistency_test.TestConsistency)}}, I ran into the following exception:
> {code}
> java.lang.RuntimeException: Error reading: org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=ActiveTasks
>         at org.apache.cassandra.metrics.ThreadPoolMetrics.getJmxMetric(ThreadPoolMetrics.java:134)
>         at org.apache.cassandra.utils.StatusLogger.log(StatusLogger.java:55)
>         at org.apache.cassandra.service.GCInspector.handleNotification(GCInspector.java:147)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754)
>         at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
>         at sun.management.GarbageCollectorImpl.createGCNotification(GarbageCollectorImpl.java:150)
> Caused by: java.lang.reflect.UndeclaredThrowableException
>         at com.sun.proxy.$Proxy3.getValue(Unknown Source)
>         at org.apache.cassandra.metrics.ThreadPoolMetrics.getJmxMetric(ThreadPoolMetrics.java:123)
>         ... 5 more
> Caused by: javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=ActiveTasks
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
>         at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:267)
>         ... 7 more
>     
> {code}
> Dtest didn't preserve the logs, which implies that this wasn't in the system.log, but printed to stderr somehow, it's unclear with all the piping dtest and ccm do. I have yet to reproduce the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)