[jira] [Commented] (CASSANDRA-9662) compactionManager reporting wrong pendingtasks

["Yuki Morishita (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603463#comment-14603463 ] 

Yuki Morishita commented on CASSANDRA-9662:
-------------------------------------------

patch here: https://github.com/yukim/cassandra/tree/9662
ci: http://cassci.datastax.com/job/yukim-9662-testall/2/ and http://cassci.datastax.com/job/yukim-9662-dtest/2/

Number of compaction pending tasks has been calculated based on estimation reported from compaction strategy and the number of tasks queued for a long time.
The latter, tasks queued, is mostly compaction task *candidate* ({{BackgroundCompactionTask}}) that can be noop.
We shoud have used active compaction task count instead.


> compactionManager reporting wrong pendingtasks
> ----------------------------------------------
>
>                 Key: CASSANDRA-9662
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9662
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>         Environment: OS: Amazon Linux AMI release 2015.03
> Cassandra: 2.0.16
> JVM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25, mixed mode)
> Java: version 1.8.0_40, vendor Oracle Corporation
> CPU: 8 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
> Memory: 32G
>            Reporter: Tony Xu
>            Assignee: Yuki Morishita
>            Priority: Minor
>              Labels: performance
>             Fix For: 2.0.16
>
>         Attachments: node1.jpg
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Yesterday I upgraded my Cassandra cluster from 2.0.14 to 2.0.16, after upgrade, I am start seeing some strange behaviours of "PendingTasks" reporting.
> The Cassandra repository I am using is datastax, steps I performed for upgrade:
> yum update -y cassandra20
> The upgrade went fine, after upgrade cluster is operating okay. "nodetool info and nodetool status" results looked fine. "nodetool version" is reporting the correct version.
> But our monitoring system start reporting some crazy "pendingtasks". For example, "pending taks" for node1 sometimes jump from 0 to 15K for about 1 minute, then drop back to 0. This issue keeps occurring, didn't have this issue with 2.0.14. Our monitoring system is checking the value of "MBeans" -> "CompactionManager" -> "PendingTasks".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)