[jira] [Commented] (CASSANDRA-9662) compactionManager reporting wrong pendingtasks

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617137#comment-14617137 ] 

Benedict commented on CASSANDRA-9662:
-------------------------------------

This patch looks like it would acceptably fix the bouncing report of compaction tasks outstanding.

However I think we should step back for a moment and consider the context. The reason this is bouncing around is because we attempt to impose a constraint in {{submitBackground}} that we enforce very badly. This leads us to accidentally submit perhaps thousands of no-op compactions. This patch masks that from the user, but we're still doing it. This constraint being enforced _at all_ is very questionable, though. It can lead us to starving entire tables from compaction for extended periods, because they happen to flush at periods during which the compaction executors are fully occupied. Ironically, CASSANDRA-9592 both mitigates and worsens this problem, depending on if you're a secondary index or not (for whom it makes things worse; everyone else it should prevent starvation for)

So: how about we always submit exactly one background task for any table whenever one is requested. That way the numbers will never bounce around so wildly: they'll be capped at a variance of about 1 per table, per flush or per minute. We can (and perhaps should) also introduce this patch, but it would be comparatively much less of a problem, and we would have fixed a potential problematic situation where tables get very far behind on compaction.

> compactionManager reporting wrong pendingtasks
> ----------------------------------------------
>
>                 Key: CASSANDRA-9662
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9662
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>         Environment: OS: Amazon Linux AMI release 2015.03
> Cassandra: 2.0.16
> JVM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25, mixed mode)
> Java: version 1.8.0_40, vendor Oracle Corporation
> CPU: 8 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
> Memory: 32G
>            Reporter: Tony Xu
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 2.0.x
>
>         Attachments: node1.jpg
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Yesterday I upgraded my Cassandra cluster from 2.0.14 to 2.0.16, after upgrade, I am start seeing some strange behaviours of "PendingTasks" reporting.
> The Cassandra repository I am using is datastax, steps I performed for upgrade:
> yum update -y cassandra20
> The upgrade went fine, after upgrade cluster is operating okay. "nodetool info and nodetool status" results looked fine. "nodetool version" is reporting the correct version.
> But our monitoring system start reporting some crazy "pendingtasks". For example, "pending taks" for node1 sometimes jump from 0 to 15K for about 1 minute, then drop back to 0. This issue keeps occurring, didn't have this issue with 2.0.14. Our monitoring system is checking the value of "MBeans" -> "CompactionManager" -> "PendingTasks".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)