You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Loic Lambiel (JIRA)" <ji...@apache.org> on 2017/10/29 09:06:03 UTC

[jira] [Created] (CASSANDRA-13980) Compaction deadlock

Loic Lambiel created CASSANDRA-13980:
----------------------------------------

             Summary: Compaction deadlock
                 Key: CASSANDRA-13980
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13980
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction
         Environment: Cassandra 3.11.2
48 nodes cluster using LCS, JBOD
Big nodes with many SSTables
            Reporter: Loic Lambiel
            Priority: Critical
             Fix For: 3.11.x
         Attachments: threaddump.log

While upgrading the cluster from 2.1.16 from 3.11.2, after a few hours most of the upgraded nodes started to go in a compaction infinite loop and showing many events like the one below (always for the same SSTable):

{code:java}
INFO  [CompactionExecutor:4] 2017-10-29 00:28:31,480 LeveledManifest.java:474 - Adding high-level (L5) BigTableReader(path='/var/lib/cassandra/data/datadisk4/blobstore/block-1d63273065b911e49cd7ef0972cffde6/blobstore-block-ka-201694-Data.db') to candidates
{code}

Since the log get spammed at a huge rate, I'm unable to get any previous events.

Tried restarts and sstablescrub -m without success. The only workaround that seems to work (so far) was sstablelevelreset.

I've attached the dump.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org