[jira] [Created] (CASSANDRA-18443) Deadlock updating sstable metadata if disk boundaries need reloading

["Jon Meredith (Jira)" <ji...@apache.org>]

Jon Meredith created CASSANDRA-18443:
----------------------------------------

             Summary: Deadlock updating sstable metadata if disk boundaries need reloading
                 Key: CASSANDRA-18443
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-18443
             Project: Cassandra
          Issue Type: Improvement
          Components: Local/Compaction, Local/Memtable, Local/SSTable
            Reporter: Jon Meredith
            Assignee: Jon Meredith


{{CompactionStrategyManager.handleNotification}} holds the read lock while processing notifications. When handling metadata changed notifications, an extra call is made to maybeReloadDiskBoundaries which tries to grab the write lock and deadlocks the thread.

Partial stacktrace
{code}
        at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
        - parking to wait for  <0x00000005cc000078> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.maybeReloadDiskBoundaries(CompactionStrategyManager.java:495)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.getCompactionStrategyFor(CompactionStrategyManager.java:343)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleMetadataChangedNotification(CompactionStrategyManager.java:796)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleNotification(CompactionStrategyManager.java:838)
        at org.apache.cassandra.db.lifecycle.Tracker.notifySSTableMetadataChanged(Tracker.java:482)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleNotification(CompactionStrategyManager.java:838)
{code}

Deadlocking with the read lock held blocks the SlabpoolCleaner while notifying ColumnFamilyStore so memtables are prevented from being flushed and recycled, causing any thread applying a mutation to the database (at least GossipStage and MutationStage) to be considered down by peers and/or back up with pending requests.

All the cases investigated were during single sstable upleveling by {{org.apache.cassandra.db.compaction.SingleSSTableLCSTask}} added in CASSANDRA-12526.

Other less critical work was also affected, JMX calls to get estimated remaining compaction tasks, the index summary manager redistributing summaries, the StatusLogger trying to log dropped messages, and the ValidationManager.

Workaround is to reboot the affected host.

The fix is to just remove the redundant disk boundary reload check on that path.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org