You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Gaurav Agarwal (Jira)" <ji...@apache.org> on 2024/10/29 19:13:00 UTC
[jira] [Commented] (CASSANDRA-19564) MemtablePostFlush deadlock leads to stuck nodes and crashes
[ https://issues.apache.org/jira/browse/CASSANDRA-19564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893965#comment-17893965 ]
Gaurav Agarwal commented on CASSANDRA-19564:
--------------------------------------------
Hi folks, we are facing the same issue on 4.1.3 quite frequently on a given cassandra cluster.
When a new node tries to join the ring during node replacement, *all* its mutationstage threads are stuck indefinitely on memtable allocator pool. There is no write spike on coordinator node but replica writes are piling up on the new node. Any ideas why this might be happening?
Below is the threaddump:
{code:java}
"MutationStage-1" - Thread t@92
java.lang.Thread.State: WAITING
at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131)
at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235)
at app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:195)
at app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.adjust(MemtableAllocator.java:165)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.finish(AtomicBTreePartition.java:448)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:152)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:190)
at app//org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:134)
at app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1431)
at app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
at app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:783)
at app//org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:580)
at app//org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:251)
at app//org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:73)
at app//org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:79)
at app//org.apache.cassandra.net.InboundSink$$Lambda$597/0x00007f00af4dc040.accept(Unknown Source)
at app//org.apache.cassandra.net.InboundSink.accept(InboundSink.java:98)
at app//org.apache.cassandra.net.InboundSink.accept(InboundSink.java:46)
at app//org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
at app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at app//org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
Locked ownable synchronizers:
- None
{code}
> MemtablePostFlush deadlock leads to stuck nodes and crashes
> -----------------------------------------------------------
>
> Key: CASSANDRA-19564
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19564
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction, Local/Memtable
> Reporter: Jon Haddad
> Priority: Urgent
> Fix For: 4.1.x
>
> Attachments: image-2024-04-16-11-55-54-750.png, image-2024-04-16-12-29-15-386.png, image-2024-04-16-13-43-11-064.png, image-2024-04-16-13-53-24-455.png, image-2024-04-17-18-46-29-474.png, image-2024-04-17-19-13-06-769.png, image-2024-04-17-19-14-34-344.png, screenshot-1.png
>
>
> I've run into an issue on a 4.1.4 cluster where an entire node has locked up due to what I believe is a deadlock in memtable flushing. Here's what I know so far. I've stitched together what happened based on conversations, logs, and some flame graphs.
> *Log reports memtable flushing*
> The last successful flush happens at 12:19.
> {noformat}
> INFO [NativePoolCleaner] 2024-04-16 12:19:53,634 AbstractAllocatorMemtable.java:286 - Flushing largest CFS(Keyspace='ks', ColumnFamily='version') to free up room. Used total: 0.24/0.33, live: 0.16/0.20, flushing: 0.09/0.13, this: 0.13/0.15
> INFO [NativePoolCleaner] 2024-04-16 12:19:53,634 ColumnFamilyStore.java:1012 - Enqueuing flush of ks.version, Reason: MEMTABLE_LIMIT, Usage: 660.521MiB (13%) on-heap, 790.606MiB (15%) off-heap
> {noformat}
> *MemtablePostFlush appears to be blocked*
> At this point, MemtablePostFlush completed tasks stops incrementing, active stays at 1 and pending starts to rise.
> {noformat}
> MemtablePostFlush 1 1 3446 0 0
> {noformat}
>
> The flame graph reveals that PostFlush.call is stuck. I don't have the line number, but I know we're stuck in {{org.apache.cassandra.db.ColumnFamilyStore.PostFlush#call}} given the visual below:
> *!image-2024-04-16-13-43-11-064.png!*
> *Memtable flushing is now blocked.*
> All MemtableFlushWriter threads are Parked waiting on {{{}OpOrder.Barrier.await{}}}. A wall clock profile of 30s reveals all time is spent here. Presumably we're waiting on the single threaded Post Flush.
> !image-2024-04-16-12-29-15-386.png!
> *Memtable allocations start to block*
> Eventually it looks like the NativeAllocator stops successfully allocating memory. I assume it's waiting on memory to be freed, but since memtable flushes are blocked, we wait indefinitely.
> Looking at a wall clock flame graph, all writer threads have reached the allocation failure path of {{MemtableAllocator.allocate()}}. I believe we're waiting on {{signal.awaitThrowUncheckedOnInterrupt()}}
> {noformat}
> MutationStage 48 828425 980253369 0 0{noformat}
> !image-2024-04-16-11-55-54-750.png!
>
> *Compaction Stops*
> Since we write to the compaction history table, and that requires memtables, compactions are now blocked as well.
>
> !image-2024-04-16-13-53-24-455.png!
>
> The node is now doing basically nothing and must be restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org