You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bowen Song via user <us...@cassandra.apache.org> on 2024/11/05 12:12:20 UTC
Unexplained stuck memtable flush
Hi all,
We have a cluster running Cassandra 4.1.1. We are seeing the memtable
flush randomly getting stuck. This has happened twice in the last 10
days, to two different nodes in the same cluster. This started to happen
after we enabled CDC, and each time it got stuck, there was at least one
repair running involving the affected node.
The signs of the stuck memtable flush is most obvious from the
"StatusLogger" logs.
At the beginning, the MemtablePostFlush and MemtableFlushWriter got
stuck, they have 1 and 2 active tasks each, and a small number of
pending tasks.
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
ReadStage 0 0 34052333 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
CompactionExecutor 0 0 1019777 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
MutationStage 0 0 14930764 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
PendingRangeCalculator 0 0 177
0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
GossipStage 0 0 889452 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 1
0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
HintsDispatcher 0 0 19 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
Repair-Task 0 0 65 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
RequestResponseStage 0 0 44447834 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
Native-Transport-Requests 0 0 8967921
0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
MemtableFlushWriter 2 10 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
MemtablePostFlush 1 11 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 21769
0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
InternalResponseStage 0 0 240 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
AntiEntropyStage 1 1773 120067 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
The number of pending tasks slowly grows larger over time, and the
number of completed tasks do not increase at all.
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
ReadStage 0 0 39905462 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
CompactionExecutor 0 0 1170100 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
MutationStage 0 0 16976992 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
GossipStage 0 0 1026877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair-Task 0 0 83 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
RequestResponseStage 0 0 51122191 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
PendingRangeCalculator 0 0 177
0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 1
0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Native-Transport-Requests 0 0 10145020
0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 21769
0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
InternalResponseStage 0 0 996 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
AntiEntropyStage 1 9694 120067 0 0
Then, things start to get worse, the MutationStage also got stuck.
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
ReadStage 0 0 40186426 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
CompactionExecutor 0 0 1171479 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
MutationStage 128 201414 16988212
0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
GossipStage 0 0 1032988 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair-Task 0 0 85 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
RequestResponseStage 0 0 51307497 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#84 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
PendingRangeCalculator 0 0 177
0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 1
0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Native-Transport-Requests 0 0 10194549
0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 21769
0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
InternalResponseStage 0 0 1087 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
AntiEntropyStage 1 10364 120067 0 0
Writes on this node starts to timeout and fail. But if left untouched,
it's only gonna get worse, and eventually lead to JVM OOM and crash.
By inspecting the heap dump created at OOM, we can see that both of the
MemtableFlushWriter threads are stuck on line 1190
<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
in the ColumnFamilyStore.java:
// mark writes older than the barrier as blocking progress,
permitting them to exceed our memory limit
// if they are stuck waiting on it, then wait for them all
to complete
writeBarrier.markBlocking();
writeBarrier.await(); // <----------- stuck here
And the MemtablePostFlush thread is stuck on line 1094
<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
in the same file.
try
{
// we wait on the latch for the commitLogUpperBound to
be set, and so that waiters
// on this task can rely on all prior flushes being
complete
latch.await();// <----------- stuck here
}
Our top suspect is CDC interacting with repair, since this started to
happen shortly after we enabled CDC on the nodes, and each time repair
was running. But we have not been able to reproduce this in a testing
cluster, and don't know what's the next step to troubleshoot this issue.
So I'm posting it in the mailing lists and hoping someone may know
something about it or point me to the right direction.
p.s.: sorry about posting this to both the user & dev mailing lists.
It's an end-user related issue but involves Cassandra internals, so I
can't decide which one is best suited.
Cheers,
Bowen
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Few ideas about timing and timestamps extraction from the heap dump:
1) Here
https://github.com/apache/cassandra/blob/c1d89c32d27921d1f77f05d29ee248b8922a4c76/src/java/org/apache/cassandra/db/Keyspace.java#L627
you can check the time when the write operation within
"read-hotness-tracker:1" started (to understand how long we have been
staying within the write logic at the moment of heap dump creation).
2)
https://github.com/apache/cassandra/blob/6a1d9de3d938d24baa2ba98b5379b0536177c786/src/java/org/apache/cassandra/db/commitlog/PeriodicCommitLogService.java#L39
you can calculate here the value of delta between expectedSyncTime and
lastSyncedAt nanoTime values (to understand for how long the commit log
sync was delayed in ns)
3) You can also get the local timestamp var value here to see the absolute
timestamp when the query execution started:
https://github.com/apache/cassandra/blob/b3842de5cf1fa1b81872effb4585fbc7e1873d59/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java#L6723
)
Regarding WaitQueue
what are the values for
org.apache.cassandra.utils.concurrent.WaitQueue.Standard.RegisteredSignal#
*state*
org.apache.cassandra.utils.concurrent.WaitQueue.Standard.RegisteredSignal#
*thread*
in the object here?:
"read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
Regards,
Dmitry
On Thu, 7 Nov 2024 at 21:30, Dmitry Konstantinov <ne...@gmail.com> wrote:
> Hi Bowen,
>
> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
> where long pollStarted = clock.now();
> the logic uses clock from: SyncRunnable sync = new
> SyncRunnable(preciseTime)
> preciseTime by default is SystemClock
> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
> System.nanoTime()
> so, the field is populated from System.nanoTime() which returns a relative
> timestamp, which cannot be treated as an absolute time like linux epoch (it
> is calculated like this:
> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
> )
>
> Can you share your cassandra.yaml file? (with removed IPs, paths and other
> security sensitive things, of course)
> Do you have any exceptions in your system.log related to commit log?
> Do you have JVM stdout/stderr captured to a file? Does it have any
> exceptions? (which can be present there if Cassandra does not catch them
> properly by itself)
>
> ---
> How many CommitLogSegment objects do you have in your heap dump?
> What are values for the following fields of CommitLogSegment objects?
> lastSyncedOffset
> lastMarkerOffset
> cdcState
>
> Do you have CDC index files written by
> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
> If you have them - what are the last offsets written there?
>
> Regards,
> Dmitry
>
> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue
>> is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>> There is no item in the queue's linked list, only the head and tail nodes,
>> each has item=null.
>>
>> The usage of the WaitQueue within the AbstractCommitLogService looks
>> correct to me:
>>
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly();
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>>
>> This matches the example given in the WaitQueue's documentation (which is
>> definitely missing a pair of curly brackets and has bad indentation):
>>
>> while (!conditionMet())
>> Signal s = q.register();
>> if (!conditionMet()) // or, perhaps more correctly,
>> !conditionChanged()
>> s.await();
>> else
>> s.cancel();
>>
>> So, why is the queue empty while the thread is blocked on waiting for the
>> signal?
>>
>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>> about 5 years ago.
>>
>> Is this suppose to be a wall clock time in milliseconds since 1 January
>> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
>> context, and I'm just wrongly interpreting its value as a wall clock time?
>>
>>
>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>> object believes that there's nothing waiting for the signal, yet the
>> "read-hotness-tracker:1" thread is clearly waiting for it.
>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>
>> I think I'm getting really close now. This seems to have something to do
>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> <------ here
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> <------ and here
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>> at
>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>> local variable:
>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>> local variable:
>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> local variable:
>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>> at java.lang.Thread.run(Thread.java:829)
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>
>>
>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>
>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>> throws RequestExecutionException
>> {
>> OpOrder.Group group = null;
>> try
>> {
>> group = Keyspace.writeOrder.start(); // <--- here
>>
>> // write the mutation to the commitlog and memtables
>> CommitLogPosition position = null;
>> if (makeDurable)
>> {
>> position = addToCommitLog(mutation); // <--- and here
>> }
>> return new CassandraWriteContext(group, position);
>> }
>> catch (Throwable t)
>> {
>> if (group != null)
>> {
>> group.close();
>> }
>> throw t;
>> }
>> }
>>
>> The Keyspace.writeOrder.start() method call blocks the
>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>
>> void awaitSyncAt(long syncTime, Context context)
>> {
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly(); // <--- here
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> }
>>
>> This then got stuck on the signal.awaitUninterruptibly()
>>
>> Now I know what is blocking the memtable flushing, but I haven't been
>> able to figure out is why it got stuck on waiting for that signal.
>>
>> I would appreciate it if anyone can offer some insight here.
>>
>>
>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go down the
>> rabbit hole tomorrow. Will keep you updated.
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>
>>
>>
>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>
>> Writes on this node starts to timeout and fail. But if left untouched,
>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that both of the
>> MemtableFlushWriter threads are stuck on line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking progress,
>> permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for them all to
>> complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the commitLogUpperBound to be
>> set, and so that waiters
>> // on this task can rely on all prior flushes being
>> complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this started to
>> happen shortly after we enabled CDC on the nodes, and each time repair was
>> running. But we have not been able to reproduce this in a testing cluster,
>> and don't know what's the next step to troubleshoot this issue. So I'm
>> posting it in the mailing lists and hoping someone may know something about
>> it or point me to the right direction.
>>
>>
>> Wouldn’t be completely surprised if CDC or repair somehow has a
>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>> read commands” that have a barrier on the memtable that prevent release.
>>
>> You’ve got the heap (great, way better than most people debugging), are
>> you able to navigate through it and look for references to that memtable or
>> other things holding a barrier?
>>
>>
>>
>>
>>
>
> --
> Dmitry Konstantinov
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
The issue is fixed now
On Fri, 15 Nov 2024 at 12:09, Dmitry Konstantinov <ne...@gmail.com>
wrote:
> Thank you!
> A ticket for the issue:
> https://issues.apache.org/jira/browse/CASSANDRA-20084
> <https://issues.apache.org/jira/browse/CASSANDRA-20084>
>
> Regards,
> Dmitry
>
> On Thu, 14 Nov 2024 at 10:33, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Impressive! It all makes sense now.
>>
>> I believe this bug was accidentally introduced back in 2021 as a part of
>> CASSANDRA-16924 <https://issues.apache.org/jira/browse/CASSANDRA-16924>,
>> commit e5b92e1
>> <https://github.com/apache/cassandra/commit/e5b92e108851d2be81c43cbf982564f92d28aa6b#diff-a8247a86ff78ae4f477adbc7f9e93c6375e7d689a1265f26470fb9c42af74995>,
>> which removed the awaitUninterruptibly() method from the
>> WaitQueue$AbstractSignal in favour of the newly introduced
>> org.apache.cassandra.utils.concurrent.Awaitable class and split the code
>> below into two places.
>>
>> public void awaitUninterruptibly()
>> {
>> boolean interrupted = false;
>> while (!isSignalled())
>> {
>> if (Thread.interrupted())
>> interrupted = true;
>> LockSupport.park();
>> }
>> if (interrupted)
>> Thread.currentThread().interrupt();
>> checkAndClear();
>> }
>>
>> It's not impossible that someone runs into the same issue twice in a row
>> in a short window of time, but I wouldn't expect this to occur frequently
>> enough to catch most people's attentions. Because it rarely occurs and can
>> be "fixed" by simply restarting Cassandra, it has stayed in the codebase
>> for years without being discovered. I'm glad that we took the opportunity
>> to deep dive into this and found the root cause.
>>
>> Good job, Dmitry, and thank you for helping me on this.
>>
>>
>> On 13/11/2024 23:48, Dmitry Konstantinov wrote:
>>
>> Thank you. I think I have got the cause, it is a non-trivial puzzle which
>> took a few hours :-).
>>
>> So, we have the following stacktrace for "read-hotness-tracker:1" thread
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>
>> where we are waiting in the await() method of
>> WaitQueue$Standard$SignalWithListener#2086139 object.
>>
>> another input: WaitQueue$Standard$SignalWithListener#2086139 has the
>> following field values based on the screenshots:
>> state = *-1* (it means CANCELLED:
>> org.apache.cassandra.utils.concurrent.WaitQueue.Standard#CANCELLED = -1)
>> thread = null
>>
>> According to the code in WaitQueue.java such combination of values only
>> possible if we invoked cancel() on this SignalWithListener object but we
>> didn't do it in our code, it is in another branch of
>> AbstractCommitLogService.awaitSyncAt
>>
>> but let's check the await() logic more carefully:
>> public Signal await() throws InterruptedException
>> {
>> while (!isSignalled())
>> {
>> checkInterrupted();
>> LockSupport.park(); <--- we are here
>> }
>> checkAndClear();
>> return this;
>> }
>> and inspect unassuming checkInterrupted() method
>> private void checkInterrupted() throws InterruptedException
>> {
>> if (Thread.interrupted())
>> {
>> *cancel();*
>> throw new InterruptedException();
>> }
>> }
>> here we see that if the currently awaiting thread is interrupted then
>> SignalWithListener cancels itself.
>> but we throw new InterruptedException() here, so the execution should
>> fail with it.
>> Checking the thread stack trace and see that AbstractSignal.await() is
>> invoked from
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>
>> how it looks inside:
>> public static <A extends Awaitable> A awaitUninterruptibly(A await)
>> {
>> boolean interrupted = false;
>> while (true)
>> {
>> try
>> {
>> await.await();
>> break;
>> }
>> catch (InterruptedException e)
>> {
>> interrupted = true;
>> }
>> }
>> if (interrupted)
>> Thread.currentThread().interrupt();
>> return await;
>> }
>> Here we have a loop, if the current thread is interrupted - we have
>> InterruptedException thrown by await.await(), we catch it here and try
>> await.await() again. But in the previous invocation of await.await() we
>> have cancelled our Signal object, so the condition inside await() method:
>> isSignalled(): state == SIGNALLED is always false now plus if another
>> thread will invoke notifyAll - nothing will happen, because as a part of
>> cancellation our Signal object unregister itself from the queue and the
>> notifying thread will not able to find it to notify.
>> I have reproduced the case with a simple unit test.
>>
>> The last element of the puzzle - why is the thread interrupted? To
>> understand it - let's check there "read-hotness-tracker:1" thread execution
>> started. It is a background periodical runnable in
>> org.apache.cassandra.io.sstable.format.SSTableReader.GlobalTidy
>> readMeterSyncFuture = new
>> WeakReference<>(syncExecutor.scheduleAtFixedRate(new Runnable()
>> {
>> public void run()
>> {
>> if (obsoletion == null)
>> {
>> meterSyncThrottle.acquire();
>> SystemKeyspace.persistSSTableReadMeter(desc.ksname,
>> desc.cfname, desc.id, readMeter);
>> }
>> }
>> }, 1, 5, TimeUnit.MINUTES));
>>
>> let's check how we deal with readMeterSyncFuture ref and we see our
>> interruption:
>> private void stopReadMeterPersistence()
>> {
>> ScheduledFuture<?> readMeterSyncFutureLocal =
>> readMeterSyncFuture.get();
>> if (readMeterSyncFutureLocal != null)
>> {
>> * readMeterSyncFutureLocal.cancel(true); *<========= here we can
>> interrupt the thread
>> readMeterSyncFuture = NULL;
>> }
>> }
>>
>> stopReadMeterPersistence() is invoked by one of the following branches of
>> logic when an SSTable is not in use anymore:
>> org.apache.cassandra.io.sstable.format.SSTableReader#markObsolete
>> org.apache.cassandra.db.lifecycle.Helpers#markObsolete
>> org.apache.cassandra.db.lifecycle.Tracker#dropSSTables(...)
>> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doCommit
>> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doAbort
>>
>> So,the issue is caused by a combination of events with a right timing:
>> - an SSTable which was become obsolete at the middle of
>> persistSSTableReadMeter background job execution for this SSTable, so we
>> interrupted the job
>> - persistSSTableReadMeter was at that moment in the middle of write
>> operation, at commit log append step
>> - and the commit log append step was stuck due to a delay in commit log
>> sync (lastSyncedAt < syncTime condition in AbstractCommitLogService), so we
>> started to wait for it using await
>>
>>
>> The last observation - timing.
>> We have
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService
>> lastSyncedAt = 1573038452910796
>>
>> com.codahale.metrics.Timer$Context#2086795 (created when the write
>> operation started to wait for commit log sync)
>> startTime = 1410129608743025
>>
>> both are from System.nanoTime()
>>
>> so, 1573038452910796 - 1410129608743025 = 162 908 844 167 771 ns = 162
>> 908 sec ~ 45 hours; the cause happened about that time ago.
>>
>> I will create a ticket for this issue to fix it.
>>
>> Regards,
>> Dmitry
>>
>>
>>
>> On Wed, 13 Nov 2024 at 22:11, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Please see the screenshots below.
>>>
>>> Unfortunately that the WaitQueue$Standard$SignalWithListener#2086139
>>> object is the GC root, and there's no references to it.
>>>
>>> I hope this helps.
>>>
>>>
>>>
>>>
>>> On 13/11/2024 21:33, Dmitry Konstantinov wrote:
>>>
>>> Hi Bowen, thank you for the details.
>>> Could you please also share a screenshot (Visual VM should be ok) for "read-hotness-tracker:1"
>>> thread with expanded local variable elements for the following lines:
>>> 1) at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
>>> particularly com.codahale.metrics.Timer$Context#2086795 internals such
>>> as startTime
>>> 2) at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
>>> particularly WaitQueue$Standard$SignalWithListener#2086139 internals
>>> such as thread, state
>>>
>>> Also, could you please render all GC paths to root for WaitQueue$Standard$SignalWithListener#2086139
>>> object and share the correspondent tree view?
>>>
>>> Regards,
>>> Dmitry
>>>
>>>
>>> On Tue, 12 Nov 2024 at 11:40, Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> Just tried Eclipse Memory Analyzer, it shows the exact same local
>>>> variables as VisualVM, so there is no difference here.
>>>>
>>>> However, Eclipse Memory Analyzer is significantly slower (lagging) to
>>>> operate via X window forwarding or VNC (possibly due to the GUI decorations
>>>> and animations) and some features rely on a built-in web browser which I
>>>> can't get it to work on a headless server. I'm doing the analysis on a
>>>> headless server because my PC doesn't have enough RAM to load the large
>>>> heap dump file.
>>>>
>>>> See screenshot below for a comparison between Eclipse Memory Analyzer
>>>> and VisualVM on the variables referenced by the PERIODIC-COMMIT-LOG-SYNCER
>>>> thread.
>>>>
>>>> Eclipse Memory Analyzer
>>>>
>>>> VisualVM
>>>>
>>>>
>>>> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>>>>
>>>> >> If Eclipse Memory Analyzer also can't find the mysterious item, how
>>>> do I proceed from there? What else can I do?
>>>> We can analyze other variables from the heap dump, read code and
>>>> repeat, until we find a clue :-). As of now I do not see a much better
>>>> option. If the issue reproduces regularly - a custom build with extra logs
>>>> can be done to add more diagnostic info printing.
>>>> It is actually lucky that you have a heap dump, it can give a lot of
>>>> info, unfortunately only in a moment of time (so we cannot see from the
>>>> issue in dynamic).
>>>>
>>>> >> This has happened twice in the last 10 days, to two different nodes
>>>> in the same cluster.
>>>> Do you have heap dumps for the other failures?
>>>> How much time did you have between a node startup and the failure?
>>>>
>>>> Can you share field values
>>>> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
>>>> using Eclipse Memory Analyzer
>>>> from the following thread?:
>>>> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#5
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
>>>> local variable: java.util.concurrent.Semaphore$NonfairSync#1
>>>> local variable:
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
>>>> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>>>>
>>>> It should looks like this:
>>>>
>>>> [image: image.png]
>>>>
>>>> [image: image.png]
>>>>
>>>> Regards,
>>>> Dmitry
>>>>
>>>>
>>>> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user <
>>>> user@cassandra.apache.org> wrote:
>>>>
>>>>> Hi Dmitry,
>>>>>
>>>>> It indeed is very strange, either the tool I'm using is wrong, or
>>>>> there's something causing the item to evaporate from the queue mysteriously
>>>>> or never got into the queue.
>>>>>
>>>>> I've been using VisualVM to inspect the heap dump file and using OQL
>>>>> to find and explore the objects in it.
>>>>>
>>>>> VisualVM does show some local variables, but not all of them. This
>>>>> isn't an issue for types with little number of instances, because I can
>>>>> search objects by type to find them. But it can be troublesome when an
>>>>> object is a very common type.
>>>>>
>>>>> I will try Eclipse Memory Analyzer next week to see if it can find the
>>>>> missing item, but I have my doubts.
>>>>>
>>>>> I've read the code of WaitQueue and the
>>>>> AbstractCommitLogService.awaitSyncAt() method, and they both look correct
>>>>> to me. If Eclipse Memory Analyzer also can't find the mysterious item, how
>>>>> do I proceed from there? What else can I do?
>>>>>
>>>>> Cheers,
>>>>> Bowen
>>>>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>>>>
>>>>> Hi Bowen,
>>>>>
>>>>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
>>>>> looks like the code is inconsistent on whether lastSyncedAt is a relative
>>>>> monotonic clock or absolute epoch time.
>>>>> Yes, I think so, I am going to create a defect regarding it.
>>>>>
>>>>> >> I've also attempted to search for all RegisteredSignal objects in
>>>>> the heap, and found 15 of them, but none of them is related to the
>>>>> read-hotness-tracker:1 thread.
>>>>> It is very strange because the stack trace for this thread waits
>>>>> within the object and read-hotness-tracker:1 thread has active
>>>>> "WaitQueue.Signal signal" local variable..
>>>>> What tool do you use for the heap dump inspection? I use Eclipse
>>>>> Memory Analyzer. It has "threads_overview" view and you can check the
>>>>> values for local variables within a thread stack.
>>>>> Like this:
>>>>> [image: image.png]
>>>>>
>>>>> Regards,
>>>>> Dmitry
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
>>>>> user@cassandra.apache.org> wrote:
>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> Thank you for helping me.
>>>>>>
>>>>>> You are right, that's a monotonic clock. I was definitely wrongly
>>>>>> interpreting its value. I got misled by this line of code
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>>>>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>>>>>> like the code is inconsistent on whether lastSyncedAt is a relative
>>>>>> monotonic clock or absolute epoch time.
>>>>>>
>>>>>> Please see the attached cassandra.yaml with sensitive information
>>>>>> masked out with asterisks.
>>>>>>
>>>>>> There's no ERROR level log in the system.log other than some
>>>>>> guardrail violations (items_per_collection, if it matters) and random and
>>>>>> rare network issues such as connection timeout in the few days run up to
>>>>>> and during the event.
>>>>>>
>>>>>> We have the stdout and stderr captured in the systemd journal, and
>>>>>> they have been quiet in during that period as well, not a single line got
>>>>>> logged.
>>>>>>
>>>>>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>>>>>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>>>>>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>>>>>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>>>>>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>>>>>> respectively.
>>>>>>
>>>>>> The CDC index files have been deleted by the CDC consumer, and we
>>>>>> don't keep a copy of these files elsewhere, so I cannot check the offsets
>>>>>> in them unfortunately.
>>>>>>
>>>>>> Regarding the WaitQueue, I found the queue to be empty in the heap
>>>>>> dump, which means there's no RegisteredSignal objects in the queue at all.
>>>>>> The queue is a ConcurrentLinkedQueue object, which has a head and a tail.
>>>>>> The head has item=null and next=null, and the tail has item=null and
>>>>>> next=head. I've also attempted to search for all RegisteredSignal objects
>>>>>> in the heap, and found 15 of them, but none of them is related to the
>>>>>> read-hotness-tracker:1 thread. They are for the following threads:
>>>>>>
>>>>>> - MemtablePostFlush:1
>>>>>> - AntiEntropyStage:1
>>>>>> - MemtableFlushWriter:910
>>>>>> - COMMIT-LOG-ALLOCATOR
>>>>>> - NativePoolCleaner
>>>>>> - MemtableFlushWriter:911
>>>>>> - Native-Transport-Requests-135
>>>>>> - Native-Transport-Requests-139
>>>>>> - Native-Transport-Requests-132
>>>>>> - Native-Transport-Requests-143
>>>>>> - Native-Transport-Requests-129
>>>>>> - Native-Transport-Requests-140
>>>>>> - Native-Transport-Requests-153
>>>>>> - Native-Transport-Requests-141
>>>>>> - Native-Transport-Requests-137
>>>>>>
>>>>>> This looks rather suspicious to me, as the read-hotness-tracker:1
>>>>>> thread is blocked and waiting for a non-existence registered signal.
>>>>>>
>>>>>> Cheers,
>>>>>> Bowen
>>>>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>>>>
>>>>>> Hi Bowen,
>>>>>>
>>>>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>>>>> pollStarted;
>>>>>> where long pollStarted = clock.now();
>>>>>> the logic uses clock from: SyncRunnable sync = new
>>>>>> SyncRunnable(preciseTime)
>>>>>> preciseTime by default is SystemClock
>>>>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>>>>> System.nanoTime()
>>>>>> so, the field is populated from System.nanoTime() which returns a
>>>>>> relative timestamp, which cannot be treated as an absolute time like
>>>>>> linux epoch (it is calculated like this:
>>>>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>>>>> )
>>>>>>
>>>>>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>>>>>> other security sensitive things, of course)
>>>>>> Do you have any exceptions in your system.log related to commit log?
>>>>>> Do you have JVM stdout/stderr captured to a file? Does it have any
>>>>>> exceptions? (which can be present there if Cassandra does not catch them
>>>>>> properly by itself)
>>>>>>
>>>>>> ---
>>>>>> How many CommitLogSegment objects do you have in your heap dump?
>>>>>> What are values for the following fields of CommitLogSegment objects?
>>>>>> lastSyncedOffset
>>>>>> lastMarkerOffset
>>>>>> cdcState
>>>>>>
>>>>>> Do you have CDC index files written by
>>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>>>>> If you have them - what are the last offsets written there?
>>>>>>
>>>>>> Regards,
>>>>>> Dmitry
>>>>>>
>>>>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>>>>>> user@cassandra.apache.org> wrote:
>>>>>>
>>>>>>> The syncComplete is an instance of WaitQueue, and the
>>>>>>> syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly,
>>>>>>> the queue is empty. There is no item in the queue's linked list, only the
>>>>>>> head and tail nodes, each has item=null.
>>>>>>>
>>>>>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>>>>>> correct to me:
>>>>>>>
>>>>>>> do
>>>>>>> {
>>>>>>> WaitQueue.Signal signal = context != null ?
>>>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>>>> if (lastSyncedAt < syncTime)
>>>>>>> signal.awaitUninterruptibly();
>>>>>>> else
>>>>>>> signal.cancel();
>>>>>>> }
>>>>>>> while (lastSyncedAt < syncTime);
>>>>>>>
>>>>>>> This matches the example given in the WaitQueue's documentation
>>>>>>> (which is definitely missing a pair of curly brackets and has bad
>>>>>>> indentation):
>>>>>>>
>>>>>>> while (!conditionMet())
>>>>>>> Signal s = q.register();
>>>>>>> if (!conditionMet()) // or, perhaps more correctly,
>>>>>>> !conditionChanged()
>>>>>>> s.await();
>>>>>>> else
>>>>>>> s.cancel();
>>>>>>>
>>>>>>> So, why is the queue empty while the thread is blocked on waiting
>>>>>>> for the signal?
>>>>>>>
>>>>>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>>>>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>>>>>> about 5 years ago.
>>>>>>>
>>>>>>> Is this suppose to be a wall clock time in milliseconds since 1
>>>>>>> January 1970 00:00? Or is it a monotonic clock time that has no meaning out
>>>>>>> of the context, and I'm just wrongly interpreting its value as a wall clock
>>>>>>> time?
>>>>>>>
>>>>>>>
>>>>>>> I've found that the syncComplete.queue is empty, meaning the
>>>>>>> WaitQueue object believes that there's nothing waiting for the signal, yet
>>>>>>> the "read-hotness-tracker:1" thread is clearly waiting for it.
>>>>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>>>>
>>>>>>> I think I'm getting really close now. This seems to have something
>>>>>>> to do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>>>>>
>>>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>>>> at
>>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>>> at
>>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>>> at
>>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>>> at
>>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>>>> at
>>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>>>>>> <------ here
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>>>>> at
>>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>>> at
>>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>>> at
>>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>>>> at
>>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>>>> at
>>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>>>>> <------ and here
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>>>> at
>>>>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>>>> at
>>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>>>> at
>>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>>>> at
>>>>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>>>> at
>>>>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>>>> local variable: java.lang.String#42806
>>>>>>> local variable: java.lang.String#47659
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>>>>> at
>>>>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>>>> at
>>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>>>> local variable:
>>>>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>>>>> at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>>>> local variable:
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>>>> local variable:
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>>> local variable:
>>>>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>>>> local variable:
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>>> local variable:
>>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>>> local variable:
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>>> local variable:
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>>> at
>>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>>>> local variable:
>>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>>>> local variable:
>>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>>>
>>>>>>>
>>>>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>>>>
>>>>>>> public WriteContext beginWrite(Mutation mutation, boolean
>>>>>>> makeDurable) throws RequestExecutionException
>>>>>>> {
>>>>>>> OpOrder.Group group = null;
>>>>>>> try
>>>>>>> {
>>>>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>>>>
>>>>>>> // write the mutation to the commitlog and memtables
>>>>>>> CommitLogPosition position = null;
>>>>>>> if (makeDurable)
>>>>>>> {
>>>>>>> position = addToCommitLog(mutation); // <--- and here
>>>>>>> }
>>>>>>> return new CassandraWriteContext(group, position);
>>>>>>> }
>>>>>>> catch (Throwable t)
>>>>>>> {
>>>>>>> if (group != null)
>>>>>>> {
>>>>>>> group.close();
>>>>>>> }
>>>>>>> throw t;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> The Keyspace.writeOrder.start() method call blocks the
>>>>>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation)
>>>>>>> which indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>>>>>
>>>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>>>> {
>>>>>>> do
>>>>>>> {
>>>>>>> WaitQueue.Signal signal = context != null ?
>>>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>>>> if (lastSyncedAt < syncTime)
>>>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>>>> else
>>>>>>> signal.cancel();
>>>>>>> }
>>>>>>> while (lastSyncedAt < syncTime);
>>>>>>> }
>>>>>>>
>>>>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>>>>
>>>>>>> Now I know what is blocking the memtable flushing, but I haven't
>>>>>>> been able to figure out is why it got stuck on waiting for that signal.
>>>>>>>
>>>>>>> I would appreciate it if anyone can offer some insight here.
>>>>>>>
>>>>>>>
>>>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>>>
>>>>>>> I will give it a try and see what I can find. I plan to go down the
>>>>>>> rabbit hole tomorrow. Will keep you updated.
>>>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>>>>>
>>>>>>> Writes on this node starts to timeout and fail. But if left
>>>>>>> untouched, it's only gonna get worse, and eventually lead to JVM OOM and
>>>>>>> crash.
>>>>>>>
>>>>>>> By inspecting the heap dump created at OOM, we can see that both of
>>>>>>> the MemtableFlushWriter threads are stuck on line 1190
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>>> in the ColumnFamilyStore.java:
>>>>>>>
>>>>>>> // mark writes older than the barrier as blocking
>>>>>>> progress, permitting them to exceed our memory limit
>>>>>>> // if they are stuck waiting on it, then wait for them
>>>>>>> all to complete
>>>>>>> writeBarrier.markBlocking();
>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>>
>>>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>>> in the same file.
>>>>>>>
>>>>>>> try
>>>>>>> {
>>>>>>> // we wait on the latch for the commitLogUpperBound
>>>>>>> to be set, and so that waiters
>>>>>>> // on this task can rely on all prior flushes being
>>>>>>> complete
>>>>>>> latch.await(); // <----------- stuck here
>>>>>>> }
>>>>>>> Our top suspect is CDC interacting with repair, since this started
>>>>>>> to happen shortly after we enabled CDC on the nodes, and each time repair
>>>>>>> was running. But we have not been able to reproduce this in a testing
>>>>>>> cluster, and don't know what's the next step to troubleshoot this issue. So
>>>>>>> I'm posting it in the mailing lists and hoping someone may know something
>>>>>>> about it or point me to the right direction.
>>>>>>>
>>>>>>>
>>>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>>>>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>>>>>> read commands” that have a barrier on the memtable that prevent release.
>>>>>>>
>>>>>>> You’ve got the heap (great, way better than most people debugging),
>>>>>>> are you able to navigate through it and look for references to that
>>>>>>> memtable or other things holding a barrier?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dmitry Konstantinov
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Dmitry Konstantinov
>>>>>
>>>>>
>>>>
>>>> --
>>>> Dmitry Konstantinov
>>>>
>>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Thank you!
A ticket for the issue:
https://issues.apache.org/jira/browse/CASSANDRA-20084
<https://issues.apache.org/jira/browse/CASSANDRA-20084>
Regards,
Dmitry
On Thu, 14 Nov 2024 at 10:33, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Impressive! It all makes sense now.
>
> I believe this bug was accidentally introduced back in 2021 as a part of
> CASSANDRA-16924 <https://issues.apache.org/jira/browse/CASSANDRA-16924>,
> commit e5b92e1
> <https://github.com/apache/cassandra/commit/e5b92e108851d2be81c43cbf982564f92d28aa6b#diff-a8247a86ff78ae4f477adbc7f9e93c6375e7d689a1265f26470fb9c42af74995>,
> which removed the awaitUninterruptibly() method from the
> WaitQueue$AbstractSignal in favour of the newly introduced
> org.apache.cassandra.utils.concurrent.Awaitable class and split the code
> below into two places.
>
> public void awaitUninterruptibly()
> {
> boolean interrupted = false;
> while (!isSignalled())
> {
> if (Thread.interrupted())
> interrupted = true;
> LockSupport.park();
> }
> if (interrupted)
> Thread.currentThread().interrupt();
> checkAndClear();
> }
>
> It's not impossible that someone runs into the same issue twice in a row
> in a short window of time, but I wouldn't expect this to occur frequently
> enough to catch most people's attentions. Because it rarely occurs and can
> be "fixed" by simply restarting Cassandra, it has stayed in the codebase
> for years without being discovered. I'm glad that we took the opportunity
> to deep dive into this and found the root cause.
>
> Good job, Dmitry, and thank you for helping me on this.
>
>
> On 13/11/2024 23:48, Dmitry Konstantinov wrote:
>
> Thank you. I think I have got the cause, it is a non-trivial puzzle which
> took a few hours :-).
>
> So, we have the following stacktrace for "read-hotness-tracker:1" thread
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable: org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>
> where we are waiting in the await() method of
> WaitQueue$Standard$SignalWithListener#2086139 object.
>
> another input: WaitQueue$Standard$SignalWithListener#2086139 has the
> following field values based on the screenshots:
> state = *-1* (it means CANCELLED:
> org.apache.cassandra.utils.concurrent.WaitQueue.Standard#CANCELLED = -1)
> thread = null
>
> According to the code in WaitQueue.java such combination of values only
> possible if we invoked cancel() on this SignalWithListener object but we
> didn't do it in our code, it is in another branch of
> AbstractCommitLogService.awaitSyncAt
>
> but let's check the await() logic more carefully:
> public Signal await() throws InterruptedException
> {
> while (!isSignalled())
> {
> checkInterrupted();
> LockSupport.park(); <--- we are here
> }
> checkAndClear();
> return this;
> }
> and inspect unassuming checkInterrupted() method
> private void checkInterrupted() throws InterruptedException
> {
> if (Thread.interrupted())
> {
> *cancel();*
> throw new InterruptedException();
> }
> }
> here we see that if the currently awaiting thread is interrupted then
> SignalWithListener cancels itself.
> but we throw new InterruptedException() here, so the execution should fail
> with it.
> Checking the thread stack trace and see that AbstractSignal.await() is
> invoked from
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>
> how it looks inside:
> public static <A extends Awaitable> A awaitUninterruptibly(A await)
> {
> boolean interrupted = false;
> while (true)
> {
> try
> {
> await.await();
> break;
> }
> catch (InterruptedException e)
> {
> interrupted = true;
> }
> }
> if (interrupted)
> Thread.currentThread().interrupt();
> return await;
> }
> Here we have a loop, if the current thread is interrupted - we have
> InterruptedException thrown by await.await(), we catch it here and try
> await.await() again. But in the previous invocation of await.await() we
> have cancelled our Signal object, so the condition inside await() method:
> isSignalled(): state == SIGNALLED is always false now plus if another
> thread will invoke notifyAll - nothing will happen, because as a part of
> cancellation our Signal object unregister itself from the queue and the
> notifying thread will not able to find it to notify.
> I have reproduced the case with a simple unit test.
>
> The last element of the puzzle - why is the thread interrupted? To
> understand it - let's check there "read-hotness-tracker:1" thread execution
> started. It is a background periodical runnable in
> org.apache.cassandra.io.sstable.format.SSTableReader.GlobalTidy
> readMeterSyncFuture = new
> WeakReference<>(syncExecutor.scheduleAtFixedRate(new Runnable()
> {
> public void run()
> {
> if (obsoletion == null)
> {
> meterSyncThrottle.acquire();
> SystemKeyspace.persistSSTableReadMeter(desc.ksname,
> desc.cfname, desc.id, readMeter);
> }
> }
> }, 1, 5, TimeUnit.MINUTES));
>
> let's check how we deal with readMeterSyncFuture ref and we see our
> interruption:
> private void stopReadMeterPersistence()
> {
> ScheduledFuture<?> readMeterSyncFutureLocal =
> readMeterSyncFuture.get();
> if (readMeterSyncFutureLocal != null)
> {
> * readMeterSyncFutureLocal.cancel(true); *<========= here we can
> interrupt the thread
> readMeterSyncFuture = NULL;
> }
> }
>
> stopReadMeterPersistence() is invoked by one of the following branches of
> logic when an SSTable is not in use anymore:
> org.apache.cassandra.io.sstable.format.SSTableReader#markObsolete
> org.apache.cassandra.db.lifecycle.Helpers#markObsolete
> org.apache.cassandra.db.lifecycle.Tracker#dropSSTables(...)
> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doCommit
> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doAbort
>
> So,the issue is caused by a combination of events with a right timing:
> - an SSTable which was become obsolete at the middle of
> persistSSTableReadMeter background job execution for this SSTable, so we
> interrupted the job
> - persistSSTableReadMeter was at that moment in the middle of write
> operation, at commit log append step
> - and the commit log append step was stuck due to a delay in commit log
> sync (lastSyncedAt < syncTime condition in AbstractCommitLogService), so we
> started to wait for it using await
>
>
> The last observation - timing.
> We have
> org.apache.cassandra.db.commitlog.AbstractCommitLogService
> lastSyncedAt = 1573038452910796
>
> com.codahale.metrics.Timer$Context#2086795 (created when the write
> operation started to wait for commit log sync)
> startTime = 1410129608743025
>
> both are from System.nanoTime()
>
> so, 1573038452910796 - 1410129608743025 = 162 908 844 167 771 ns = 162 908
> sec ~ 45 hours; the cause happened about that time ago.
>
> I will create a ticket for this issue to fix it.
>
> Regards,
> Dmitry
>
>
>
> On Wed, 13 Nov 2024 at 22:11, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Please see the screenshots below.
>>
>> Unfortunately that the WaitQueue$Standard$SignalWithListener#2086139
>> object is the GC root, and there's no references to it.
>>
>> I hope this helps.
>>
>>
>>
>>
>> On 13/11/2024 21:33, Dmitry Konstantinov wrote:
>>
>> Hi Bowen, thank you for the details.
>> Could you please also share a screenshot (Visual VM should be ok) for "read-hotness-tracker:1"
>> thread with expanded local variable elements for the following lines:
>> 1) at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
>> particularly com.codahale.metrics.Timer$Context#2086795 internals such
>> as startTime
>> 2) at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
>> particularly WaitQueue$Standard$SignalWithListener#2086139 internals
>> such as thread, state
>>
>> Also, could you please render all GC paths to root for WaitQueue$Standard$SignalWithListener#2086139
>> object and share the correspondent tree view?
>>
>> Regards,
>> Dmitry
>>
>>
>> On Tue, 12 Nov 2024 at 11:40, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Just tried Eclipse Memory Analyzer, it shows the exact same local
>>> variables as VisualVM, so there is no difference here.
>>>
>>> However, Eclipse Memory Analyzer is significantly slower (lagging) to
>>> operate via X window forwarding or VNC (possibly due to the GUI decorations
>>> and animations) and some features rely on a built-in web browser which I
>>> can't get it to work on a headless server. I'm doing the analysis on a
>>> headless server because my PC doesn't have enough RAM to load the large
>>> heap dump file.
>>>
>>> See screenshot below for a comparison between Eclipse Memory Analyzer
>>> and VisualVM on the variables referenced by the PERIODIC-COMMIT-LOG-SYNCER
>>> thread.
>>>
>>> Eclipse Memory Analyzer
>>>
>>> VisualVM
>>>
>>>
>>> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>>>
>>> >> If Eclipse Memory Analyzer also can't find the mysterious item, how
>>> do I proceed from there? What else can I do?
>>> We can analyze other variables from the heap dump, read code and repeat,
>>> until we find a clue :-). As of now I do not see a much better option. If
>>> the issue reproduces regularly - a custom build with extra logs can be done
>>> to add more diagnostic info printing.
>>> It is actually lucky that you have a heap dump, it can give a lot of
>>> info, unfortunately only in a moment of time (so we cannot see from the
>>> issue in dynamic).
>>>
>>> >> This has happened twice in the last 10 days, to two different nodes
>>> in the same cluster.
>>> Do you have heap dumps for the other failures?
>>> How much time did you have between a node startup and the failure?
>>>
>>> Can you share field values
>>> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
>>> using Eclipse Memory Analyzer
>>> from the following thread?:
>>> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#5
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
>>> local variable: java.util.concurrent.Semaphore$NonfairSync#1
>>> local variable:
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
>>> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
>>> at
>>> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>>>
>>> It should looks like this:
>>>
>>> [image: image.png]
>>>
>>> [image: image.png]
>>>
>>> Regards,
>>> Dmitry
>>>
>>>
>>> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> It indeed is very strange, either the tool I'm using is wrong, or
>>>> there's something causing the item to evaporate from the queue mysteriously
>>>> or never got into the queue.
>>>>
>>>> I've been using VisualVM to inspect the heap dump file and using OQL to
>>>> find and explore the objects in it.
>>>>
>>>> VisualVM does show some local variables, but not all of them. This
>>>> isn't an issue for types with little number of instances, because I can
>>>> search objects by type to find them. But it can be troublesome when an
>>>> object is a very common type.
>>>>
>>>> I will try Eclipse Memory Analyzer next week to see if it can find the
>>>> missing item, but I have my doubts.
>>>>
>>>> I've read the code of WaitQueue and the
>>>> AbstractCommitLogService.awaitSyncAt() method, and they both look correct
>>>> to me. If Eclipse Memory Analyzer also can't find the mysterious item, how
>>>> do I proceed from there? What else can I do?
>>>>
>>>> Cheers,
>>>> Bowen
>>>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>>>
>>>> Hi Bowen,
>>>>
>>>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
>>>> looks like the code is inconsistent on whether lastSyncedAt is a relative
>>>> monotonic clock or absolute epoch time.
>>>> Yes, I think so, I am going to create a defect regarding it.
>>>>
>>>> >> I've also attempted to search for all RegisteredSignal objects in
>>>> the heap, and found 15 of them, but none of them is related to the
>>>> read-hotness-tracker:1 thread.
>>>> It is very strange because the stack trace for this thread waits within
>>>> the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
>>>> signal" local variable..
>>>> What tool do you use for the heap dump inspection? I use Eclipse Memory
>>>> Analyzer. It has "threads_overview" view and you can check the values for
>>>> local variables within a thread stack.
>>>> Like this:
>>>> [image: image.png]
>>>>
>>>> Regards,
>>>> Dmitry
>>>>
>>>>
>>>>
>>>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
>>>> user@cassandra.apache.org> wrote:
>>>>
>>>>> Hi Dmitry,
>>>>>
>>>>> Thank you for helping me.
>>>>>
>>>>> You are right, that's a monotonic clock. I was definitely wrongly
>>>>> interpreting its value. I got misled by this line of code
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>>>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>>>>> like the code is inconsistent on whether lastSyncedAt is a relative
>>>>> monotonic clock or absolute epoch time.
>>>>>
>>>>> Please see the attached cassandra.yaml with sensitive information
>>>>> masked out with asterisks.
>>>>>
>>>>> There's no ERROR level log in the system.log other than some guardrail
>>>>> violations (items_per_collection, if it matters) and random and rare
>>>>> network issues such as connection timeout in the few days run up to and
>>>>> during the event.
>>>>>
>>>>> We have the stdout and stderr captured in the systemd journal, and
>>>>> they have been quiet in during that period as well, not a single line got
>>>>> logged.
>>>>>
>>>>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>>>>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>>>>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>>>>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>>>>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>>>>> respectively.
>>>>>
>>>>> The CDC index files have been deleted by the CDC consumer, and we
>>>>> don't keep a copy of these files elsewhere, so I cannot check the offsets
>>>>> in them unfortunately.
>>>>>
>>>>> Regarding the WaitQueue, I found the queue to be empty in the heap
>>>>> dump, which means there's no RegisteredSignal objects in the queue at all.
>>>>> The queue is a ConcurrentLinkedQueue object, which has a head and a tail.
>>>>> The head has item=null and next=null, and the tail has item=null and
>>>>> next=head. I've also attempted to search for all RegisteredSignal objects
>>>>> in the heap, and found 15 of them, but none of them is related to the
>>>>> read-hotness-tracker:1 thread. They are for the following threads:
>>>>>
>>>>> - MemtablePostFlush:1
>>>>> - AntiEntropyStage:1
>>>>> - MemtableFlushWriter:910
>>>>> - COMMIT-LOG-ALLOCATOR
>>>>> - NativePoolCleaner
>>>>> - MemtableFlushWriter:911
>>>>> - Native-Transport-Requests-135
>>>>> - Native-Transport-Requests-139
>>>>> - Native-Transport-Requests-132
>>>>> - Native-Transport-Requests-143
>>>>> - Native-Transport-Requests-129
>>>>> - Native-Transport-Requests-140
>>>>> - Native-Transport-Requests-153
>>>>> - Native-Transport-Requests-141
>>>>> - Native-Transport-Requests-137
>>>>>
>>>>> This looks rather suspicious to me, as the read-hotness-tracker:1
>>>>> thread is blocked and waiting for a non-existence registered signal.
>>>>>
>>>>> Cheers,
>>>>> Bowen
>>>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>>>
>>>>> Hi Bowen,
>>>>>
>>>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>>>> pollStarted;
>>>>> where long pollStarted = clock.now();
>>>>> the logic uses clock from: SyncRunnable sync = new
>>>>> SyncRunnable(preciseTime)
>>>>> preciseTime by default is SystemClock
>>>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>>>> System.nanoTime()
>>>>> so, the field is populated from System.nanoTime() which returns a
>>>>> relative timestamp, which cannot be treated as an absolute time like
>>>>> linux epoch (it is calculated like this:
>>>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>>>> )
>>>>>
>>>>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>>>>> other security sensitive things, of course)
>>>>> Do you have any exceptions in your system.log related to commit log?
>>>>> Do you have JVM stdout/stderr captured to a file? Does it have any
>>>>> exceptions? (which can be present there if Cassandra does not catch them
>>>>> properly by itself)
>>>>>
>>>>> ---
>>>>> How many CommitLogSegment objects do you have in your heap dump?
>>>>> What are values for the following fields of CommitLogSegment objects?
>>>>> lastSyncedOffset
>>>>> lastMarkerOffset
>>>>> cdcState
>>>>>
>>>>> Do you have CDC index files written by
>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>>>> If you have them - what are the last offsets written there?
>>>>>
>>>>> Regards,
>>>>> Dmitry
>>>>>
>>>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>>>>> user@cassandra.apache.org> wrote:
>>>>>
>>>>>> The syncComplete is an instance of WaitQueue, and the
>>>>>> syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly,
>>>>>> the queue is empty. There is no item in the queue's linked list, only the
>>>>>> head and tail nodes, each has item=null.
>>>>>>
>>>>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>>>>> correct to me:
>>>>>>
>>>>>> do
>>>>>> {
>>>>>> WaitQueue.Signal signal = context != null ?
>>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>>> if (lastSyncedAt < syncTime)
>>>>>> signal.awaitUninterruptibly();
>>>>>> else
>>>>>> signal.cancel();
>>>>>> }
>>>>>> while (lastSyncedAt < syncTime);
>>>>>>
>>>>>> This matches the example given in the WaitQueue's documentation
>>>>>> (which is definitely missing a pair of curly brackets and has bad
>>>>>> indentation):
>>>>>>
>>>>>> while (!conditionMet())
>>>>>> Signal s = q.register();
>>>>>> if (!conditionMet()) // or, perhaps more correctly,
>>>>>> !conditionChanged()
>>>>>> s.await();
>>>>>> else
>>>>>> s.cancel();
>>>>>>
>>>>>> So, why is the queue empty while the thread is blocked on waiting for
>>>>>> the signal?
>>>>>>
>>>>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>>>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>>>>> about 5 years ago.
>>>>>>
>>>>>> Is this suppose to be a wall clock time in milliseconds since 1
>>>>>> January 1970 00:00? Or is it a monotonic clock time that has no meaning out
>>>>>> of the context, and I'm just wrongly interpreting its value as a wall clock
>>>>>> time?
>>>>>>
>>>>>>
>>>>>> I've found that the syncComplete.queue is empty, meaning the
>>>>>> WaitQueue object believes that there's nothing waiting for the signal, yet
>>>>>> the "read-hotness-tracker:1" thread is clearly waiting for it.
>>>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>>>
>>>>>> I think I'm getting really close now. This seems to have something to
>>>>>> do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>>>>
>>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>>>>> <------ here
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>>>> local variable:
>>>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>>> at
>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>>> at
>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>>>> <------ and here
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>>> at
>>>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>>> at
>>>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>>> local variable: java.lang.String#42806
>>>>>> local variable: java.lang.String#47659
>>>>>> local variable:
>>>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>>> local variable:
>>>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>>>> at
>>>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>>> local variable:
>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>>> local variable:
>>>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>>>> at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>> local variable:
>>>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>>> local variable:
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>> local variable:
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>> at
>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>>
>>>>>>
>>>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>>>
>>>>>> public WriteContext beginWrite(Mutation mutation, boolean
>>>>>> makeDurable) throws RequestExecutionException
>>>>>> {
>>>>>> OpOrder.Group group = null;
>>>>>> try
>>>>>> {
>>>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>>>
>>>>>> // write the mutation to the commitlog and memtables
>>>>>> CommitLogPosition position = null;
>>>>>> if (makeDurable)
>>>>>> {
>>>>>> position = addToCommitLog(mutation); // <--- and here
>>>>>> }
>>>>>> return new CassandraWriteContext(group, position);
>>>>>> }
>>>>>> catch (Throwable t)
>>>>>> {
>>>>>> if (group != null)
>>>>>> {
>>>>>> group.close();
>>>>>> }
>>>>>> throw t;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> The Keyspace.writeOrder.start() method call blocks the
>>>>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation)
>>>>>> which indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>>>>
>>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>>> {
>>>>>> do
>>>>>> {
>>>>>> WaitQueue.Signal signal = context != null ?
>>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>>> if (lastSyncedAt < syncTime)
>>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>>> else
>>>>>> signal.cancel();
>>>>>> }
>>>>>> while (lastSyncedAt < syncTime);
>>>>>> }
>>>>>>
>>>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>>>
>>>>>> Now I know what is blocking the memtable flushing, but I haven't been
>>>>>> able to figure out is why it got stuck on waiting for that signal.
>>>>>>
>>>>>> I would appreciate it if anyone can offer some insight here.
>>>>>>
>>>>>>
>>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>>
>>>>>> I will give it a try and see what I can find. I plan to go down the
>>>>>> rabbit hole tomorrow. Will keep you updated.
>>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>>>>
>>>>>> Writes on this node starts to timeout and fail. But if left
>>>>>> untouched, it's only gonna get worse, and eventually lead to JVM OOM and
>>>>>> crash.
>>>>>>
>>>>>> By inspecting the heap dump created at OOM, we can see that both of
>>>>>> the MemtableFlushWriter threads are stuck on line 1190
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>> in the ColumnFamilyStore.java:
>>>>>>
>>>>>> // mark writes older than the barrier as blocking
>>>>>> progress, permitting them to exceed our memory limit
>>>>>> // if they are stuck waiting on it, then wait for them
>>>>>> all to complete
>>>>>> writeBarrier.markBlocking();
>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>
>>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>> in the same file.
>>>>>>
>>>>>> try
>>>>>> {
>>>>>> // we wait on the latch for the commitLogUpperBound
>>>>>> to be set, and so that waiters
>>>>>> // on this task can rely on all prior flushes being
>>>>>> complete
>>>>>> latch.await(); // <----------- stuck here
>>>>>> }
>>>>>> Our top suspect is CDC interacting with repair, since this started to
>>>>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>>>>> running. But we have not been able to reproduce this in a testing cluster,
>>>>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>>>>> posting it in the mailing lists and hoping someone may know something about
>>>>>> it or point me to the right direction.
>>>>>>
>>>>>>
>>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>>>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>>>>> read commands” that have a barrier on the memtable that prevent release.
>>>>>>
>>>>>> You’ve got the heap (great, way better than most people debugging),
>>>>>> are you able to navigate through it and look for references to that
>>>>>> memtable or other things holding a barrier?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Dmitry Konstantinov
>>>>>
>>>>>
>>>>
>>>> --
>>>> Dmitry Konstantinov
>>>>
>>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Impressive! It all makes sense now.
I believe this bug was accidentally introduced back in 2021 as a part of
CASSANDRA-16924 <https://issues.apache.org/jira/browse/CASSANDRA-16924>,
commit e5b92e1
<https://github.com/apache/cassandra/commit/e5b92e108851d2be81c43cbf982564f92d28aa6b#diff-a8247a86ff78ae4f477adbc7f9e93c6375e7d689a1265f26470fb9c42af74995>,
which removed the awaitUninterruptibly() method from the
WaitQueue$AbstractSignal in favour of the newly introduced
org.apache.cassandra.utils.concurrent.Awaitable class and split the code
below into two places.
public void awaitUninterruptibly()
{
boolean interrupted = false;
while (!isSignalled())
{
if (Thread.interrupted())
interrupted = true;
LockSupport.park();
}
if (interrupted)
Thread.currentThread().interrupt();
checkAndClear();
}
It's not impossible that someone runs into the same issue twice in a row
in a short window of time, but I wouldn't expect this to occur
frequently enough to catch most people's attentions. Because it rarely
occurs and can be "fixed" by simply restarting Cassandra, it has stayed
in the codebase for years without being discovered. I'm glad that we
took the opportunity to deep dive into this and found the root cause.
Good job, Dmitry, and thank you for helping me on this.
On 13/11/2024 23:48, Dmitry Konstantinov wrote:
> Thank you. I think I have got the cause, it is a non-trivial puzzle
> which took a few hours :-).
>
> So, we have the following stacktrace for "read-hotness-tracker:1" thread
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable:
> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable: org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>
> where we are waiting in the await() method of
> WaitQueue$Standard$SignalWithListener#2086139 object.
>
> another input: WaitQueue$Standard$SignalWithListener#2086139 has the
> following field values based on the screenshots:
> state = *-1* (it means CANCELLED:
> org.apache.cassandra.utils.concurrent.WaitQueue.Standard#CANCELLED = -1)
> thread = null
>
> According to the code in WaitQueue.java such combination of values
> only possible if we invoked cancel() on this SignalWithListener object
> but we didn't do it in our code, it is in another branch of
> AbstractCommitLogService.awaitSyncAt
>
> but let's check the await() logic more carefully:
> public Signal await() throws InterruptedException
> {
> while (!isSignalled())
> {
> checkInterrupted();
> LockSupport.park(); <--- we are here
> }
> checkAndClear();
> return this;
> }
> and inspect unassuming checkInterrupted() method
> private void checkInterrupted() throws InterruptedException
> {
> if (Thread.interrupted())
> {
> *cancel();*
> throw new InterruptedException();
> }
> }
> here we see that if the currently awaiting thread is interrupted then
> SignalWithListener cancels itself.
> but we throw new InterruptedException() here, so the execution should
> fail with it.
> Checking the thread stack trace and see that AbstractSignal.await() is
> invoked from
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>
> how it looks inside:
> public static <A extends Awaitable> A awaitUninterruptibly(A await)
> {
> boolean interrupted = false;
> while (true)
> {
> try
> {
> await.await();
> break;
> }
> catch (InterruptedException e)
> {
> interrupted = true;
> }
> }
> if (interrupted)
> Thread.currentThread().interrupt();
> return await;
> }
> Here we have a loop, if the current thread is interrupted - we have
> InterruptedException thrown by await.await(), we catch it here and try
> await.await() again. But in the previous invocation of await.await()
> we have cancelled our Signal object, so the condition inside await()
> method:
> isSignalled(): state == SIGNALLED is always false now plus if another
> thread will invoke notifyAll - nothing will happen, because as a part
> of cancellation our Signal object unregister itself from the queue and
> the notifying thread will not able to find it to notify.
> I have reproduced the case with a simple unit test.
>
> The last element of the puzzle - why is the thread interrupted? To
> understand it - let's check there "read-hotness-tracker:1" thread
> execution started. It is a background periodical runnable in
> org.apache.cassandra.io.sstable.format.SSTableReader.GlobalTidy
> readMeterSyncFuture = new
> WeakReference<>(syncExecutor.scheduleAtFixedRate(new Runnable()
> {
> public void run()
> {
> if (obsoletion == null)
> {
> meterSyncThrottle.acquire();
> SystemKeyspace.persistSSTableReadMeter(desc.ksname, desc.cfname,
> desc.id <http://desc.id>, readMeter);
> }
> }
> }, 1, 5, TimeUnit.MINUTES));
>
> let's check how we deal with readMeterSyncFuture ref and we see our
> interruption:
> private void stopReadMeterPersistence()
> {
> ScheduledFuture<?> readMeterSyncFutureLocal =
> readMeterSyncFuture.get();
> if (readMeterSyncFutureLocal != null)
> {
> *readMeterSyncFutureLocal.cancel(true); *<========= here we can
> interrupt the thread
> readMeterSyncFuture = NULL;
> }
> }
>
> stopReadMeterPersistence() is invoked by one of the following branches
> of logic when an SSTable is not in use anymore:
> org.apache.cassandra.io.sstable.format.SSTableReader#markObsolete
> org.apache.cassandra.db.lifecycle.Helpers#markObsolete
> org.apache.cassandra.db.lifecycle.Tracker#dropSSTables(...)
> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doCommit
> org.apache.cassandra.db.lifecycle.LifecycleTransaction#doAbort
>
> So,the issue is caused by a combination of events with a right timing:
> - an SSTable which was become obsolete at the middle of
> persistSSTableReadMeter background job execution for this SSTable, so
> we interrupted the job
> - persistSSTableReadMeter was at that moment in the middle of write
> operation, at commit log append step
> - and the commit log append step was stuck due to a delay in commit
> log sync (lastSyncedAt < syncTime condition in
> AbstractCommitLogService), so we started to wait for it using await
>
>
> The last observation - timing.
> We have
> org.apache.cassandra.db.commitlog.AbstractCommitLogService
> lastSyncedAt = 1573038452910796
>
> com.codahale.metrics.Timer$Context#2086795 (created when the write
> operation started to wait for commit log sync)
> startTime = 1410129608743025
>
> both are from System.nanoTime()
>
> so, 1573038452910796 - 1410129608743025 = 162 908 844 167 771 ns = 162
> 908 sec ~ 45 hours; the cause happened about that time ago.
>
> I will create a ticket for this issue to fix it.
>
> Regards,
> Dmitry
>
>
>
> On Wed, 13 Nov 2024 at 22:11, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Please see the screenshots below.
>
> Unfortunately that the
> WaitQueue$Standard$SignalWithListener#2086139 object is the GC
> root, and there's no references to it.
>
> I hope this helps.
>
>
>
>
> On 13/11/2024 21:33, Dmitry Konstantinov wrote:
>> Hi Bowen, thank you for the details.
>> Could you please also share a screenshot (Visual VM should be ok)
>> for "read-hotness-tracker:1" thread with expanded local variable
>> elements for the following lines:
>> 1) at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
>> particularly com.codahale.metrics.Timer$Context#2086795 internals
>> such as startTime
>> 2) at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
>> particularly WaitQueue$Standard$SignalWithListener#2086139
>> internals such as thread, state
>>
>> Also, could you please render all GC paths to root for
>> WaitQueue$Standard$SignalWithListener#2086139 object and share
>> the correspondent tree view?
>>
>> Regards,
>> Dmitry
>>
>>
>> On Tue, 12 Nov 2024 at 11:40, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Just tried Eclipse Memory Analyzer, it shows the exact same
>> local variables as VisualVM, so there is no difference here.
>>
>> However, Eclipse Memory Analyzer is significantly slower
>> (lagging) to operate via X window forwarding or VNC (possibly
>> due to the GUI decorations and animations) and some features
>> rely on a built-in web browser which I can't get it to work
>> on a headless server. I'm doing the analysis on a headless
>> server because my PC doesn't have enough RAM to load the
>> large heap dump file.
>>
>> See screenshot below for a comparison between Eclipse Memory
>> Analyzer and VisualVM on the variables referenced by the
>> PERIODIC-COMMIT-LOG-SYNCER thread.
>>
>> Eclipse Memory Analyzer
>>
>> VisualVM
>>
>>
>> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>>> >> If Eclipse Memory Analyzer also can't find the mysterious
>>> item, how do I proceed from there? What else can I do?
>>> We can analyze other variables from the heap dump, read code
>>> and repeat, until we find a clue :-). As of now I do not see
>>> a much better option. If the issue reproduces regularly - a
>>> custom build with extra logs can be done to add more
>>> diagnostic info printing.
>>> It is actually lucky that you have a heap dump, it can give
>>> a lot of info, unfortunately only in a moment of time (so we
>>> cannot see from the issue in dynamic).
>>>
>>> >> This has happened twice in the last 10 days, to two
>>> different nodes in the same cluster.
>>> Do you have heap dumps for the other failures?
>>> How much time did you have between a node startup and the
>>> failure?
>>>
>>> Can you share field values
>>> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService
>>> and
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
>>> using Eclipse Memory Analyzer
>>> from the following thread?:
>>> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalThread#5
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
>>> local variable:
>>> java.util.concurrent.Semaphore$NonfairSync#1
>>> local variable:
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
>>> at
>>> java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
>>> at
>>> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>>>
>>> It should looks like this:
>>>
>>> image.png
>>>
>>> image.png
>>>
>>> Regards,
>>> Dmitry
>>>
>>>
>>> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> Hi Dmitry,
>>>
>>> It indeed is very strange, either the tool I'm using is
>>> wrong, or there's something causing the item to
>>> evaporate from the queue mysteriously or never got into
>>> the queue.
>>>
>>> I've been using VisualVM to inspect the heap dump file
>>> and using OQL to find and explore the objects in it.
>>>
>>> VisualVM does show some local variables, but not all of
>>> them. This isn't an issue for types with little number
>>> of instances, because I can search objects by type to
>>> find them. But it can be troublesome when an object is a
>>> very common type.
>>>
>>> I will try Eclipse Memory Analyzer next week to see if
>>> it can find the missing item, but I have my doubts.
>>>
>>> I've read the code of WaitQueue and the
>>> AbstractCommitLogService.awaitSyncAt() method, and they
>>> both look correct to me. If Eclipse Memory Analyzer also
>>> can't find the mysterious item, how do I proceed from
>>> there? What else can I do?
>>>
>>> Cheers,
>>> Bowen
>>>
>>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>>> Hi Bowen,
>>>>
>>>> >> protected volatile long lastSyncedAt =
>>>> currentTimeMillis();". It looks like the code is
>>>> inconsistent on whether lastSyncedAt is a relative
>>>> monotonic clock or absolute epoch time.
>>>> Yes, I think so, I am going to create a defect
>>>> regarding it.
>>>>
>>>> >> I've also attempted to search for all
>>>> RegisteredSignal objects in the heap, and found 15 of
>>>> them, but none of them is related to the
>>>> read-hotness-tracker:1 thread.
>>>> It is very strange because the stack trace for this
>>>> thread waits within the object and
>>>> read-hotness-tracker:1 thread has active
>>>> "WaitQueue.Signal signal" local variable..
>>>> What tool do you use for the heap dump inspection? I
>>>> use Eclipse Memory Analyzer. It has "threads_overview"
>>>> view and you can check the values for local variables
>>>> within a thread stack.
>>>> Like this:
>>>> image.png
>>>>
>>>> Regards,
>>>> Dmitry
>>>>
>>>>
>>>>
>>>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
>>>> <us...@cassandra.apache.org> wrote:
>>>>
>>>> Hi Dmitry,
>>>>
>>>> Thank you for helping me.
>>>>
>>>> You are right, that's a monotonic clock. I was
>>>> definitely wrongly interpreting its value. I got
>>>> misled by this line of code
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>>> "protected volatile long lastSyncedAt =
>>>> currentTimeMillis();". It looks like the code is
>>>> inconsistent on whether lastSyncedAt is a relative
>>>> monotonic clock or absolute epoch time.
>>>>
>>>> Please see the attached cassandra.yaml with
>>>> sensitive information masked out with asterisks.
>>>>
>>>> There's no ERROR level log in the system.log other
>>>> than some guardrail violations
>>>> (items_per_collection, if it matters) and random
>>>> and rare network issues such as connection timeout
>>>> in the few days run up to and during the event.
>>>>
>>>> We have the stdout and stderr captured in the
>>>> systemd journal, and they have been quiet in during
>>>> that period as well, not a single line got logged.
>>>>
>>>> There's 196 CommitLogSegment (more specifically,
>>>> MemoryMappedSegment) objects in the heap dump. Of
>>>> these, 194 has cdcState = CONTAINS lastSyncedOffset
>>>> = 33554432 lastMarkerOffset = 33554432. The other
>>>> two has cdcState = CONTAINS lastSyncedOffset =
>>>> 27007818 lastMarkerOffset = 27007818 and cdcState =
>>>> PERMITTED lastSyncedOffset = 20 lastMarkerOffset =
>>>> 20 respectively.
>>>>
>>>> The CDC index files have been deleted by the CDC
>>>> consumer, and we don't keep a copy of these files
>>>> elsewhere, so I cannot check the offsets in them
>>>> unfortunately.
>>>>
>>>> Regarding the WaitQueue, I found the queue to be
>>>> empty in the heap dump, which means there's no
>>>> RegisteredSignal objects in the queue at all. The
>>>> queue is a ConcurrentLinkedQueue object, which has
>>>> a head and a tail. The head has item=null and
>>>> next=null, and the tail has item=null and
>>>> next=head. I've also attempted to search for all
>>>> RegisteredSignal objects in the heap, and found 15
>>>> of them, but none of them is related to the
>>>> read-hotness-tracker:1 thread. They are for the
>>>> following threads:
>>>>
>>>> * MemtablePostFlush:1
>>>> * AntiEntropyStage:1
>>>> * MemtableFlushWriter:910
>>>> * COMMIT-LOG-ALLOCATOR
>>>> * NativePoolCleaner
>>>> * MemtableFlushWriter:911
>>>> * Native-Transport-Requests-135
>>>> * Native-Transport-Requests-139
>>>> * Native-Transport-Requests-132
>>>> * Native-Transport-Requests-143
>>>> * Native-Transport-Requests-129
>>>> * Native-Transport-Requests-140
>>>> * Native-Transport-Requests-153
>>>> * Native-Transport-Requests-141
>>>> * Native-Transport-Requests-137
>>>>
>>>> This looks rather suspicious to me, as the
>>>> read-hotness-tracker:1 thread is blocked and
>>>> waiting for a non-existence registered signal.
>>>>
>>>> Cheers,
>>>> Bowen
>>>>
>>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>>> Hi Bowen,
>>>>>
>>>>> lastSyncedAt is updated by taking pollStarted:
>>>>> lastSyncedAt = pollStarted;
>>>>> where long pollStarted = clock.now();
>>>>> the logic uses clock from: SyncRunnable sync = new
>>>>> SyncRunnable(preciseTime)
>>>>> preciseTime by default is SystemClock
>>>>> clock.now() ->
>>>>> org.apache.cassandra.utils.Clock.Default#nanoTime
>>>>> -> System.nanoTime()
>>>>> so, the field is populated from System.nanoTime()
>>>>> which returns a relative timestamp, which cannot
>>>>> be treated as an absolute time like
>>>>> linux epoch (it is calculated like this:
>>>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>>>> )
>>>>>
>>>>> Can you share your cassandra.yaml file? (with
>>>>> removed IPs, paths and other security sensitive
>>>>> things, of course)
>>>>> Do you have any exceptions in your system.log
>>>>> related to commit log?
>>>>> Do you have JVM stdout/stderr captured to a file?
>>>>> Does it have any exceptions? (which can be present
>>>>> there if Cassandra does not catch them properly by
>>>>> itself)
>>>>>
>>>>> ---
>>>>> How many CommitLogSegment objects do you have in
>>>>> your heap dump?
>>>>> What are values for the following fields of
>>>>> CommitLogSegment objects?
>>>>> lastSyncedOffset
>>>>> lastMarkerOffset
>>>>> cdcState
>>>>>
>>>>> Do you have CDC index files written by
>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>>>>> ?
>>>>> If you have them - what are the last offsets
>>>>> written there?
>>>>>
>>>>> Regards,
>>>>> Dmitry
>>>>>
>>>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>>>>> <us...@cassandra.apache.org> wrote:
>>>>>
>>>>> The syncComplete is an instance of WaitQueue,
>>>>> and the syncComplete.queue is an instance of
>>>>> ConcurrentLinkedQueue. Surprisingly, the queue
>>>>> is empty. There is no item in the queue's
>>>>> linked list, only the head and tail nodes,
>>>>> each has item=null.
>>>>>
>>>>> The usage of the WaitQueue within the
>>>>> AbstractCommitLogService looks correct to me:
>>>>>
>>>>> do
>>>>> {
>>>>> WaitQueue.Signal signal = context != null ?
>>>>> syncComplete.register(context, Context::stop)
>>>>> : syncComplete.register();
>>>>> if (lastSyncedAt < syncTime)
>>>>> signal.awaitUninterruptibly();
>>>>> else
>>>>> signal.cancel();
>>>>> }
>>>>> while (lastSyncedAt < syncTime);
>>>>>
>>>>> This matches the example given in the
>>>>> WaitQueue's documentation (which is definitely
>>>>> missing a pair of curly brackets and has bad
>>>>> indentation):
>>>>>
>>>>> while (!conditionMet())
>>>>> Signal s = q.register();
>>>>> if (!conditionMet()) // or, perhaps
>>>>> more correctly, !conditionChanged()
>>>>> s.await();
>>>>> else
>>>>> s.cancel();
>>>>>
>>>>> So, why is the queue empty while the thread is
>>>>> blocked on waiting for the signal?
>>>>>
>>>>> Further digging showed that
>>>>> PeriodicCommitLogService.lastSyncedAt =
>>>>> 1573038452910796, which translates to 6
>>>>> November 2019 11:07:32.910796, about 5 years ago.
>>>>>
>>>>> Is this suppose to be a wall clock time in
>>>>> milliseconds since 1 January 1970 00:00? Or is
>>>>> it a monotonic clock time that has no meaning
>>>>> out of the context, and I'm just wrongly
>>>>> interpreting its value as a wall clock time?
>>>>>
>>>>>
>>>>> I've found that the syncComplete.queue is
>>>>> empty, meaning the WaitQueue object believes
>>>>> that there's nothing waiting for the signal,
>>>>> yet the "read-hotness-tracker:1" thread is
>>>>> clearly waiting for it.
>>>>>
>>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>>>
>>>>>> I think I'm getting really close now. This
>>>>>> seems to have something to do with the
>>>>>> "read-hotness-tracker:1" thread. The thread
>>>>>> dump is:
>>>>>>
>>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93
>>>>>> WAITING
>>>>>> at jdk.internal.misc.Unsafe.park(Native
>>>>>> Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>>> at
>>>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)<------
>>>>>> here
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> local variable:
>>>>>> com.codahale.metrics.Timer$Context#2086795
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.CommitLog#1
>>>>>> local variable:
>>>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>>> at
>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>>> at
>>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>>>> <------ and here
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>>> at
>>>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.Keyspace#8
>>>>>> local variable:
>>>>>> org.apache.cassandra.db.Mutation#54491
>>>>>> at
>>>>>> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>>> at
>>>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>>> at
>>>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>>> at
>>>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>>> at
>>>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>>> local variable: java.lang.String#42806
>>>>>> local variable: java.lang.String#47659
>>>>>> local variable:
>>>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>>> local variable:
>>>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>>>> at
>>>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>>> local variable:
>>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>>> local variable:
>>>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>>> local variable:
>>>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>>>> at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>>> local variable:
>>>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>>> local variable:
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>> local variable:
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>>> local variable:
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>>> at
>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>>> local variable:
>>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>>
>>>>>>
>>>>>> In the
>>>>>> CassandraKeyspaceWriteHandler.beginWrite()
>>>>>> method:
>>>>>>
>>>>>> public WriteContext beginWrite(Mutation
>>>>>> mutation, boolean makeDurable) throws
>>>>>> RequestExecutionException
>>>>>> {
>>>>>> OpOrder.Group group = null;
>>>>>> try
>>>>>> {
>>>>>> group = Keyspace.writeOrder.start();
>>>>>> // <--- here
>>>>>>
>>>>>> // write the mutation to the
>>>>>> commitlog and memtables
>>>>>> CommitLogPosition position = null;
>>>>>> if (makeDurable)
>>>>>> {
>>>>>> position = addToCommitLog(mutation);// <---
>>>>>> and here
>>>>>> }
>>>>>> return new
>>>>>> CassandraWriteContext(group, position);
>>>>>> }
>>>>>> catch (Throwable t)
>>>>>> {
>>>>>> if (group != null)
>>>>>> {
>>>>>> group.close();
>>>>>> }
>>>>>> throw t;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> The Keyspace.writeOrder.start() method call
>>>>>> blocks the MemtableFlushWriter thread, and
>>>>>> then it calls addToCommitLog(mutation) which
>>>>>> indirectly calls
>>>>>> AbstractCommitLogService.awaitSyncAt()
>>>>>>
>>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>>> {
>>>>>> do
>>>>>> {
>>>>>> WaitQueue.Signal signal = context != null ?
>>>>>> syncComplete.register(context, Context::stop)
>>>>>> : syncComplete.register();
>>>>>> if (lastSyncedAt < syncTime)
>>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>>> else
>>>>>> signal.cancel();
>>>>>> }
>>>>>> while (lastSyncedAt < syncTime);
>>>>>> }
>>>>>>
>>>>>> This then got stuck on the
>>>>>> signal.awaitUninterruptibly()
>>>>>>
>>>>>> Now I know what is blocking the memtable
>>>>>> flushing, but I haven't been able to figure
>>>>>> out is why it got stuck on waiting for that
>>>>>> signal.
>>>>>>
>>>>>> I would appreciate it if anyone can offer
>>>>>> some insight here.
>>>>>>
>>>>>>
>>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>>>
>>>>>>> I will give it a try and see what I can
>>>>>>> find. I plan to go down the rabbit hole
>>>>>>> tomorrow. Will keep you updated.
>>>>>>>
>>>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via
>>>>>>>>> user <us...@cassandra.apache.org>
>>>>>>>>> <ma...@cassandra.apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Writes on this node starts to timeout and
>>>>>>>>> fail. But if left untouched, it's only
>>>>>>>>> gonna get worse, and eventually lead to
>>>>>>>>> JVM OOM and crash.
>>>>>>>>>
>>>>>>>>> By inspecting the heap dump created at
>>>>>>>>> OOM, we can see that both of the
>>>>>>>>> MemtableFlushWriter threads are stuck on
>>>>>>>>> line 1190
>>>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>>>>> in the ColumnFamilyStore.java:
>>>>>>>>>
>>>>>>>>> // mark writes older than the
>>>>>>>>> barrier as blocking progress, permitting
>>>>>>>>> them to exceed our memory limit
>>>>>>>>> // if they are stuck waiting
>>>>>>>>> on it, then wait for them all to complete
>>>>>>>>> writeBarrier.markBlocking();
>>>>>>>>> writeBarrier.await(); // <-----------
>>>>>>>>> stuck here
>>>>>>>>>
>>>>>>>>> And the MemtablePostFlush thread is stuck
>>>>>>>>> on line 1094
>>>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>>>>> in the same file.
>>>>>>>>>
>>>>>>>>> try
>>>>>>>>> {
>>>>>>>>> // we wait on the latch
>>>>>>>>> for the commitLogUpperBound to be set, and
>>>>>>>>> so that waiters
>>>>>>>>> // on this task can rely
>>>>>>>>> on all prior flushes being complete
>>>>>>>>> latch.await(); //
>>>>>>>>> <----------- stuck here
>>>>>>>>> }
>>>>>>>>> Our top suspect is CDC interacting with
>>>>>>>>> repair, since this started to happen
>>>>>>>>> shortly after we enabled CDC on the nodes,
>>>>>>>>> and each time repair was running. But we
>>>>>>>>> have not been able to reproduce this in a
>>>>>>>>> testing cluster, and don't know what's the
>>>>>>>>> next step to troubleshoot this issue. So
>>>>>>>>> I'm posting it in the mailing lists and
>>>>>>>>> hoping someone may know something about it
>>>>>>>>> or point me to the right direction.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Wouldn’t be completely surprised if CDC
>>>>>>>> or repair somehow has a barrier, I’ve also
>>>>>>>> seen similar behavior pre-3.0 with “very
>>>>>>>> long running read commands” that have a
>>>>>>>> barrier on the memtable that prevent release.
>>>>>>>>
>>>>>>>> You’ve got the heap (great, way better than
>>>>>>>> most people debugging), are you able to
>>>>>>>> navigate through it and look for references
>>>>>>>> to that memtable or other things holding a
>>>>>>>> barrier?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dmitry Konstantinov
>>>>
>>>>
>>>>
>>>> --
>>>> Dmitry Konstantinov
>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Thank you. I think I have got the cause, it is a non-trivial puzzle which
took a few hours :-).
So, we have the following stacktrace for "read-hotness-tracker:1" thread
"read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
local variable: com.codahale.metrics.Timer$Context#2086795
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
local variable: org.apache.cassandra.db.commitlog.CommitLog#1
local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
local variable:
org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
at
org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
at
org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
local variable:
org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
local variable: org.apache.cassandra.db.Keyspace#8
local variable: org.apache.cassandra.db.Mutation#54491
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
at
org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
local variable: java.lang.String#42806
local variable: java.lang.String#47659
local variable:
org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
local variable: org.apache.cassandra.metrics.RestorableMeter#752
at
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
where we are waiting in the await() method of
WaitQueue$Standard$SignalWithListener#2086139 object.
another input: WaitQueue$Standard$SignalWithListener#2086139 has the
following field values based on the screenshots:
state = *-1* (it means CANCELLED:
org.apache.cassandra.utils.concurrent.WaitQueue.Standard#CANCELLED = -1)
thread = null
According to the code in WaitQueue.java such combination of values only
possible if we invoked cancel() on this SignalWithListener object but we
didn't do it in our code, it is in another branch of
AbstractCommitLogService.awaitSyncAt
but let's check the await() logic more carefully:
public Signal await() throws InterruptedException
{
while (!isSignalled())
{
checkInterrupted();
LockSupport.park(); <--- we are here
}
checkAndClear();
return this;
}
and inspect unassuming checkInterrupted() method
private void checkInterrupted() throws InterruptedException
{
if (Thread.interrupted())
{
*cancel();*
throw new InterruptedException();
}
}
here we see that if the currently awaiting thread is interrupted then
SignalWithListener cancels itself.
but we throw new InterruptedException() here, so the execution should fail
with it.
Checking the thread stack trace and see that AbstractSignal.await() is
invoked from
org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
how it looks inside:
public static <A extends Awaitable> A awaitUninterruptibly(A await)
{
boolean interrupted = false;
while (true)
{
try
{
await.await();
break;
}
catch (InterruptedException e)
{
interrupted = true;
}
}
if (interrupted)
Thread.currentThread().interrupt();
return await;
}
Here we have a loop, if the current thread is interrupted - we have
InterruptedException thrown by await.await(), we catch it here and try
await.await() again. But in the previous invocation of await.await() we
have cancelled our Signal object, so the condition inside await() method:
isSignalled(): state == SIGNALLED is always false now plus if another
thread will invoke notifyAll - nothing will happen, because as a part of
cancellation our Signal object unregister itself from the queue and the
notifying thread will not able to find it to notify.
I have reproduced the case with a simple unit test.
The last element of the puzzle - why is the thread interrupted? To
understand it - let's check there "read-hotness-tracker:1" thread execution
started. It is a background periodical runnable in
org.apache.cassandra.io.sstable.format.SSTableReader.GlobalTidy
readMeterSyncFuture = new
WeakReference<>(syncExecutor.scheduleAtFixedRate(new Runnable()
{
public void run()
{
if (obsoletion == null)
{
meterSyncThrottle.acquire();
SystemKeyspace.persistSSTableReadMeter(desc.ksname,
desc.cfname, desc.id, readMeter);
}
}
}, 1, 5, TimeUnit.MINUTES));
let's check how we deal with readMeterSyncFuture ref and we see our
interruption:
private void stopReadMeterPersistence()
{
ScheduledFuture<?> readMeterSyncFutureLocal = readMeterSyncFuture.get();
if (readMeterSyncFutureLocal != null)
{
* readMeterSyncFutureLocal.cancel(true); *<========= here we can
interrupt the thread
readMeterSyncFuture = NULL;
}
}
stopReadMeterPersistence() is invoked by one of the following branches of
logic when an SSTable is not in use anymore:
org.apache.cassandra.io.sstable.format.SSTableReader#markObsolete
org.apache.cassandra.db.lifecycle.Helpers#markObsolete
org.apache.cassandra.db.lifecycle.Tracker#dropSSTables(...)
org.apache.cassandra.db.lifecycle.LifecycleTransaction#doCommit
org.apache.cassandra.db.lifecycle.LifecycleTransaction#doAbort
So,the issue is caused by a combination of events with a right timing:
- an SSTable which was become obsolete at the middle of
persistSSTableReadMeter background job execution for this SSTable, so we
interrupted the job
- persistSSTableReadMeter was at that moment in the middle of write
operation, at commit log append step
- and the commit log append step was stuck due to a delay in commit log
sync (lastSyncedAt < syncTime condition in AbstractCommitLogService), so we
started to wait for it using await
The last observation - timing.
We have
org.apache.cassandra.db.commitlog.AbstractCommitLogService
lastSyncedAt = 1573038452910796
com.codahale.metrics.Timer$Context#2086795 (created when the write
operation started to wait for commit log sync)
startTime = 1410129608743025
both are from System.nanoTime()
so, 1573038452910796 - 1410129608743025 = 162 908 844 167 771 ns = 162 908
sec ~ 45 hours; the cause happened about that time ago.
I will create a ticket for this issue to fix it.
Regards,
Dmitry
On Wed, 13 Nov 2024 at 22:11, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Please see the screenshots below.
>
> Unfortunately that the WaitQueue$Standard$SignalWithListener#2086139
> object is the GC root, and there's no references to it.
>
> I hope this helps.
>
>
>
>
> On 13/11/2024 21:33, Dmitry Konstantinov wrote:
>
> Hi Bowen, thank you for the details.
> Could you please also share a screenshot (Visual VM should be ok) for "read-hotness-tracker:1"
> thread with expanded local variable elements for the following lines:
> 1) at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
> particularly com.codahale.metrics.Timer$Context#2086795 internals such as
> startTime
> 2) at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
> particularly WaitQueue$Standard$SignalWithListener#2086139 internals such
> as thread, state
>
> Also, could you please render all GC paths to root for WaitQueue$Standard$SignalWithListener#2086139
> object and share the correspondent tree view?
>
> Regards,
> Dmitry
>
>
> On Tue, 12 Nov 2024 at 11:40, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Just tried Eclipse Memory Analyzer, it shows the exact same local
>> variables as VisualVM, so there is no difference here.
>>
>> However, Eclipse Memory Analyzer is significantly slower (lagging) to
>> operate via X window forwarding or VNC (possibly due to the GUI decorations
>> and animations) and some features rely on a built-in web browser which I
>> can't get it to work on a headless server. I'm doing the analysis on a
>> headless server because my PC doesn't have enough RAM to load the large
>> heap dump file.
>>
>> See screenshot below for a comparison between Eclipse Memory Analyzer and
>> VisualVM on the variables referenced by the PERIODIC-COMMIT-LOG-SYNCER
>> thread.
>>
>> Eclipse Memory Analyzer
>>
>> VisualVM
>>
>>
>> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>>
>> >> If Eclipse Memory Analyzer also can't find the mysterious item, how do
>> I proceed from there? What else can I do?
>> We can analyze other variables from the heap dump, read code and repeat,
>> until we find a clue :-). As of now I do not see a much better option. If
>> the issue reproduces regularly - a custom build with extra logs can be done
>> to add more diagnostic info printing.
>> It is actually lucky that you have a heap dump, it can give a lot of
>> info, unfortunately only in a moment of time (so we cannot see from the
>> issue in dynamic).
>>
>> >> This has happened twice in the last 10 days, to two different nodes in
>> the same cluster.
>> Do you have heap dumps for the other failures?
>> How much time did you have between a node startup and the failure?
>>
>> Can you share field values
>> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
>> using Eclipse Memory Analyzer
>> from the following thread?:
>> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#5
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
>> local variable: java.util.concurrent.Semaphore$NonfairSync#1
>> local variable:
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
>> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
>> at
>> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>>
>> It should looks like this:
>>
>> [image: image.png]
>>
>> [image: image.png]
>>
>> Regards,
>> Dmitry
>>
>>
>> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Hi Dmitry,
>>>
>>> It indeed is very strange, either the tool I'm using is wrong, or
>>> there's something causing the item to evaporate from the queue mysteriously
>>> or never got into the queue.
>>>
>>> I've been using VisualVM to inspect the heap dump file and using OQL to
>>> find and explore the objects in it.
>>>
>>> VisualVM does show some local variables, but not all of them. This isn't
>>> an issue for types with little number of instances, because I can search
>>> objects by type to find them. But it can be troublesome when an object is a
>>> very common type.
>>>
>>> I will try Eclipse Memory Analyzer next week to see if it can find the
>>> missing item, but I have my doubts.
>>>
>>> I've read the code of WaitQueue and the
>>> AbstractCommitLogService.awaitSyncAt() method, and they both look correct
>>> to me. If Eclipse Memory Analyzer also can't find the mysterious item, how
>>> do I proceed from there? What else can I do?
>>>
>>> Cheers,
>>> Bowen
>>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>>
>>> Hi Bowen,
>>>
>>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
>>> looks like the code is inconsistent on whether lastSyncedAt is a relative
>>> monotonic clock or absolute epoch time.
>>> Yes, I think so, I am going to create a defect regarding it.
>>>
>>> >> I've also attempted to search for all RegisteredSignal objects in
>>> the heap, and found 15 of them, but none of them is related to the
>>> read-hotness-tracker:1 thread.
>>> It is very strange because the stack trace for this thread waits within
>>> the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
>>> signal" local variable..
>>> What tool do you use for the heap dump inspection? I use Eclipse Memory
>>> Analyzer. It has "threads_overview" view and you can check the values for
>>> local variables within a thread stack.
>>> Like this:
>>> [image: image.png]
>>>
>>> Regards,
>>> Dmitry
>>>
>>>
>>>
>>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> Thank you for helping me.
>>>>
>>>> You are right, that's a monotonic clock. I was definitely wrongly
>>>> interpreting its value. I got misled by this line of code
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>>>> like the code is inconsistent on whether lastSyncedAt is a relative
>>>> monotonic clock or absolute epoch time.
>>>>
>>>> Please see the attached cassandra.yaml with sensitive information
>>>> masked out with asterisks.
>>>>
>>>> There's no ERROR level log in the system.log other than some guardrail
>>>> violations (items_per_collection, if it matters) and random and rare
>>>> network issues such as connection timeout in the few days run up to and
>>>> during the event.
>>>>
>>>> We have the stdout and stderr captured in the systemd journal, and they
>>>> have been quiet in during that period as well, not a single line got logged.
>>>>
>>>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>>>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>>>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>>>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>>>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>>>> respectively.
>>>>
>>>> The CDC index files have been deleted by the CDC consumer, and we don't
>>>> keep a copy of these files elsewhere, so I cannot check the offsets in them
>>>> unfortunately.
>>>>
>>>> Regarding the WaitQueue, I found the queue to be empty in the heap
>>>> dump, which means there's no RegisteredSignal objects in the queue at all.
>>>> The queue is a ConcurrentLinkedQueue object, which has a head and a tail.
>>>> The head has item=null and next=null, and the tail has item=null and
>>>> next=head. I've also attempted to search for all RegisteredSignal objects
>>>> in the heap, and found 15 of them, but none of them is related to the
>>>> read-hotness-tracker:1 thread. They are for the following threads:
>>>>
>>>> - MemtablePostFlush:1
>>>> - AntiEntropyStage:1
>>>> - MemtableFlushWriter:910
>>>> - COMMIT-LOG-ALLOCATOR
>>>> - NativePoolCleaner
>>>> - MemtableFlushWriter:911
>>>> - Native-Transport-Requests-135
>>>> - Native-Transport-Requests-139
>>>> - Native-Transport-Requests-132
>>>> - Native-Transport-Requests-143
>>>> - Native-Transport-Requests-129
>>>> - Native-Transport-Requests-140
>>>> - Native-Transport-Requests-153
>>>> - Native-Transport-Requests-141
>>>> - Native-Transport-Requests-137
>>>>
>>>> This looks rather suspicious to me, as the read-hotness-tracker:1
>>>> thread is blocked and waiting for a non-existence registered signal.
>>>>
>>>> Cheers,
>>>> Bowen
>>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>>
>>>> Hi Bowen,
>>>>
>>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>>> pollStarted;
>>>> where long pollStarted = clock.now();
>>>> the logic uses clock from: SyncRunnable sync = new
>>>> SyncRunnable(preciseTime)
>>>> preciseTime by default is SystemClock
>>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>>> System.nanoTime()
>>>> so, the field is populated from System.nanoTime() which returns a
>>>> relative timestamp, which cannot be treated as an absolute time like
>>>> linux epoch (it is calculated like this:
>>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>>> )
>>>>
>>>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>>>> other security sensitive things, of course)
>>>> Do you have any exceptions in your system.log related to commit log?
>>>> Do you have JVM stdout/stderr captured to a file? Does it have any
>>>> exceptions? (which can be present there if Cassandra does not catch them
>>>> properly by itself)
>>>>
>>>> ---
>>>> How many CommitLogSegment objects do you have in your heap dump?
>>>> What are values for the following fields of CommitLogSegment objects?
>>>> lastSyncedOffset
>>>> lastMarkerOffset
>>>> cdcState
>>>>
>>>> Do you have CDC index files written by
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>>> If you have them - what are the last offsets written there?
>>>>
>>>> Regards,
>>>> Dmitry
>>>>
>>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>>>> user@cassandra.apache.org> wrote:
>>>>
>>>>> The syncComplete is an instance of WaitQueue, and the
>>>>> syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly,
>>>>> the queue is empty. There is no item in the queue's linked list, only the
>>>>> head and tail nodes, each has item=null.
>>>>>
>>>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>>>> correct to me:
>>>>>
>>>>> do
>>>>> {
>>>>> WaitQueue.Signal signal = context != null ?
>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>> if (lastSyncedAt < syncTime)
>>>>> signal.awaitUninterruptibly();
>>>>> else
>>>>> signal.cancel();
>>>>> }
>>>>> while (lastSyncedAt < syncTime);
>>>>>
>>>>> This matches the example given in the WaitQueue's documentation (which
>>>>> is definitely missing a pair of curly brackets and has bad indentation):
>>>>>
>>>>> while (!conditionMet())
>>>>> Signal s = q.register();
>>>>> if (!conditionMet()) // or, perhaps more correctly,
>>>>> !conditionChanged()
>>>>> s.await();
>>>>> else
>>>>> s.cancel();
>>>>>
>>>>> So, why is the queue empty while the thread is blocked on waiting for
>>>>> the signal?
>>>>>
>>>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>>>> about 5 years ago.
>>>>>
>>>>> Is this suppose to be a wall clock time in milliseconds since 1
>>>>> January 1970 00:00? Or is it a monotonic clock time that has no meaning out
>>>>> of the context, and I'm just wrongly interpreting its value as a wall clock
>>>>> time?
>>>>>
>>>>>
>>>>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>>>>> object believes that there's nothing waiting for the signal, yet the
>>>>> "read-hotness-tracker:1" thread is clearly waiting for it.
>>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>>
>>>>> I think I'm getting really close now. This seems to have something to
>>>>> do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>>>
>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>>>> <------ here
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>>> local variable:
>>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>> at
>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>> at
>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>>> <------ and here
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>> at
>>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>> at
>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>> at
>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>> at
>>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>> at
>>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>> local variable: java.lang.String#42806
>>>>> local variable: java.lang.String#47659
>>>>> local variable:
>>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>>>> at
>>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>> at
>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>> local variable:
>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>> local variable:
>>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>> local variable:
>>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>> local variable:
>>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> at
>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>> local variable:
>>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>
>>>>>
>>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>>
>>>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>>>>> throws RequestExecutionException
>>>>> {
>>>>> OpOrder.Group group = null;
>>>>> try
>>>>> {
>>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>>
>>>>> // write the mutation to the commitlog and memtables
>>>>> CommitLogPosition position = null;
>>>>> if (makeDurable)
>>>>> {
>>>>> position = addToCommitLog(mutation); // <--- and here
>>>>> }
>>>>> return new CassandraWriteContext(group, position);
>>>>> }
>>>>> catch (Throwable t)
>>>>> {
>>>>> if (group != null)
>>>>> {
>>>>> group.close();
>>>>> }
>>>>> throw t;
>>>>> }
>>>>> }
>>>>>
>>>>> The Keyspace.writeOrder.start() method call blocks the
>>>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation)
>>>>> which indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>>>
>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>> {
>>>>> do
>>>>> {
>>>>> WaitQueue.Signal signal = context != null ?
>>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>> if (lastSyncedAt < syncTime)
>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>> else
>>>>> signal.cancel();
>>>>> }
>>>>> while (lastSyncedAt < syncTime);
>>>>> }
>>>>>
>>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>>
>>>>> Now I know what is blocking the memtable flushing, but I haven't been
>>>>> able to figure out is why it got stuck on waiting for that signal.
>>>>>
>>>>> I would appreciate it if anyone can offer some insight here.
>>>>>
>>>>>
>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>
>>>>> I will give it a try and see what I can find. I plan to go down the
>>>>> rabbit hole tomorrow. Will keep you updated.
>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>>>
>>>>> Writes on this node starts to timeout and fail. But if left untouched,
>>>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>>>
>>>>> By inspecting the heap dump created at OOM, we can see that both of
>>>>> the MemtableFlushWriter threads are stuck on line 1190
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>> in the ColumnFamilyStore.java:
>>>>>
>>>>> // mark writes older than the barrier as blocking
>>>>> progress, permitting them to exceed our memory limit
>>>>> // if they are stuck waiting on it, then wait for them all
>>>>> to complete
>>>>> writeBarrier.markBlocking();
>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>
>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>> in the same file.
>>>>>
>>>>> try
>>>>> {
>>>>> // we wait on the latch for the commitLogUpperBound to
>>>>> be set, and so that waiters
>>>>> // on this task can rely on all prior flushes being
>>>>> complete
>>>>> latch.await(); // <----------- stuck here
>>>>> }
>>>>> Our top suspect is CDC interacting with repair, since this started to
>>>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>>>> running. But we have not been able to reproduce this in a testing cluster,
>>>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>>>> posting it in the mailing lists and hoping someone may know something about
>>>>> it or point me to the right direction.
>>>>>
>>>>>
>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>>>> read commands” that have a barrier on the memtable that prevent release.
>>>>>
>>>>> You’ve got the heap (great, way better than most people debugging),
>>>>> are you able to navigate through it and look for references to that
>>>>> memtable or other things holding a barrier?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Dmitry Konstantinov
>>>>
>>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Please see the screenshots below.
Unfortunately that the WaitQueue$Standard$SignalWithListener#2086139
object is the GC root, and there's no references to it.
I hope this helps.
On 13/11/2024 21:33, Dmitry Konstantinov wrote:
> Hi Bowen, thank you for the details.
> Could you please also share a screenshot (Visual VM should be ok) for
> "read-hotness-tracker:1" thread with expanded local variable elements
> for the following lines:
> 1) at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
> particularly com.codahale.metrics.Timer$Context#2086795 internals such
> as startTime
> 2) at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
> particularly WaitQueue$Standard$SignalWithListener#2086139 internals
> such as thread, state
>
> Also, could you please render all GC paths to root for
> WaitQueue$Standard$SignalWithListener#2086139 object and share the
> correspondent tree view?
>
> Regards,
> Dmitry
>
>
> On Tue, 12 Nov 2024 at 11:40, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Just tried Eclipse Memory Analyzer, it shows the exact same local
> variables as VisualVM, so there is no difference here.
>
> However, Eclipse Memory Analyzer is significantly slower (lagging)
> to operate via X window forwarding or VNC (possibly due to the GUI
> decorations and animations) and some features rely on a built-in
> web browser which I can't get it to work on a headless server. I'm
> doing the analysis on a headless server because my PC doesn't have
> enough RAM to load the large heap dump file.
>
> See screenshot below for a comparison between Eclipse Memory
> Analyzer and VisualVM on the variables referenced by the
> PERIODIC-COMMIT-LOG-SYNCER thread.
>
> Eclipse Memory Analyzer
>
> VisualVM
>
>
> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>> >> If Eclipse Memory Analyzer also can't find the mysterious
>> item, how do I proceed from there? What else can I do?
>> We can analyze other variables from the heap dump, read code and
>> repeat, until we find a clue :-). As of now I do not see a much
>> better option. If the issue reproduces regularly - a custom build
>> with extra logs can be done to add more diagnostic info printing.
>> It is actually lucky that you have a heap dump, it can give a lot
>> of info, unfortunately only in a moment of time (so we cannot see
>> from the issue in dynamic).
>>
>> >> This has happened twice in the last 10 days, to two different
>> nodes in the same cluster.
>> Do you have heap dumps for the other failures?
>> How much time did you have between a node startup and the failure?
>>
>> Can you share field values
>> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService
>> and
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
>> using Eclipse Memory Analyzer
>> from the following thread?:
>> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalThread#5
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
>> local variable: java.util.concurrent.Semaphore$NonfairSync#1
>> local variable:
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
>> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
>> at
>> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>>
>> It should looks like this:
>>
>> image.png
>>
>> image.png
>>
>> Regards,
>> Dmitry
>>
>>
>> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Hi Dmitry,
>>
>> It indeed is very strange, either the tool I'm using is
>> wrong, or there's something causing the item to evaporate
>> from the queue mysteriously or never got into the queue.
>>
>> I've been using VisualVM to inspect the heap dump file and
>> using OQL to find and explore the objects in it.
>>
>> VisualVM does show some local variables, but not all of them.
>> This isn't an issue for types with little number of
>> instances, because I can search objects by type to find them.
>> But it can be troublesome when an object is a very common type.
>>
>> I will try Eclipse Memory Analyzer next week to see if it can
>> find the missing item, but I have my doubts.
>>
>> I've read the code of WaitQueue and the
>> AbstractCommitLogService.awaitSyncAt() method, and they both
>> look correct to me. If Eclipse Memory Analyzer also can't
>> find the mysterious item, how do I proceed from there? What
>> else can I do?
>>
>> Cheers,
>> Bowen
>>
>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>> Hi Bowen,
>>>
>>> >> protected volatile long lastSyncedAt =
>>> currentTimeMillis();". It looks like the code is
>>> inconsistent on whether lastSyncedAt is a relative monotonic
>>> clock or absolute epoch time.
>>> Yes, I think so, I am going to create a defect regarding it.
>>>
>>> >> I've also attempted to search for all RegisteredSignal
>>> objects in the heap, and found 15 of them, but none of them
>>> is related to the read-hotness-tracker:1 thread.
>>> It is very strange because the stack trace for this thread
>>> waits within the object and read-hotness-tracker:1 thread
>>> has active "WaitQueue.Signal signal" local variable..
>>> What tool do you use for the heap dump inspection? I use
>>> Eclipse Memory Analyzer. It has "threads_overview" view and
>>> you can check the values for local variables within a thread
>>> stack.
>>> Like this:
>>> image.png
>>>
>>> Regards,
>>> Dmitry
>>>
>>>
>>>
>>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> Hi Dmitry,
>>>
>>> Thank you for helping me.
>>>
>>> You are right, that's a monotonic clock. I was
>>> definitely wrongly interpreting its value. I got misled
>>> by this line of code
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>> "protected volatile long lastSyncedAt =
>>> currentTimeMillis();". It looks like the code is
>>> inconsistent on whether lastSyncedAt is a relative
>>> monotonic clock or absolute epoch time.
>>>
>>> Please see the attached cassandra.yaml with sensitive
>>> information masked out with asterisks.
>>>
>>> There's no ERROR level log in the system.log other than
>>> some guardrail violations (items_per_collection, if it
>>> matters) and random and rare network issues such as
>>> connection timeout in the few days run up to and during
>>> the event.
>>>
>>> We have the stdout and stderr captured in the systemd
>>> journal, and they have been quiet in during that period
>>> as well, not a single line got logged.
>>>
>>> There's 196 CommitLogSegment (more specifically,
>>> MemoryMappedSegment) objects in the heap dump. Of these,
>>> 194 has cdcState = CONTAINS lastSyncedOffset = 33554432
>>> lastMarkerOffset = 33554432. The other two has cdcState
>>> = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset
>>> = 27007818 and cdcState = PERMITTED lastSyncedOffset =
>>> 20 lastMarkerOffset = 20 respectively.
>>>
>>> The CDC index files have been deleted by the CDC
>>> consumer, and we don't keep a copy of these files
>>> elsewhere, so I cannot check the offsets in them
>>> unfortunately.
>>>
>>> Regarding the WaitQueue, I found the queue to be empty
>>> in the heap dump, which means there's no
>>> RegisteredSignal objects in the queue at all. The queue
>>> is a ConcurrentLinkedQueue object, which has a head and
>>> a tail. The head has item=null and next=null, and the
>>> tail has item=null and next=head. I've also attempted to
>>> search for all RegisteredSignal objects in the heap, and
>>> found 15 of them, but none of them is related to the
>>> read-hotness-tracker:1 thread. They are for the
>>> following threads:
>>>
>>> * MemtablePostFlush:1
>>> * AntiEntropyStage:1
>>> * MemtableFlushWriter:910
>>> * COMMIT-LOG-ALLOCATOR
>>> * NativePoolCleaner
>>> * MemtableFlushWriter:911
>>> * Native-Transport-Requests-135
>>> * Native-Transport-Requests-139
>>> * Native-Transport-Requests-132
>>> * Native-Transport-Requests-143
>>> * Native-Transport-Requests-129
>>> * Native-Transport-Requests-140
>>> * Native-Transport-Requests-153
>>> * Native-Transport-Requests-141
>>> * Native-Transport-Requests-137
>>>
>>> This looks rather suspicious to me, as the
>>> read-hotness-tracker:1 thread is blocked and waiting for
>>> a non-existence registered signal.
>>>
>>> Cheers,
>>> Bowen
>>>
>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>> Hi Bowen,
>>>>
>>>> lastSyncedAt is updated by taking pollStarted:
>>>> lastSyncedAt = pollStarted;
>>>> where long pollStarted = clock.now();
>>>> the logic uses clock from: SyncRunnable sync = new
>>>> SyncRunnable(preciseTime)
>>>> preciseTime by default is SystemClock
>>>> clock.now() ->
>>>> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>>> System.nanoTime()
>>>> so, the field is populated from System.nanoTime() which
>>>> returns a relative timestamp, which cannot be treated
>>>> as an absolute time like linux epoch (it is calculated
>>>> like this:
>>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>>> )
>>>>
>>>> Can you share your cassandra.yaml file? (with removed
>>>> IPs, paths and other security sensitive things, of course)
>>>> Do you have any exceptions in your system.log related
>>>> to commit log?
>>>> Do you have JVM stdout/stderr captured to a file? Does
>>>> it have any exceptions? (which can be present there if
>>>> Cassandra does not catch them properly by itself)
>>>>
>>>> ---
>>>> How many CommitLogSegment objects do you have in your
>>>> heap dump?
>>>> What are values for the following fields of
>>>> CommitLogSegment objects?
>>>> lastSyncedOffset
>>>> lastMarkerOffset
>>>> cdcState
>>>>
>>>> Do you have CDC index files written by
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>>>> ?
>>>> If you have them - what are the last offsets written there?
>>>>
>>>> Regards,
>>>> Dmitry
>>>>
>>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>>>> <us...@cassandra.apache.org> wrote:
>>>>
>>>> The syncComplete is an instance of WaitQueue, and
>>>> the syncComplete.queue is an instance of
>>>> ConcurrentLinkedQueue. Surprisingly, the queue is
>>>> empty. There is no item in the queue's linked list,
>>>> only the head and tail nodes, each has item=null.
>>>>
>>>> The usage of the WaitQueue within the
>>>> AbstractCommitLogService looks correct to me:
>>>>
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context !=
>>>> null ? syncComplete.register(context,
>>>> Context::stop) : syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly();
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>>
>>>> This matches the example given in the WaitQueue's
>>>> documentation (which is definitely missing a pair
>>>> of curly brackets and has bad indentation):
>>>>
>>>> while (!conditionMet())
>>>> Signal s = q.register();
>>>> if (!conditionMet()) // or, perhaps more
>>>> correctly, !conditionChanged()
>>>> s.await();
>>>> else
>>>> s.cancel();
>>>>
>>>> So, why is the queue empty while the thread is
>>>> blocked on waiting for the signal?
>>>>
>>>> Further digging showed that
>>>> PeriodicCommitLogService.lastSyncedAt =
>>>> 1573038452910796, which translates to 6 November
>>>> 2019 11:07:32.910796, about 5 years ago.
>>>>
>>>> Is this suppose to be a wall clock time in
>>>> milliseconds since 1 January 1970 00:00? Or is it a
>>>> monotonic clock time that has no meaning out of the
>>>> context, and I'm just wrongly interpreting its
>>>> value as a wall clock time?
>>>>
>>>>
>>>> I've found that the syncComplete.queue is empty,
>>>> meaning the WaitQueue object believes that there's
>>>> nothing waiting for the signal, yet the
>>>> "read-hotness-tracker:1" thread is clearly waiting
>>>> for it.
>>>>
>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>>
>>>>> I think I'm getting really close now. This seems
>>>>> to have something to do with the
>>>>> "read-hotness-tracker:1" thread. The thread dump is:
>>>>>
>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at
>>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)<------
>>>>> here
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> local variable:
>>>>> com.codahale.metrics.Timer$Context#2086795
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.CommitLog#1
>>>>> local variable:
>>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>> local variable:
>>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>> at
>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>> at
>>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>>> <------ and here
>>>>> local variable:
>>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>> at
>>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>> local variable:
>>>>> org.apache.cassandra.db.Keyspace#8
>>>>> local variable:
>>>>> org.apache.cassandra.db.Mutation#54491
>>>>> at
>>>>> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>> at
>>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>> at
>>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>> at
>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>> at
>>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>> at
>>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>> at
>>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>> local variable: java.lang.String#42806
>>>>> local variable: java.lang.String#47659
>>>>> local variable:
>>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>> local variable:
>>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>>> at
>>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>> at
>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>> local variable:
>>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>> local variable:
>>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>> local variable:
>>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>> local variable:
>>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>> local variable:
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> local variable:
>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>> local variable:
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>> local variable:
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> at
>>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>> local variable:
>>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>> local variable:
>>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>
>>>>>
>>>>> In the CassandraKeyspaceWriteHandler.beginWrite()
>>>>> method:
>>>>>
>>>>> public WriteContext beginWrite(Mutation mutation,
>>>>> boolean makeDurable) throws RequestExecutionException
>>>>> {
>>>>> OpOrder.Group group = null;
>>>>> try
>>>>> {
>>>>> group = Keyspace.writeOrder.start(); //
>>>>> <--- here
>>>>>
>>>>> // write the mutation to the commitlog and
>>>>> memtables
>>>>> CommitLogPosition position = null;
>>>>> if (makeDurable)
>>>>> {
>>>>> position = addToCommitLog(mutation);
>>>>> // <--- and here
>>>>> }
>>>>> return new CassandraWriteContext(group,
>>>>> position);
>>>>> }
>>>>> catch (Throwable t)
>>>>> {
>>>>> if (group != null)
>>>>> {
>>>>> group.close();
>>>>> }
>>>>> throw t;
>>>>> }
>>>>> }
>>>>>
>>>>> The Keyspace.writeOrder.start() method call blocks
>>>>> the MemtableFlushWriter thread, and then it calls
>>>>> addToCommitLog(mutation) which indirectly calls
>>>>> AbstractCommitLogService.awaitSyncAt()
>>>>>
>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>> {
>>>>> do
>>>>> {
>>>>> WaitQueue.Signal signal = context !=
>>>>> null ? syncComplete.register(context,
>>>>> Context::stop) : syncComplete.register();
>>>>> if (lastSyncedAt < syncTime)
>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>> else
>>>>> signal.cancel();
>>>>> }
>>>>> while (lastSyncedAt < syncTime);
>>>>> }
>>>>>
>>>>> This then got stuck on the
>>>>> signal.awaitUninterruptibly()
>>>>>
>>>>> Now I know what is blocking the memtable flushing,
>>>>> but I haven't been able to figure out is why it
>>>>> got stuck on waiting for that signal.
>>>>>
>>>>> I would appreciate it if anyone can offer some
>>>>> insight here.
>>>>>
>>>>>
>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>>
>>>>>> I will give it a try and see what I can find. I
>>>>>> plan to go down the rabbit hole tomorrow. Will
>>>>>> keep you updated.
>>>>>>
>>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>>>> <us...@cassandra.apache.org>
>>>>>>>> <ma...@cassandra.apache.org> wrote:
>>>>>>>>
>>>>>>>> Writes on this node starts to timeout and fail.
>>>>>>>> But if left untouched, it's only gonna get
>>>>>>>> worse, and eventually lead to JVM OOM and crash.
>>>>>>>>
>>>>>>>> By inspecting the heap dump created at OOM, we
>>>>>>>> can see that both of the MemtableFlushWriter
>>>>>>>> threads are stuck on line 1190
>>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>>>> in the ColumnFamilyStore.java:
>>>>>>>>
>>>>>>>> // mark writes older than the barrier as
>>>>>>>> blocking progress, permitting them to exceed
>>>>>>>> our memory limit
>>>>>>>> // if they are stuck waiting on it,
>>>>>>>> then wait for them all to complete
>>>>>>>> writeBarrier.markBlocking();
>>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>>>
>>>>>>>> And the MemtablePostFlush thread is stuck on
>>>>>>>> line 1094
>>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>>>> in the same file.
>>>>>>>>
>>>>>>>> try
>>>>>>>> {
>>>>>>>> // we wait on the latch for the
>>>>>>>> commitLogUpperBound to be set, and so that waiters
>>>>>>>> // on this task can rely on all prior flushes
>>>>>>>> being complete
>>>>>>>> latch.await();// <----------- stuck here
>>>>>>>> }
>>>>>>>> Our top suspect is CDC interacting with repair,
>>>>>>>> since this started to happen shortly after we
>>>>>>>> enabled CDC on the nodes, and each time repair
>>>>>>>> was running. But we have not been able to
>>>>>>>> reproduce this in a testing cluster, and don't
>>>>>>>> know what's the next step to troubleshoot this
>>>>>>>> issue. So I'm posting it in the mailing lists
>>>>>>>> and hoping someone may know something about it
>>>>>>>> or point me to the right direction.
>>>>>>>>
>>>>>>>
>>>>>>> Wouldn’t be completely surprised if CDC or
>>>>>>> repair somehow has a barrier, I’ve also seen
>>>>>>> similar behavior pre-3.0 with “very long running
>>>>>>> read commands” that have a barrier on the
>>>>>>> memtable that prevent release.
>>>>>>>
>>>>>>> You’ve got the heap (great, way better than most
>>>>>>> people debugging), are you able to navigate
>>>>>>> through it and look for references to that
>>>>>>> memtable or other things holding a barrier?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>>> --
>>>> Dmitry Konstantinov
>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Hi Bowen, thank you for the details.
Could you please also share a screenshot (Visual VM should be ok) for
"read-hotness-tracker:1"
thread with expanded local variable elements for the following lines:
1) at
org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324),
particularly com.codahale.metrics.Timer$Context#2086795 internals such as
startTime
2) at
org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186),
particularly WaitQueue$Standard$SignalWithListener#2086139 internals such
as thread, state
Also, could you please render all GC paths to root for
WaitQueue$Standard$SignalWithListener#2086139
object and share the correspondent tree view?
Regards,
Dmitry
On Tue, 12 Nov 2024 at 11:40, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Just tried Eclipse Memory Analyzer, it shows the exact same local
> variables as VisualVM, so there is no difference here.
>
> However, Eclipse Memory Analyzer is significantly slower (lagging) to
> operate via X window forwarding or VNC (possibly due to the GUI decorations
> and animations) and some features rely on a built-in web browser which I
> can't get it to work on a headless server. I'm doing the analysis on a
> headless server because my PC doesn't have enough RAM to load the large
> heap dump file.
>
> See screenshot below for a comparison between Eclipse Memory Analyzer and
> VisualVM on the variables referenced by the PERIODIC-COMMIT-LOG-SYNCER
> thread.
>
> Eclipse Memory Analyzer
>
> VisualVM
>
>
> On 11/11/2024 20:48, Dmitry Konstantinov wrote:
>
> >> If Eclipse Memory Analyzer also can't find the mysterious item, how do
> I proceed from there? What else can I do?
> We can analyze other variables from the heap dump, read code and repeat,
> until we find a clue :-). As of now I do not see a much better option. If
> the issue reproduces regularly - a custom build with extra logs can be done
> to add more diagnostic info printing.
> It is actually lucky that you have a heap dump, it can give a lot of info,
> unfortunately only in a moment of time (so we cannot see from the issue in
> dynamic).
>
> >> This has happened twice in the last 10 days, to two different nodes in
> the same cluster.
> Do you have heap dumps for the other failures?
> How much time did you have between a node startup and the failure?
>
> Can you share field values
> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
> using Eclipse Memory Analyzer
> from the following thread?:
> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#5
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
> local variable: java.util.concurrent.Semaphore$NonfairSync#1
> local variable:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
> at
> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>
> It should looks like this:
>
> [image: image.png]
>
> [image: image.png]
>
> Regards,
> Dmitry
>
>
> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Hi Dmitry,
>>
>> It indeed is very strange, either the tool I'm using is wrong, or there's
>> something causing the item to evaporate from the queue mysteriously or
>> never got into the queue.
>>
>> I've been using VisualVM to inspect the heap dump file and using OQL to
>> find and explore the objects in it.
>>
>> VisualVM does show some local variables, but not all of them. This isn't
>> an issue for types with little number of instances, because I can search
>> objects by type to find them. But it can be troublesome when an object is a
>> very common type.
>>
>> I will try Eclipse Memory Analyzer next week to see if it can find the
>> missing item, but I have my doubts.
>>
>> I've read the code of WaitQueue and the
>> AbstractCommitLogService.awaitSyncAt() method, and they both look correct
>> to me. If Eclipse Memory Analyzer also can't find the mysterious item, how
>> do I proceed from there? What else can I do?
>>
>> Cheers,
>> Bowen
>> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>>
>> Hi Bowen,
>>
>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>> like the code is inconsistent on whether lastSyncedAt is a relative
>> monotonic clock or absolute epoch time.
>> Yes, I think so, I am going to create a defect regarding it.
>>
>> >> I've also attempted to search for all RegisteredSignal objects in the
>> heap, and found 15 of them, but none of them is related to the
>> read-hotness-tracker:1 thread.
>> It is very strange because the stack trace for this thread waits within
>> the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
>> signal" local variable..
>> What tool do you use for the heap dump inspection? I use Eclipse Memory
>> Analyzer. It has "threads_overview" view and you can check the values for
>> local variables within a thread stack.
>> Like this:
>> [image: image.png]
>>
>> Regards,
>> Dmitry
>>
>>
>>
>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Hi Dmitry,
>>>
>>> Thank you for helping me.
>>>
>>> You are right, that's a monotonic clock. I was definitely wrongly
>>> interpreting its value. I got misled by this line of code
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>>> like the code is inconsistent on whether lastSyncedAt is a relative
>>> monotonic clock or absolute epoch time.
>>>
>>> Please see the attached cassandra.yaml with sensitive information masked
>>> out with asterisks.
>>>
>>> There's no ERROR level log in the system.log other than some guardrail
>>> violations (items_per_collection, if it matters) and random and rare
>>> network issues such as connection timeout in the few days run up to and
>>> during the event.
>>>
>>> We have the stdout and stderr captured in the systemd journal, and they
>>> have been quiet in during that period as well, not a single line got logged.
>>>
>>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>>> respectively.
>>>
>>> The CDC index files have been deleted by the CDC consumer, and we don't
>>> keep a copy of these files elsewhere, so I cannot check the offsets in them
>>> unfortunately.
>>>
>>> Regarding the WaitQueue, I found the queue to be empty in the heap dump,
>>> which means there's no RegisteredSignal objects in the queue at all. The
>>> queue is a ConcurrentLinkedQueue object, which has a head and a tail. The
>>> head has item=null and next=null, and the tail has item=null and next=head.
>>> I've also attempted to search for all RegisteredSignal objects in the heap,
>>> and found 15 of them, but none of them is related to the
>>> read-hotness-tracker:1 thread. They are for the following threads:
>>>
>>> - MemtablePostFlush:1
>>> - AntiEntropyStage:1
>>> - MemtableFlushWriter:910
>>> - COMMIT-LOG-ALLOCATOR
>>> - NativePoolCleaner
>>> - MemtableFlushWriter:911
>>> - Native-Transport-Requests-135
>>> - Native-Transport-Requests-139
>>> - Native-Transport-Requests-132
>>> - Native-Transport-Requests-143
>>> - Native-Transport-Requests-129
>>> - Native-Transport-Requests-140
>>> - Native-Transport-Requests-153
>>> - Native-Transport-Requests-141
>>> - Native-Transport-Requests-137
>>>
>>> This looks rather suspicious to me, as the read-hotness-tracker:1 thread
>>> is blocked and waiting for a non-existence registered signal.
>>>
>>> Cheers,
>>> Bowen
>>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>>
>>> Hi Bowen,
>>>
>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>> pollStarted;
>>> where long pollStarted = clock.now();
>>> the logic uses clock from: SyncRunnable sync = new
>>> SyncRunnable(preciseTime)
>>> preciseTime by default is SystemClock
>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>> System.nanoTime()
>>> so, the field is populated from System.nanoTime() which returns a
>>> relative timestamp, which cannot be treated as an absolute time like
>>> linux epoch (it is calculated like this:
>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>> )
>>>
>>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>>> other security sensitive things, of course)
>>> Do you have any exceptions in your system.log related to commit log?
>>> Do you have JVM stdout/stderr captured to a file? Does it have any
>>> exceptions? (which can be present there if Cassandra does not catch them
>>> properly by itself)
>>>
>>> ---
>>> How many CommitLogSegment objects do you have in your heap dump?
>>> What are values for the following fields of CommitLogSegment objects?
>>> lastSyncedOffset
>>> lastMarkerOffset
>>> cdcState
>>>
>>> Do you have CDC index files written by
>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>> If you have them - what are the last offsets written there?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> The syncComplete is an instance of WaitQueue, and the
>>>> syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly,
>>>> the queue is empty. There is no item in the queue's linked list, only the
>>>> head and tail nodes, each has item=null.
>>>>
>>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>>> correct to me:
>>>>
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ?
>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly();
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>>
>>>> This matches the example given in the WaitQueue's documentation (which
>>>> is definitely missing a pair of curly brackets and has bad indentation):
>>>>
>>>> while (!conditionMet())
>>>> Signal s = q.register();
>>>> if (!conditionMet()) // or, perhaps more correctly,
>>>> !conditionChanged()
>>>> s.await();
>>>> else
>>>> s.cancel();
>>>>
>>>> So, why is the queue empty while the thread is blocked on waiting for
>>>> the signal?
>>>>
>>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>>> about 5 years ago.
>>>>
>>>> Is this suppose to be a wall clock time in milliseconds since 1 January
>>>> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
>>>> context, and I'm just wrongly interpreting its value as a wall clock time?
>>>>
>>>>
>>>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>>>> object believes that there's nothing waiting for the signal, yet the
>>>> "read-hotness-tracker:1" thread is clearly waiting for it.
>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>
>>>> I think I'm getting really close now. This seems to have something to
>>>> do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>>
>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>>> <------ here
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>> at
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>> local variable:
>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>> <------ and here
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>> at
>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>> local variable: java.lang.String#42806
>>>> local variable: java.lang.String#47659
>>>> local variable:
>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>>> at
>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>> at
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>> local variable:
>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> local variable:
>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> at
>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>> at java.lang.Thread.run(Thread.java:829)
>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>
>>>>
>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>
>>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>>>> throws RequestExecutionException
>>>> {
>>>> OpOrder.Group group = null;
>>>> try
>>>> {
>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>
>>>> // write the mutation to the commitlog and memtables
>>>> CommitLogPosition position = null;
>>>> if (makeDurable)
>>>> {
>>>> position = addToCommitLog(mutation); // <--- and here
>>>> }
>>>> return new CassandraWriteContext(group, position);
>>>> }
>>>> catch (Throwable t)
>>>> {
>>>> if (group != null)
>>>> {
>>>> group.close();
>>>> }
>>>> throw t;
>>>> }
>>>> }
>>>>
>>>> The Keyspace.writeOrder.start() method call blocks the
>>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>>>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>>
>>>> void awaitSyncAt(long syncTime, Context context)
>>>> {
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ?
>>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly(); // <--- here
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>> }
>>>>
>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>
>>>> Now I know what is blocking the memtable flushing, but I haven't been
>>>> able to figure out is why it got stuck on waiting for that signal.
>>>>
>>>> I would appreciate it if anyone can offer some insight here.
>>>>
>>>>
>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>
>>>> I will give it a try and see what I can find. I plan to go down the
>>>> rabbit hole tomorrow. Will keep you updated.
>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>
>>>>
>>>>
>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>>
>>>> Writes on this node starts to timeout and fail. But if left untouched,
>>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>>
>>>> By inspecting the heap dump created at OOM, we can see that both of the
>>>> MemtableFlushWriter threads are stuck on line 1190
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>> in the ColumnFamilyStore.java:
>>>>
>>>> // mark writes older than the barrier as blocking progress,
>>>> permitting them to exceed our memory limit
>>>> // if they are stuck waiting on it, then wait for them all
>>>> to complete
>>>> writeBarrier.markBlocking();
>>>> writeBarrier.await(); // <----------- stuck here
>>>>
>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>> in the same file.
>>>>
>>>> try
>>>> {
>>>> // we wait on the latch for the commitLogUpperBound to
>>>> be set, and so that waiters
>>>> // on this task can rely on all prior flushes being
>>>> complete
>>>> latch.await(); // <----------- stuck here
>>>> }
>>>> Our top suspect is CDC interacting with repair, since this started to
>>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>>> running. But we have not been able to reproduce this in a testing cluster,
>>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>>> posting it in the mailing lists and hoping someone may know something about
>>>> it or point me to the right direction.
>>>>
>>>>
>>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>>> read commands” that have a barrier on the memtable that prevent release.
>>>>
>>>> You’ve got the heap (great, way better than most people debugging), are
>>>> you able to navigate through it and look for references to that memtable or
>>>> other things holding a barrier?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Just tried Eclipse Memory Analyzer, it shows the exact same local
variables as VisualVM, so there is no difference here.
However, Eclipse Memory Analyzer is significantly slower (lagging) to
operate via X window forwarding or VNC (possibly due to the GUI
decorations and animations) and some features rely on a built-in web
browser which I can't get it to work on a headless server. I'm doing the
analysis on a headless server because my PC doesn't have enough RAM to
load the large heap dump file.
See screenshot below for a comparison between Eclipse Memory Analyzer
and VisualVM on the variables referenced by the
PERIODIC-COMMIT-LOG-SYNCER thread.
Eclipse Memory Analyzer
VisualVM
On 11/11/2024 20:48, Dmitry Konstantinov wrote:
> >> If Eclipse Memory Analyzer also can't find the mysterious item, how
> do I proceed from there? What else can I do?
> We can analyze other variables from the heap dump, read code and
> repeat, until we find a clue :-). As of now I do not see a much better
> option. If the issue reproduces regularly - a custom build with extra
> logs can be done to add more diagnostic info printing.
> It is actually lucky that you have a heap dump, it can give a lot of
> info, unfortunately only in a moment of time (so we cannot see from
> the issue in dynamic).
>
> >> This has happened twice in the last 10 days, to two different nodes
> in the same cluster.
> Do you have heap dumps for the other failures?
> How much time did you have between a node startup and the failure?
>
> Can you share field values
> for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable objects
> using Eclipse Memory Analyzer
> from the following thread?:
> "PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#5
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
> local variable: java.util.concurrent.Semaphore$NonfairSync#1
> local variable:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
> at
> org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
>
> It should looks like this:
>
> image.png
>
> image.png
>
> Regards,
> Dmitry
>
>
> On Fri, 8 Nov 2024 at 19:01, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Hi Dmitry,
>
> It indeed is very strange, either the tool I'm using is wrong, or
> there's something causing the item to evaporate from the queue
> mysteriously or never got into the queue.
>
> I've been using VisualVM to inspect the heap dump file and using
> OQL to find and explore the objects in it.
>
> VisualVM does show some local variables, but not all of them. This
> isn't an issue for types with little number of instances, because
> I can search objects by type to find them. But it can be
> troublesome when an object is a very common type.
>
> I will try Eclipse Memory Analyzer next week to see if it can find
> the missing item, but I have my doubts.
>
> I've read the code of WaitQueue and the
> AbstractCommitLogService.awaitSyncAt() method, and they both look
> correct to me. If Eclipse Memory Analyzer also can't find the
> mysterious item, how do I proceed from there? What else can I do?
>
> Cheers,
> Bowen
>
> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>> Hi Bowen,
>>
>> >> protected volatile long lastSyncedAt = currentTimeMillis();".
>> It looks like the code is inconsistent on whether lastSyncedAt is
>> a relative monotonic clock or absolute epoch time.
>> Yes, I think so, I am going to create a defect regarding it.
>>
>> >> I've also attempted to search for all RegisteredSignal
>> objects in the heap, and found 15 of them, but none of them is
>> related to the read-hotness-tracker:1 thread.
>> It is very strange because the stack trace for this thread waits
>> within the object and read-hotness-tracker:1 thread has active
>> "WaitQueue.Signal signal" local variable..
>> What tool do you use for the heap dump inspection? I use Eclipse
>> Memory Analyzer. It has "threads_overview" view and you can check
>> the values for local variables within a thread stack.
>> Like this:
>> image.png
>>
>> Regards,
>> Dmitry
>>
>>
>>
>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Hi Dmitry,
>>
>> Thank you for helping me.
>>
>> You are right, that's a monotonic clock. I was definitely
>> wrongly interpreting its value. I got misled by this line of
>> code
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>> "protected volatile long lastSyncedAt =
>> currentTimeMillis();". It looks like the code is inconsistent
>> on whether lastSyncedAt is a relative monotonic clock or
>> absolute epoch time.
>>
>> Please see the attached cassandra.yaml with sensitive
>> information masked out with asterisks.
>>
>> There's no ERROR level log in the system.log other than some
>> guardrail violations (items_per_collection, if it matters)
>> and random and rare network issues such as connection timeout
>> in the few days run up to and during the event.
>>
>> We have the stdout and stderr captured in the systemd
>> journal, and they have been quiet in during that period as
>> well, not a single line got logged.
>>
>> There's 196 CommitLogSegment (more specifically,
>> MemoryMappedSegment) objects in the heap dump. Of these, 194
>> has cdcState = CONTAINS lastSyncedOffset = 33554432
>> lastMarkerOffset = 33554432. The other two has cdcState =
>> CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset =
>> 27007818 and cdcState = PERMITTED lastSyncedOffset = 20
>> lastMarkerOffset = 20 respectively.
>>
>> The CDC index files have been deleted by the CDC consumer,
>> and we don't keep a copy of these files elsewhere, so I
>> cannot check the offsets in them unfortunately.
>>
>> Regarding the WaitQueue, I found the queue to be empty in the
>> heap dump, which means there's no RegisteredSignal objects in
>> the queue at all. The queue is a ConcurrentLinkedQueue
>> object, which has a head and a tail. The head has item=null
>> and next=null, and the tail has item=null and next=head. I've
>> also attempted to search for all RegisteredSignal objects in
>> the heap, and found 15 of them, but none of them is related
>> to the read-hotness-tracker:1 thread. They are for the
>> following threads:
>>
>> * MemtablePostFlush:1
>> * AntiEntropyStage:1
>> * MemtableFlushWriter:910
>> * COMMIT-LOG-ALLOCATOR
>> * NativePoolCleaner
>> * MemtableFlushWriter:911
>> * Native-Transport-Requests-135
>> * Native-Transport-Requests-139
>> * Native-Transport-Requests-132
>> * Native-Transport-Requests-143
>> * Native-Transport-Requests-129
>> * Native-Transport-Requests-140
>> * Native-Transport-Requests-153
>> * Native-Transport-Requests-141
>> * Native-Transport-Requests-137
>>
>> This looks rather suspicious to me, as the
>> read-hotness-tracker:1 thread is blocked and waiting for a
>> non-existence registered signal.
>>
>> Cheers,
>> Bowen
>>
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>> Hi Bowen,
>>>
>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt
>>> = pollStarted;
>>> where long pollStarted = clock.now();
>>> the logic uses clock from: SyncRunnable sync = new
>>> SyncRunnable(preciseTime)
>>> preciseTime by default is SystemClock
>>> clock.now() ->
>>> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>> System.nanoTime()
>>> so, the field is populated from System.nanoTime() which
>>> returns a relative timestamp, which cannot be treated as an
>>> absolute time like linux epoch (it is calculated like this:
>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>>> )
>>>
>>> Can you share your cassandra.yaml file? (with removed IPs,
>>> paths and other security sensitive things, of course)
>>> Do you have any exceptions in your system.log related to
>>> commit log?
>>> Do you have JVM stdout/stderr captured to a file? Does it
>>> have any exceptions? (which can be present there if
>>> Cassandra does not catch them properly by itself)
>>>
>>> ---
>>> How many CommitLogSegment objects do you have in your heap dump?
>>> What are values for the following fields of CommitLogSegment
>>> objects?
>>> lastSyncedOffset
>>> lastMarkerOffset
>>> cdcState
>>>
>>> Do you have CDC index files written by
>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>>> ?
>>> If you have them - what are the last offsets written there?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> The syncComplete is an instance of WaitQueue, and the
>>> syncComplete.queue is an instance of
>>> ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>>> There is no item in the queue's linked list, only the
>>> head and tail nodes, each has item=null.
>>>
>>> The usage of the WaitQueue within the
>>> AbstractCommitLogService looks correct to me:
>>>
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) :
>>> syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly();
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>>
>>> This matches the example given in the WaitQueue's
>>> documentation (which is definitely missing a pair of
>>> curly brackets and has bad indentation):
>>>
>>> while (!conditionMet())
>>> Signal s = q.register();
>>> if (!conditionMet()) // or, perhaps more
>>> correctly, !conditionChanged()
>>> s.await();
>>> else
>>> s.cancel();
>>>
>>> So, why is the queue empty while the thread is blocked
>>> on waiting for the signal?
>>>
>>> Further digging showed that
>>> PeriodicCommitLogService.lastSyncedAt =
>>> 1573038452910796, which translates to 6 November 2019
>>> 11:07:32.910796, about 5 years ago.
>>>
>>> Is this suppose to be a wall clock time in milliseconds
>>> since 1 January 1970 00:00? Or is it a monotonic clock
>>> time that has no meaning out of the context, and I'm
>>> just wrongly interpreting its value as a wall clock time?
>>>
>>>
>>> I've found that the syncComplete.queue is empty, meaning
>>> the WaitQueue object believes that there's nothing
>>> waiting for the signal, yet the "read-hotness-tracker:1"
>>> thread is clearly waiting for it.
>>>
>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>
>>>> I think I'm getting really close now. This seems to
>>>> have something to do with the "read-hotness-tracker:1"
>>>> thread. The thread dump is:
>>>>
>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>>> <------ here
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> local variable:
>>>> com.codahale.metrics.Timer$Context#2086795
>>>> at
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.CommitLog#1
>>>> local variable:
>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>> <------ and here
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>> at
>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>> local variable:
>>>> org.apache.cassandra.db.Mutation#54491
>>>> at
>>>> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>> at
>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>> at
>>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>> at
>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>> local variable: java.lang.String#42806
>>>> local variable: java.lang.String#47659
>>>> local variable:
>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>> local variable:
>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>> at
>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>> at
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>> local variable:
>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> local variable:
>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>> at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>> local variable:
>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> local variable:
>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> at
>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>> at java.lang.Thread.run(Thread.java:829)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>
>>>>
>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>
>>>> public WriteContext beginWrite(Mutation mutation,
>>>> boolean makeDurable) throws RequestExecutionException
>>>> {
>>>> OpOrder.Group group = null;
>>>> try
>>>> {
>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>
>>>> // write the mutation to the commitlog and
>>>> memtables
>>>> CommitLogPosition position = null;
>>>> if (makeDurable)
>>>> {
>>>> position = addToCommitLog(mutation); //
>>>> <--- and here
>>>> }
>>>> return new CassandraWriteContext(group, position);
>>>> }
>>>> catch (Throwable t)
>>>> {
>>>> if (group != null)
>>>> {
>>>> group.close();
>>>> }
>>>> throw t;
>>>> }
>>>> }
>>>>
>>>> The Keyspace.writeOrder.start() method call blocks the
>>>> MemtableFlushWriter thread, and then it calls
>>>> addToCommitLog(mutation) which indirectly calls
>>>> AbstractCommitLogService.awaitSyncAt()
>>>>
>>>> void awaitSyncAt(long syncTime, Context context)
>>>> {
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ?
>>>> syncComplete.register(context, Context::stop) :
>>>> syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly(); // <--- here
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>> }
>>>>
>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>
>>>> Now I know what is blocking the memtable flushing, but
>>>> I haven't been able to figure out is why it got stuck
>>>> on waiting for that signal.
>>>>
>>>> I would appreciate it if anyone can offer some insight
>>>> here.
>>>>
>>>>
>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>
>>>>> I will give it a try and see what I can find. I plan
>>>>> to go down the rabbit hole tomorrow. Will keep you
>>>>> updated.
>>>>>
>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>
>>>>>>
>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>>> <us...@cassandra.apache.org>
>>>>>>> <ma...@cassandra.apache.org> wrote:
>>>>>>>
>>>>>>> Writes on this node starts to timeout and fail. But
>>>>>>> if left untouched, it's only gonna get worse, and
>>>>>>> eventually lead to JVM OOM and crash.
>>>>>>>
>>>>>>> By inspecting the heap dump created at OOM, we can
>>>>>>> see that both of the MemtableFlushWriter threads are
>>>>>>> stuck on line 1190
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>>> in the ColumnFamilyStore.java:
>>>>>>>
>>>>>>> // mark writes older than the barrier as blocking
>>>>>>> progress, permitting them to exceed our memory limit
>>>>>>> // if they are stuck waiting on it, then
>>>>>>> wait for them all to complete
>>>>>>> writeBarrier.markBlocking();
>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>>
>>>>>>> And the MemtablePostFlush thread is stuck on line
>>>>>>> 1094
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>>> in the same file.
>>>>>>>
>>>>>>> try
>>>>>>> {
>>>>>>> // we wait on the latch for the
>>>>>>> commitLogUpperBound to be set, and so that waiters
>>>>>>> // on this task can rely on all
>>>>>>> prior flushes being complete
>>>>>>> latch.await(); // <----------- stuck here
>>>>>>> }
>>>>>>> Our top suspect is CDC interacting with repair,
>>>>>>> since this started to happen shortly after we
>>>>>>> enabled CDC on the nodes, and each time repair was
>>>>>>> running. But we have not been able to reproduce this
>>>>>>> in a testing cluster, and don't know what's the next
>>>>>>> step to troubleshoot this issue. So I'm posting it
>>>>>>> in the mailing lists and hoping someone may know
>>>>>>> something about it or point me to the right direction.
>>>>>>>
>>>>>>
>>>>>> Wouldn’t be completely surprised if CDC or repair
>>>>>> somehow has a barrier, I’ve also seen similar
>>>>>> behavior pre-3.0 with “very long running read
>>>>>> commands” that have a barrier on the memtable that
>>>>>> prevent release.
>>>>>>
>>>>>> You’ve got the heap (great, way better than most
>>>>>> people debugging), are you able to navigate through
>>>>>> it and look for references to that memtable or other
>>>>>> things holding a barrier?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
>> If Eclipse Memory Analyzer also can't find the mysterious item, how do I
proceed from there? What else can I do?
We can analyze other variables from the heap dump, read code and repeat,
until we find a clue :-). As of now I do not see a much better option. If
the issue reproduces regularly - a custom build with extra logs can be done
to add more diagnostic info printing.
It is actually lucky that you have a heap dump, it can give a lot of info,
unfortunately only in a moment of time (so we cannot see from the issue in
dynamic).
>> This has happened twice in the last 10 days, to two different nodes in
the same cluster.
Do you have heap dumps for the other failures?
How much time did you have between a node startup and the failure?
Can you share field values
for org.apache.cassandra.db.commitlog.PeriodicCommitLogService and
org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable
objects
using Eclipse Memory Analyzer
from the following thread?:
"PERIODIC-COMMIT-LOG-SYNCER" prio=5 tid=42 TIMED_WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
local variable: io.netty.util.concurrent.FastThreadLocalThread#5
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
local variable: java.util.concurrent.Semaphore$NonfairSync#1
local variable:
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#157
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
at
org.apache.cassandra.utils.concurrent.Semaphore$Standard.tryAcquireUntil(Semaphore.java:142)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService$SyncRunnable.run(AbstractCommitLogService.java:216)
It should looks like this:
[image: image.png]
[image: image.png]
Regards,
Dmitry
On Fri, 8 Nov 2024 at 19:01, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Hi Dmitry,
>
> It indeed is very strange, either the tool I'm using is wrong, or there's
> something causing the item to evaporate from the queue mysteriously or
> never got into the queue.
>
> I've been using VisualVM to inspect the heap dump file and using OQL to
> find and explore the objects in it.
>
> VisualVM does show some local variables, but not all of them. This isn't
> an issue for types with little number of instances, because I can search
> objects by type to find them. But it can be troublesome when an object is a
> very common type.
>
> I will try Eclipse Memory Analyzer next week to see if it can find the
> missing item, but I have my doubts.
>
> I've read the code of WaitQueue and the
> AbstractCommitLogService.awaitSyncAt() method, and they both look correct
> to me. If Eclipse Memory Analyzer also can't find the mysterious item, how
> do I proceed from there? What else can I do?
>
> Cheers,
> Bowen
> On 08/11/2024 17:22, Dmitry Konstantinov wrote:
>
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();". It looks
> like the code is inconsistent on whether lastSyncedAt is a relative
> monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects in the
> heap, and found 15 of them, but none of them is related to the
> read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits within
> the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
> signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse Memory
> Analyzer. It has "threads_overview" view and you can check the values for
> local variables within a thread stack.
> Like this:
> [image: image.png]
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Hi Dmitry,
>>
>> Thank you for helping me.
>>
>> You are right, that's a monotonic clock. I was definitely wrongly
>> interpreting its value. I got misled by this line of code
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>> like the code is inconsistent on whether lastSyncedAt is a relative
>> monotonic clock or absolute epoch time.
>>
>> Please see the attached cassandra.yaml with sensitive information masked
>> out with asterisks.
>>
>> There's no ERROR level log in the system.log other than some guardrail
>> violations (items_per_collection, if it matters) and random and rare
>> network issues such as connection timeout in the few days run up to and
>> during the event.
>>
>> We have the stdout and stderr captured in the systemd journal, and they
>> have been quiet in during that period as well, not a single line got logged.
>>
>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>> respectively.
>>
>> The CDC index files have been deleted by the CDC consumer, and we don't
>> keep a copy of these files elsewhere, so I cannot check the offsets in them
>> unfortunately.
>>
>> Regarding the WaitQueue, I found the queue to be empty in the heap dump,
>> which means there's no RegisteredSignal objects in the queue at all. The
>> queue is a ConcurrentLinkedQueue object, which has a head and a tail. The
>> head has item=null and next=null, and the tail has item=null and next=head.
>> I've also attempted to search for all RegisteredSignal objects in the heap,
>> and found 15 of them, but none of them is related to the
>> read-hotness-tracker:1 thread. They are for the following threads:
>>
>> - MemtablePostFlush:1
>> - AntiEntropyStage:1
>> - MemtableFlushWriter:910
>> - COMMIT-LOG-ALLOCATOR
>> - NativePoolCleaner
>> - MemtableFlushWriter:911
>> - Native-Transport-Requests-135
>> - Native-Transport-Requests-139
>> - Native-Transport-Requests-132
>> - Native-Transport-Requests-143
>> - Native-Transport-Requests-129
>> - Native-Transport-Requests-140
>> - Native-Transport-Requests-153
>> - Native-Transport-Requests-141
>> - Native-Transport-Requests-137
>>
>> This looks rather suspicious to me, as the read-hotness-tracker:1 thread
>> is blocked and waiting for a non-existence registered signal.
>>
>> Cheers,
>> Bowen
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>
>> Hi Bowen,
>>
>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
>> where long pollStarted = clock.now();
>> the logic uses clock from: SyncRunnable sync = new
>> SyncRunnable(preciseTime)
>> preciseTime by default is SystemClock
>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>> System.nanoTime()
>> so, the field is populated from System.nanoTime() which returns a
>> relative timestamp, which cannot be treated as an absolute time like
>> linux epoch (it is calculated like this:
>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>> )
>>
>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>> other security sensitive things, of course)
>> Do you have any exceptions in your system.log related to commit log?
>> Do you have JVM stdout/stderr captured to a file? Does it have any
>> exceptions? (which can be present there if Cassandra does not catch them
>> properly by itself)
>>
>> ---
>> How many CommitLogSegment objects do you have in your heap dump?
>> What are values for the following fields of CommitLogSegment objects?
>> lastSyncedOffset
>> lastMarkerOffset
>> cdcState
>>
>> Do you have CDC index files written by
>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>> If you have them - what are the last offsets written there?
>>
>> Regards,
>> Dmitry
>>
>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue
>>> is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>>> There is no item in the queue's linked list, only the head and tail nodes,
>>> each has item=null.
>>>
>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>> correct to me:
>>>
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly();
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>>
>>> This matches the example given in the WaitQueue's documentation (which
>>> is definitely missing a pair of curly brackets and has bad indentation):
>>>
>>> while (!conditionMet())
>>> Signal s = q.register();
>>> if (!conditionMet()) // or, perhaps more correctly,
>>> !conditionChanged()
>>> s.await();
>>> else
>>> s.cancel();
>>>
>>> So, why is the queue empty while the thread is blocked on waiting for
>>> the signal?
>>>
>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>> about 5 years ago.
>>>
>>> Is this suppose to be a wall clock time in milliseconds since 1 January
>>> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
>>> context, and I'm just wrongly interpreting its value as a wall clock time?
>>>
>>>
>>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>>> object believes that there's nothing waiting for the signal, yet the
>>> "read-hotness-tracker:1" thread is clearly waiting for it.
>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>
>>> I think I'm getting really close now. This seems to have something to do
>>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>>
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>> <------ here
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>> at
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable:
>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>> <------ and here
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable: org.apache.cassandra.db.Mutation#54491
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at
>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable:
>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at
>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable:
>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable:
>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable:
>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>
>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>>> throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); // <--- and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>>
>>> The Keyspace.writeOrder.start() method call blocks the
>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>>
>>> This then got stuck on the signal.awaitUninterruptibly()
>>>
>>> Now I know what is blocking the memtable flushing, but I haven't been
>>> able to figure out is why it got stuck on waiting for that signal.
>>>
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>
>>> I will give it a try and see what I can find. I plan to go down the
>>> rabbit hole tomorrow. Will keep you updated.
>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>
>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>
>>> Writes on this node starts to timeout and fail. But if left untouched,
>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can see that both of the
>>> MemtableFlushWriter threads are stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as blocking progress,
>>> permitting them to exceed our memory limit
>>> // if they are stuck waiting on it, then wait for them all
>>> to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <----------- stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the commitLogUpperBound to
>>> be set, and so that waiters
>>> // on this task can rely on all prior flushes being
>>> complete
>>> latch.await(); // <----------- stuck here
>>> }
>>> Our top suspect is CDC interacting with repair, since this started to
>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>> running. But we have not been able to reproduce this in a testing cluster,
>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>> posting it in the mailing lists and hoping someone may know something about
>>> it or point me to the right direction.
>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>> read commands” that have a barrier on the memtable that prevent release.
>>>
>>> You’ve got the heap (great, way better than most people debugging), are
>>> you able to navigate through it and look for references to that memtable or
>>> other things holding a barrier?
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Dmitry,
It indeed is very strange, either the tool I'm using is wrong, or
there's something causing the item to evaporate from the queue
mysteriously or never got into the queue.
I've been using VisualVM to inspect the heap dump file and using OQL to
find and explore the objects in it.
VisualVM does show some local variables, but not all of them. This isn't
an issue for types with little number of instances, because I can search
objects by type to find them. But it can be troublesome when an object
is a very common type.
I will try Eclipse Memory Analyzer next week to see if it can find the
missing item, but I have my doubts.
I've read the code of WaitQueue and the
AbstractCommitLogService.awaitSyncAt() method, and they both look
correct to me. If Eclipse Memory Analyzer also can't find the mysterious
item, how do I proceed from there? What else can I do?
Cheers,
Bowen
On 08/11/2024 17:22, Dmitry Konstantinov wrote:
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
> looks like the code is inconsistent on whether lastSyncedAt is a
> relative monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects in
> the heap, and found 15 of them, but none of them is related to the
> read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits
> within the object and read-hotness-tracker:1 thread has active
> "WaitQueue.Signal signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse
> Memory Analyzer. It has "threads_overview" view and you can check the
> values for local variables within a thread stack.
> Like this:
> image.png
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Hi Dmitry,
>
> Thank you for helping me.
>
> You are right, that's a monotonic clock. I was definitely wrongly
> interpreting its value. I got misled by this line of code
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
> "protected volatile long lastSyncedAt = currentTimeMillis();". It
> looks like the code is inconsistent on whether lastSyncedAt is a
> relative monotonic clock or absolute epoch time.
>
> Please see the attached cassandra.yaml with sensitive information
> masked out with asterisks.
>
> There's no ERROR level log in the system.log other than some
> guardrail violations (items_per_collection, if it matters) and
> random and rare network issues such as connection timeout in the
> few days run up to and during the event.
>
> We have the stdout and stderr captured in the systemd journal, and
> they have been quiet in during that period as well, not a single
> line got logged.
>
> There's 196 CommitLogSegment (more specifically,
> MemoryMappedSegment) objects in the heap dump. Of these, 194 has
> cdcState = CONTAINS lastSyncedOffset = 33554432 lastMarkerOffset =
> 33554432. The other two has cdcState = CONTAINS lastSyncedOffset =
> 27007818 lastMarkerOffset = 27007818 and cdcState = PERMITTED
> lastSyncedOffset = 20 lastMarkerOffset = 20 respectively.
>
> The CDC index files have been deleted by the CDC consumer, and we
> don't keep a copy of these files elsewhere, so I cannot check the
> offsets in them unfortunately.
>
> Regarding the WaitQueue, I found the queue to be empty in the heap
> dump, which means there's no RegisteredSignal objects in the queue
> at all. The queue is a ConcurrentLinkedQueue object, which has a
> head and a tail. The head has item=null and next=null, and the
> tail has item=null and next=head. I've also attempted to search
> for all RegisteredSignal objects in the heap, and found 15 of
> them, but none of them is related to the read-hotness-tracker:1
> thread. They are for the following threads:
>
> * MemtablePostFlush:1
> * AntiEntropyStage:1
> * MemtableFlushWriter:910
> * COMMIT-LOG-ALLOCATOR
> * NativePoolCleaner
> * MemtableFlushWriter:911
> * Native-Transport-Requests-135
> * Native-Transport-Requests-139
> * Native-Transport-Requests-132
> * Native-Transport-Requests-143
> * Native-Transport-Requests-129
> * Native-Transport-Requests-140
> * Native-Transport-Requests-153
> * Native-Transport-Requests-141
> * Native-Transport-Requests-137
>
> This looks rather suspicious to me, as the read-hotness-tracker:1
> thread is blocked and waiting for a non-existence registered signal.
>
> Cheers,
> Bowen
>
> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>> Hi Bowen,
>>
>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>> pollStarted;
>> where long pollStarted = clock.now();
>> the logic uses clock from: SyncRunnable sync = new
>> SyncRunnable(preciseTime)
>> preciseTime by default is SystemClock
>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime
>> -> System.nanoTime()
>> so, the field is populated from System.nanoTime() which returns a
>> relative timestamp, which cannot be treated as an absolute time
>> like linux epoch (it is calculated like this:
>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>> )
>>
>> Can you share your cassandra.yaml file? (with removed IPs, paths
>> and other security sensitive things, of course)
>> Do you have any exceptions in your system.log related to commit log?
>> Do you have JVM stdout/stderr captured to a file? Does it have
>> any exceptions? (which can be present there if Cassandra does not
>> catch them properly by itself)
>>
>> ---
>> How many CommitLogSegment objects do you have in your heap dump?
>> What are values for the following fields of CommitLogSegment objects?
>> lastSyncedOffset
>> lastMarkerOffset
>> cdcState
>>
>> Do you have CDC index files written by
>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>> ?
>> If you have them - what are the last offsets written there?
>>
>> Regards,
>> Dmitry
>>
>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> The syncComplete is an instance of WaitQueue, and the
>> syncComplete.queue is an instance of ConcurrentLinkedQueue.
>> Surprisingly, the queue is empty. There is no item in the
>> queue's linked list, only the head and tail nodes, each has
>> item=null.
>>
>> The usage of the WaitQueue within the
>> AbstractCommitLogService looks correct to me:
>>
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) :
>> syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly();
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>>
>> This matches the example given in the WaitQueue's
>> documentation (which is definitely missing a pair of curly
>> brackets and has bad indentation):
>>
>> while (!conditionMet())
>> Signal s = q.register();
>> if (!conditionMet()) // or, perhaps more
>> correctly, !conditionChanged()
>> s.await();
>> else
>> s.cancel();
>>
>> So, why is the queue empty while the thread is blocked on
>> waiting for the signal?
>>
>> Further digging showed that
>> PeriodicCommitLogService.lastSyncedAt = 1573038452910796,
>> which translates to 6 November 2019 11:07:32.910796, about 5
>> years ago.
>>
>> Is this suppose to be a wall clock time in milliseconds since
>> 1 January 1970 00:00? Or is it a monotonic clock time that
>> has no meaning out of the context, and I'm just wrongly
>> interpreting its value as a wall clock time?
>>
>>
>> I've found that the syncComplete.queue is empty, meaning the
>> WaitQueue object believes that there's nothing waiting for
>> the signal, yet the "read-hotness-tracker:1" thread is
>> clearly waiting for it.
>>
>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>
>>> I think I'm getting really close now. This seems to have
>>> something to do with the "read-hotness-tracker:1" thread.
>>> The thread dump is:
>>>
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>> <------ here
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable:
>>> com.codahale.metrics.Timer$Context#2086795
>>> at
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable:
>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>> <------ and here
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at
>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable: org.apache.cassandra.db.Mutation#54491
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at
>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable:
>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable:
>>> org.apache.cassandra.metrics.RestorableMeter#752
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at
>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable:
>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable:
>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable:
>>> java.util.concurrent.Executors$RunnableAdapter#934
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable:
>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable:
>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable:
>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>
>>> public WriteContext beginWrite(Mutation mutation, boolean
>>> makeDurable) throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); // <---
>>> and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>>
>>> The Keyspace.writeOrder.start() method call blocks the
>>> MemtableFlushWriter thread, and then it calls
>>> addToCommitLog(mutation) which indirectly calls
>>> AbstractCommitLogService.awaitSyncAt()
>>>
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) :
>>> syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>>
>>> This then got stuck on the signal.awaitUninterruptibly()
>>>
>>> Now I know what is blocking the memtable flushing, but I
>>> haven't been able to figure out is why it got stuck on
>>> waiting for that signal.
>>>
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>
>>>> I will give it a try and see what I can find. I plan to go
>>>> down the rabbit hole tomorrow. Will keep you updated.
>>>>
>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>
>>>>>
>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>> <us...@cassandra.apache.org>
>>>>>> <ma...@cassandra.apache.org> wrote:
>>>>>>
>>>>>> Writes on this node starts to timeout and fail. But if
>>>>>> left untouched, it's only gonna get worse, and eventually
>>>>>> lead to JVM OOM and crash.
>>>>>>
>>>>>> By inspecting the heap dump created at OOM, we can see
>>>>>> that both of the MemtableFlushWriter threads are stuck on
>>>>>> line 1190
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>> in the ColumnFamilyStore.java:
>>>>>>
>>>>>> // mark writes older than the barrier as blocking
>>>>>> progress, permitting them to exceed our memory limit
>>>>>> // if they are stuck waiting on it, then wait
>>>>>> for them all to complete
>>>>>> writeBarrier.markBlocking();
>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>
>>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>> in the same file.
>>>>>>
>>>>>> try
>>>>>> {
>>>>>> // we wait on the latch for the
>>>>>> commitLogUpperBound to be set, and so that waiters
>>>>>> // on this task can rely on all prior
>>>>>> flushes being complete
>>>>>> latch.await(); // <----------- stuck here
>>>>>> }
>>>>>> Our top suspect is CDC interacting with repair, since
>>>>>> this started to happen shortly after we enabled CDC on
>>>>>> the nodes, and each time repair was running. But we have
>>>>>> not been able to reproduce this in a testing cluster, and
>>>>>> don't know what's the next step to troubleshoot this
>>>>>> issue. So I'm posting it in the mailing lists and hoping
>>>>>> someone may know something about it or point me to the
>>>>>> right direction.
>>>>>>
>>>>>
>>>>> Wouldn’t be completely surprised if CDC or repair
>>>>> somehow has a barrier, I’ve also seen similar behavior
>>>>> pre-3.0 with “very long running read commands” that have a
>>>>> barrier on the memtable that prevent release.
>>>>>
>>>>> You’ve got the heap (great, way better than most people
>>>>> debugging), are you able to navigate through it and look
>>>>> for references to that memtable or other things holding a
>>>>> barrier?
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
It's interesting how they organised the documentation.
So it is guaranteed that the ConcurrentLinkedQueue can be modified and
won't break the iterator.
But I don't see anything mentioning the reverse. Can an iterator
removing items from the middle of a queue (which by definition is FIFO)
break the queue?
On 13/11/2024 20:37, Sebastian Marsching wrote:
> Hi Bowen,
>>
>> I've been reading the source code and the Java documentation.
>>
>> In the code
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java#L244>:
>>
>> public void signalAll()
>> {
>> // ...
>> Iterator<RegisteredSignal> iter = queue.iterator();
>> while (iter.hasNext())
>> {
>> RegisteredSignal signal = iter.next();
>> // ...
>> iter.remove(); // <--- here
>> }
>> }
>>
>> and in the documentation
>> <https://docs.oracle.com/javase/8/docs/api///java/util/Iterator.html#remove-->:
>>
>> /> The behavior of an iterator is unspecified if the underlying
>> collection is modified while the iteration is in progress in any way
>> other than by calling this method./
>>
> The documentation for the Iterator interface is misleading because
> most implementations of this interface are not thread-safe, but the
> implementation provided by the ConcurrentLinkedQueue is different.
>
> The documentation for the ConcurrentLinkedQueue.iterator() method says:
>
> public Iterator
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Iterator.html><E
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/concurrent/ConcurrentLinkedQueue.html>> iterator()
> Returns an iterator over the elements in this queue in proper
> sequence. The elements will be returned in order from first (head) to
> last (tail).
>
> The returned iterator is /weakly consistent/
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/concurrent/package-summary.html#Weakly>.
>
> And when following the link for “weakly consistent”, we find the
> following definition:
>
> Most concurrent Collection implementations (including most Queues)
> also differ from the usual |java.util|conventions in that
> theirIterators
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Iterator.html>andSpliterators
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Spliterator.html>provide/weakly
> consistent/rather than fast-fail traversal:
>
> * they may proceed concurrently with other operations
> * they will never throw |ConcurrentModificationException|
> <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/ConcurrentModificationException.html>
> * they are guaranteed to traverse elements as they existed upon
> construction exactly once, and may (but are not guaranteed to)
> reflect any modifications subsequent to construction.
>
> So, concurrent modifications to a ConcurrentLinkedQueue should not
> result in undefined behavior of an iterator returned by such a queue.
>
> -Sebastian
>
Re: Unexplained stuck memtable flush
Posted by Sebastian Marsching <se...@marsching.com>.
Hi Bowen,
> I've been reading the source code and the Java documentation.
>
> In the code <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java#L244>:
>
> public void signalAll()
> {
> // ...
> Iterator<RegisteredSignal> iter = queue.iterator();
> while (iter.hasNext())
> {
> RegisteredSignal signal = iter.next();
> // ...
> iter.remove(); // <--- here
> }
> }
>
> and in the documentation <https://docs.oracle.com/javase/8/docs/api///java/util/Iterator.html#remove-->:
>
> > The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.
The documentation for the Iterator interface is misleading because most implementations of this interface are not thread-safe, but the implementation provided by the ConcurrentLinkedQueue is different.
The documentation for the ConcurrentLinkedQueue.iterator() method says:
public Iterator <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Iterator.html><E <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/concurrent/ConcurrentLinkedQueue.html>> iterator()
Returns an iterator over the elements in this queue in proper sequence. The elements will be returned in order from first (head) to last (tail).
The returned iterator is weakly consistent <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/concurrent/package-summary.html#Weakly>.
And when following the link for “weakly consistent”, we find the following definition:
Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Iterator.html> and Spliterators <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/Spliterator.html> provide weakly consistent rather than fast-fail traversal:
they may proceed concurrently with other operations
they will never throw ConcurrentModificationException <dfile:///Users/termi/Library/Application%20Support/Dash/Versioned%20DocSets/Java%20-%20DHDocsetDownloader/SE11/Java.docset/Contents/Resources/Documents/java.base/java/util/ConcurrentModificationException.html>
they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
So, concurrent modifications to a ConcurrentLinkedQueue should not result in undefined behavior of an iterator returned by such a queue.
-Sebastian
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Thank you for taking care of that.
I've been reading the source code and the Java documentation.
In the code
<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java#L244>:
public void signalAll()
{
// ...
Iterator<RegisteredSignal> iter = queue.iterator();
while (iter.hasNext())
{
RegisteredSignal signal = iter.next();
// ...
iter.remove(); // <--- here
}
}
and in the documentation
<https://docs.oracle.com/javase/8/docs/api///java/util/Iterator.html#remove-->:
/> The behavior of an iterator is unspecified if the underlying
collection is modified while the iteration is in progress in any way
other than by calling this method./
This looks suspicious to me.
Does anyone know whether it is safe to call iter.remove() in
ConcurrentLinkedQueue while the queue is being modified by other threads?
On 12/11/2024 13:19, Dmitry Konstantinov wrote:
> >>>> protected volatile long lastSyncedAt = currentTimeMillis();". It looks like the code is inconsistent on
> whether lastSyncedAt is a relative monotonic clock or absolute epoch time.
> >> Yes, I think so, I am going to create a defect regarding it.
>
> Created: https://issues.apache.org/jira/browse/CASSANDRA-20074
> <https://issues.apache.org/jira/browse/CASSANDRA-20074>
>
> On Fri, 8 Nov 2024 at 17:22, Dmitry Konstantinov <ne...@gmail.com>
> wrote:
>
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();".
> It looks like the code is inconsistent on whether lastSyncedAt is
> a relative monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects
> in the heap, and found 15 of them, but none of them is related to
> the read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits
> within the object and read-hotness-tracker:1 thread has active
> "WaitQueue.Signal signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse
> Memory Analyzer. It has "threads_overview" view and you can check
> the values for local variables within a thread stack.
> Like this:
> image.png
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Hi Dmitry,
>
> Thank you for helping me.
>
> You are right, that's a monotonic clock. I was definitely
> wrongly interpreting its value. I got misled by this line of
> code
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
> "protected volatile long lastSyncedAt = currentTimeMillis();".
> It looks like the code is inconsistent on whether lastSyncedAt
> is a relative monotonic clock or absolute epoch time.
>
> Please see the attached cassandra.yaml with sensitive
> information masked out with asterisks.
>
> There's no ERROR level log in the system.log other than some
> guardrail violations (items_per_collection, if it matters) and
> random and rare network issues such as connection timeout in
> the few days run up to and during the event.
>
> We have the stdout and stderr captured in the systemd journal,
> and they have been quiet in during that period as well, not a
> single line got logged.
>
> There's 196 CommitLogSegment (more specifically,
> MemoryMappedSegment) objects in the heap dump. Of these, 194
> has cdcState = CONTAINS lastSyncedOffset = 33554432
> lastMarkerOffset = 33554432. The other two has cdcState =
> CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset =
> 27007818 and cdcState = PERMITTED lastSyncedOffset = 20
> lastMarkerOffset = 20 respectively.
>
> The CDC index files have been deleted by the CDC consumer, and
> we don't keep a copy of these files elsewhere, so I cannot
> check the offsets in them unfortunately.
>
> Regarding the WaitQueue, I found the queue to be empty in the
> heap dump, which means there's no RegisteredSignal objects in
> the queue at all. The queue is a ConcurrentLinkedQueue object,
> which has a head and a tail. The head has item=null and
> next=null, and the tail has item=null and next=head. I've also
> attempted to search for all RegisteredSignal objects in the
> heap, and found 15 of them, but none of them is related to the
> read-hotness-tracker:1 thread. They are for the following threads:
>
> * MemtablePostFlush:1
> * AntiEntropyStage:1
> * MemtableFlushWriter:910
> * COMMIT-LOG-ALLOCATOR
> * NativePoolCleaner
> * MemtableFlushWriter:911
> * Native-Transport-Requests-135
> * Native-Transport-Requests-139
> * Native-Transport-Requests-132
> * Native-Transport-Requests-143
> * Native-Transport-Requests-129
> * Native-Transport-Requests-140
> * Native-Transport-Requests-153
> * Native-Transport-Requests-141
> * Native-Transport-Requests-137
>
> This looks rather suspicious to me, as the
> read-hotness-tracker:1 thread is blocked and waiting for a
> non-existence registered signal.
>
> Cheers,
> Bowen
>
> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>> Hi Bowen,
>>
>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>> pollStarted;
>> where long pollStarted = clock.now();
>> the logic uses clock from: SyncRunnable sync = new
>> SyncRunnable(preciseTime)
>> preciseTime by default is SystemClock
>> clock.now() ->
>> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>> System.nanoTime()
>> so, the field is populated from System.nanoTime() which
>> returns a relative timestamp, which cannot be treated as an
>> absolute time like linux epoch (it is calculated like this:
>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>> )
>>
>> Can you share your cassandra.yaml file? (with removed IPs,
>> paths and other security sensitive things, of course)
>> Do you have any exceptions in your system.log related to
>> commit log?
>> Do you have JVM stdout/stderr captured to a file? Does it
>> have any exceptions? (which can be present there if Cassandra
>> does not catch them properly by itself)
>>
>> ---
>> How many CommitLogSegment objects do you have in your heap dump?
>> What are values for the following fields of CommitLogSegment
>> objects?
>> lastSyncedOffset
>> lastMarkerOffset
>> cdcState
>>
>> Do you have CDC index files written by
>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>> ?
>> If you have them - what are the last offsets written there?
>>
>> Regards,
>> Dmitry
>>
>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> The syncComplete is an instance of WaitQueue, and the
>> syncComplete.queue is an instance of
>> ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>> There is no item in the queue's linked list, only the
>> head and tail nodes, each has item=null.
>>
>> The usage of the WaitQueue within the
>> AbstractCommitLogService looks correct to me:
>>
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) :
>> syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly();
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>>
>> This matches the example given in the WaitQueue's
>> documentation (which is definitely missing a pair of
>> curly brackets and has bad indentation):
>>
>> while (!conditionMet())
>> Signal s = q.register();
>> if (!conditionMet()) // or, perhaps more
>> correctly, !conditionChanged()
>> s.await();
>> else
>> s.cancel();
>>
>> So, why is the queue empty while the thread is blocked on
>> waiting for the signal?
>>
>> Further digging showed that
>> PeriodicCommitLogService.lastSyncedAt = 1573038452910796,
>> which translates to 6 November 2019 11:07:32.910796,
>> about 5 years ago.
>>
>> Is this suppose to be a wall clock time in milliseconds
>> since 1 January 1970 00:00? Or is it a monotonic clock
>> time that has no meaning out of the context, and I'm just
>> wrongly interpreting its value as a wall clock time?
>>
>>
>> I've found that the syncComplete.queue is empty, meaning
>> the WaitQueue object believes that there's nothing
>> waiting for the signal, yet the "read-hotness-tracker:1"
>> thread is clearly waiting for it.
>>
>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>
>>> I think I'm getting really close now. This seems to have
>>> something to do with the "read-hotness-tracker:1"
>>> thread. The thread dump is:
>>>
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>> <------ here
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable:
>>> com.codahale.metrics.Timer$Context#2086795
>>> at
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable:
>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>> <------ and here
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at
>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable:
>>> org.apache.cassandra.db.Mutation#54491
>>> at
>>> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at
>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at
>>> org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at
>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable:
>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable:
>>> org.apache.cassandra.metrics.RestorableMeter#752
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at
>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable:
>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable:
>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable:
>>> java.util.concurrent.Executors$RunnableAdapter#934
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable:
>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable:
>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable:
>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable:
>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>
>>> public WriteContext beginWrite(Mutation mutation,
>>> boolean makeDurable) throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); //
>>> <--- and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>>
>>> The Keyspace.writeOrder.start() method call blocks the
>>> MemtableFlushWriter thread, and then it calls
>>> addToCommitLog(mutation) which indirectly calls
>>> AbstractCommitLogService.awaitSyncAt()
>>>
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) :
>>> syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>>
>>> This then got stuck on the signal.awaitUninterruptibly()
>>>
>>> Now I know what is blocking the memtable flushing, but I
>>> haven't been able to figure out is why it got stuck on
>>> waiting for that signal.
>>>
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>
>>>> I will give it a try and see what I can find. I plan to
>>>> go down the rabbit hole tomorrow. Will keep you updated.
>>>>
>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>
>>>>>
>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>> <us...@cassandra.apache.org>
>>>>>> <ma...@cassandra.apache.org> wrote:
>>>>>>
>>>>>> Writes on this node starts to timeout and fail. But
>>>>>> if left untouched, it's only gonna get worse, and
>>>>>> eventually lead to JVM OOM and crash.
>>>>>>
>>>>>> By inspecting the heap dump created at OOM, we can
>>>>>> see that both of the MemtableFlushWriter threads are
>>>>>> stuck on line 1190
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>>> in the ColumnFamilyStore.java:
>>>>>>
>>>>>> // mark writes older than the barrier as blocking
>>>>>> progress, permitting them to exceed our memory limit
>>>>>> // if they are stuck waiting on it, then
>>>>>> wait for them all to complete
>>>>>> writeBarrier.markBlocking();
>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>
>>>>>> And the MemtablePostFlush thread is stuck on line
>>>>>> 1094
>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>>> in the same file.
>>>>>>
>>>>>> try
>>>>>> {
>>>>>> // we wait on the latch for the
>>>>>> commitLogUpperBound to be set, and so that waiters
>>>>>> // on this task can rely on all prior
>>>>>> flushes being complete
>>>>>> latch.await(); // <----------- stuck here
>>>>>> }
>>>>>> Our top suspect is CDC interacting with repair, since
>>>>>> this started to happen shortly after we enabled CDC
>>>>>> on the nodes, and each time repair was running. But
>>>>>> we have not been able to reproduce this in a testing
>>>>>> cluster, and don't know what's the next step to
>>>>>> troubleshoot this issue. So I'm posting it in the
>>>>>> mailing lists and hoping someone may know something
>>>>>> about it or point me to the right direction.
>>>>>>
>>>>>
>>>>> Wouldn’t be completely surprised if CDC or repair
>>>>> somehow has a barrier, I’ve also seen similar behavior
>>>>> pre-3.0 with “very long running read commands” that
>>>>> have a barrier on the memtable that prevent release.
>>>>>
>>>>> You’ve got the heap (great, way better than most
>>>>> people debugging), are you able to navigate through it
>>>>> and look for references to that memtable or other
>>>>> things holding a barrier?
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
>>>> protected volatile long lastSyncedAt = currentTimeMillis();". It looks
like the code is inconsistent on whether lastSyncedAt is a relative
monotonic clock or absolute epoch time.
>> Yes, I think so, I am going to create a defect regarding it.
Created: https://issues.apache.org/jira/browse/CASSANDRA-20074
<https://issues.apache.org/jira/browse/CASSANDRA-20074>
On Fri, 8 Nov 2024 at 17:22, Dmitry Konstantinov <ne...@gmail.com> wrote:
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();". It looks
> like the code is inconsistent on whether lastSyncedAt is a relative
> monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects in the
> heap, and found 15 of them, but none of them is related to the
> read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits within
> the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
> signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse Memory
> Analyzer. It has "threads_overview" view and you can check the values for
> local variables within a thread stack.
> Like this:
> [image: image.png]
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Hi Dmitry,
>>
>> Thank you for helping me.
>>
>> You are right, that's a monotonic clock. I was definitely wrongly
>> interpreting its value. I got misled by this line of code
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
>> like the code is inconsistent on whether lastSyncedAt is a relative
>> monotonic clock or absolute epoch time.
>>
>> Please see the attached cassandra.yaml with sensitive information masked
>> out with asterisks.
>>
>> There's no ERROR level log in the system.log other than some guardrail
>> violations (items_per_collection, if it matters) and random and rare
>> network issues such as connection timeout in the few days run up to and
>> during the event.
>>
>> We have the stdout and stderr captured in the systemd journal, and they
>> have been quiet in during that period as well, not a single line got logged.
>>
>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
>> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
>> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>> respectively.
>>
>> The CDC index files have been deleted by the CDC consumer, and we don't
>> keep a copy of these files elsewhere, so I cannot check the offsets in them
>> unfortunately.
>>
>> Regarding the WaitQueue, I found the queue to be empty in the heap dump,
>> which means there's no RegisteredSignal objects in the queue at all. The
>> queue is a ConcurrentLinkedQueue object, which has a head and a tail. The
>> head has item=null and next=null, and the tail has item=null and next=head.
>> I've also attempted to search for all RegisteredSignal objects in the heap,
>> and found 15 of them, but none of them is related to the
>> read-hotness-tracker:1 thread. They are for the following threads:
>>
>> - MemtablePostFlush:1
>> - AntiEntropyStage:1
>> - MemtableFlushWriter:910
>> - COMMIT-LOG-ALLOCATOR
>> - NativePoolCleaner
>> - MemtableFlushWriter:911
>> - Native-Transport-Requests-135
>> - Native-Transport-Requests-139
>> - Native-Transport-Requests-132
>> - Native-Transport-Requests-143
>> - Native-Transport-Requests-129
>> - Native-Transport-Requests-140
>> - Native-Transport-Requests-153
>> - Native-Transport-Requests-141
>> - Native-Transport-Requests-137
>>
>> This looks rather suspicious to me, as the read-hotness-tracker:1 thread
>> is blocked and waiting for a non-existence registered signal.
>>
>> Cheers,
>> Bowen
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>
>> Hi Bowen,
>>
>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
>> where long pollStarted = clock.now();
>> the logic uses clock from: SyncRunnable sync = new
>> SyncRunnable(preciseTime)
>> preciseTime by default is SystemClock
>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>> System.nanoTime()
>> so, the field is populated from System.nanoTime() which returns a
>> relative timestamp, which cannot be treated as an absolute time like
>> linux epoch (it is calculated like this:
>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
>> )
>>
>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>> other security sensitive things, of course)
>> Do you have any exceptions in your system.log related to commit log?
>> Do you have JVM stdout/stderr captured to a file? Does it have any
>> exceptions? (which can be present there if Cassandra does not catch them
>> properly by itself)
>>
>> ---
>> How many CommitLogSegment objects do you have in your heap dump?
>> What are values for the following fields of CommitLogSegment objects?
>> lastSyncedOffset
>> lastMarkerOffset
>> cdcState
>>
>> Do you have CDC index files written by
>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>> If you have them - what are the last offsets written there?
>>
>> Regards,
>> Dmitry
>>
>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue
>>> is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>>> There is no item in the queue's linked list, only the head and tail nodes,
>>> each has item=null.
>>>
>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>> correct to me:
>>>
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly();
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>>
>>> This matches the example given in the WaitQueue's documentation (which
>>> is definitely missing a pair of curly brackets and has bad indentation):
>>>
>>> while (!conditionMet())
>>> Signal s = q.register();
>>> if (!conditionMet()) // or, perhaps more correctly,
>>> !conditionChanged()
>>> s.await();
>>> else
>>> s.cancel();
>>>
>>> So, why is the queue empty while the thread is blocked on waiting for
>>> the signal?
>>>
>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>>> about 5 years ago.
>>>
>>> Is this suppose to be a wall clock time in milliseconds since 1 January
>>> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
>>> context, and I'm just wrongly interpreting its value as a wall clock time?
>>>
>>>
>>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>>> object believes that there's nothing waiting for the signal, yet the
>>> "read-hotness-tracker:1" thread is clearly waiting for it.
>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>
>>> I think I'm getting really close now. This seems to have something to do
>>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>>
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>> <------ here
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>> at
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable:
>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>> <------ and here
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable: org.apache.cassandra.db.Mutation#54491
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at
>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable:
>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at
>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable:
>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable:
>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable:
>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>
>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>>> throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); // <--- and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>>
>>> The Keyspace.writeOrder.start() method call blocks the
>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>>
>>> This then got stuck on the signal.awaitUninterruptibly()
>>>
>>> Now I know what is blocking the memtable flushing, but I haven't been
>>> able to figure out is why it got stuck on waiting for that signal.
>>>
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>
>>> I will give it a try and see what I can find. I plan to go down the
>>> rabbit hole tomorrow. Will keep you updated.
>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>
>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>
>>> Writes on this node starts to timeout and fail. But if left untouched,
>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can see that both of the
>>> MemtableFlushWriter threads are stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as blocking progress,
>>> permitting them to exceed our memory limit
>>> // if they are stuck waiting on it, then wait for them all
>>> to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <----------- stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the commitLogUpperBound to
>>> be set, and so that waiters
>>> // on this task can rely on all prior flushes being
>>> complete
>>> latch.await(); // <----------- stuck here
>>> }
>>> Our top suspect is CDC interacting with repair, since this started to
>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>> running. But we have not been able to reproduce this in a testing cluster,
>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>> posting it in the mailing lists and hoping someone may know something about
>>> it or point me to the right direction.
>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>> read commands” that have a barrier on the memtable that prevent release.
>>>
>>> You’ve got the heap (great, way better than most people debugging), are
>>> you able to navigate through it and look for references to that memtable or
>>> other things holding a barrier?
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Abe,
The com.datastax.oss.cdc.agent thing in the thread dump is the code
forked from https://github.com/datastax/cdc-apache-cassandra/ . It sends
the CDC data to Kafka instead of Pulsar, and some other additional
features/changes.
Due to the lack of documentation and examples on how to read the CDC
commitlog files, our developers couldn't figure out how to do it
independently, so they built on top of the Datastax CDC agent, and used
that to process the files. I don't like the idea of Java agent either.
It makes my life hard too. Each time they make changes and release a new
version of the jar file, I'll have to do a full cluster rolling restart.
But this is the only thing we have, so we don't really have a choice.
Since this issue only happens on our production environment, and the CDC
agent is actively being used, I can't remove it without impacting
the production workload.
The pool-5-thread-1 thread you mentioned is waiting on this line
<https://github.com/datastax/cdc-apache-cassandra/blob/cd886ceb2ab9669a04ded2c7742b357846e8fefa/agent/src/main/java/com/datastax/oss/cdc/agent/CommitLogReaderService.java#L96>:
@Override
public void run() {
while(true) {
try {
File file = commitLogQueue.take(); // <--- here
submitCommitLog(file);
} catch (InterruptedException e) {
log.error("error:", e);
}
}
}
Note that the line number in thread dump won't match exactly, because
our developers forked it and made changes to the code.
Cheers,
Bowen
On 08/11/2024 18:35, Abe Ratnofsky wrote:
> Hey Bowen,
>
> Are you able to reproduce the issue without running
> com.datastax.oss.cdc.agent? I don't see any glaringly obvious bugs
> there indicated by the thread dump, but it would be useful to rule
> that out, particularly because it runs as an agent and manages new
> instances of C*-defined classes, potentially on executors that aren't
> expecting to run those classes.
>
> I'm curious what threads like this one are waiting on:
>
> "pool-5-thread-1" prio=5 tid=37 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> local variable: java.lang.Thread#25
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
> local variable:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#16
> local variable:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#62
> at
> java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:546)
> local variable: java.util.concurrent.PriorityBlockingQueue#1
> local variable: java.util.concurrent.locks.ReentrantLock#24
> at
> com.datastax.oss.cdc.agent.CommitLogReaderService.run(CommitLogReaderService.java:95)
> local variable:
> com.datastax.oss.cdc.agent.CommitLogReaderServiceImpl#1
>
>> On Nov 8, 2024, at 12:22, Dmitry Konstantinov <ne...@gmail.com> wrote:
>>
>> Hi Bowen,
>>
>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
>> looks like the code is inconsistent on whether lastSyncedAt is a
>> relative monotonic clock or absolute epoch time.
>> Yes, I think so, I am going to create a defect regarding it.
>>
>> >> I've also attempted to search for all RegisteredSignal objects in
>> the heap, and found 15 of them, but none of them is related to the
>> read-hotness-tracker:1 thread.
>> It is very strange because the stack trace for this thread waits
>> within the object and read-hotness-tracker:1 thread has active
>> "WaitQueue.Signal signal" local variable..
>> What tool do you use for the heap dump inspection? I use Eclipse
>> Memory Analyzer. It has "threads_overview" view and you can check the
>> values for local variables within a thread stack.
>> Like this:<image.png>
>>
>>
>> Regards,
>> Dmitry
>>
>>
>>
>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>> Hi Dmitry,
>> Thank you for helping me.
>> You are right, that's a monotonic clock. I was definitely wrongly
>> interpreting its value. I got misled by this line of code: "protected
>> volatile long lastSyncedAt = currentTimeMillis();". It looks like the
>> code is inconsistent on whether lastSyncedAt is a relative monotonic
>> clock or absolute epoch time.
>> Please see the attached cassandra.yaml with sensitive information
>> masked out with asterisks.
>> There's no ERROR level log in the system.log other than some
>> guardrail violations (items_per_collection, if it matters) and random
>> and rare network issues such as connection timeout in the few days
>> run up to and during the event.
>> We have the stdout and stderr captured in the systemd journal, and
>> they have been quiet in during that period as well, not a single line
>> got logged.
>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
>> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
>> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other
>> two has cdcState = CONTAINS lastSyncedOffset = 27007818
>> lastMarkerOffset = 27007818 and cdcState = PERMITTED lastSyncedOffset
>> = 20 lastMarkerOffset = 20 respectively.
>> The CDC index files have been deleted by the CDC consumer, and we
>> don't keep a copy of these files elsewhere, so I cannot check the
>> offsets in them unfortunately.
>> Regarding the WaitQueue, I found the queue to be empty in the heap
>> dump, which means there's no RegisteredSignal objects in the queue at
>> all. The queue is a ConcurrentLinkedQueue object, which has a head
>> and a tail. The head has item=null and next=null, and the tail has
>> item=null and next=head. I've also attempted to search for all
>> RegisteredSignal objects in the heap, and found 15 of them, but none
>> of them is related to the read-hotness-tracker:1 thread. They are for
>> the following threads:
>> • MemtablePostFlush:1
>> • AntiEntropyStage:1
>> • MemtableFlushWriter:910
>> • COMMIT-LOG-ALLOCATOR
>> • NativePoolCleaner
>> • MemtableFlushWriter:911
>> • Native-Transport-Requests-135
>> • Native-Transport-Requests-139
>> • Native-Transport-Requests-132
>> • Native-Transport-Requests-143
>> • Native-Transport-Requests-129
>> • Native-Transport-Requests-140
>> • Native-Transport-Requests-153
>> • Native-Transport-Requests-141
>> • Native-Transport-Requests-137
>> This looks rather suspicious to me, as the read-hotness-tracker:1
>> thread is blocked and waiting for a non-existence registered signal.
>> Cheers,
>> Bowen
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>> Hi Bowen,
>>>
>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>> pollStarted;
>>> where long pollStarted = clock.now();
>>> the logic uses clock from: SyncRunnable sync = new
>>> SyncRunnable(preciseTime)
>>> preciseTime by default is SystemClock
>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
>>> System.nanoTime()
>>> so, the field is populated from System.nanoTime() which returns a
>>> relative timestamp, which cannot be treated as an absolute time like
>>> linux epoch (it is calculated like this:
>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301 )
>>>
>>>
>>> Can you share your cassandra.yaml file? (with removed IPs, paths and
>>> other security sensitive things, of course)
>>> Do you have any exceptions in your system.log related to commit log?
>>> Do you have JVM stdout/stderr captured to a file? Does it have any
>>> exceptions? (which can be present there if Cassandra does not catch
>>> them properly by itself)
>>>
>>> ---
>>> How many CommitLogSegment objects do you have in your heap dump?
>>> What are values for the following fields of CommitLogSegment objects?
>>> lastSyncedOffset
>>> lastMarkerOffset
>>> cdcState
>>>
>>> Do you have CDC index files written by
>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>> If you have them - what are the last offsets written there?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>> The syncComplete is an instance of WaitQueue, and the
>>> syncComplete.queue is an instance of ConcurrentLinkedQueue.
>>> Surprisingly, the queue is empty. There is no item in the queue's
>>> linked list, only the head and tail nodes, each has item=null.
>>> The usage of the WaitQueue within the AbstractCommitLogService looks
>>> correct to me:
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly();
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> This matches the example given in the WaitQueue's documentation
>>> (which is definitely missing a pair of curly brackets and has bad
>>> indentation):
>>> while (!conditionMet())
>>> Signal s = q.register();
>>> if (!conditionMet()) // or, perhaps more correctly,
>>> !conditionChanged()
>>> s.await();
>>> else
>>> s.cancel();
>>> So, why is the queue empty while the thread is blocked on waiting
>>> for the signal?
>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>>> 1573038452910796, which translates to 6 November 2019
>>> 11:07:32.910796, about 5 years ago.
>>> Is this suppose to be a wall clock time in milliseconds since 1
>>> January 1970 00:00? Or is it a monotonic clock time that has no
>>> meaning out of the context, and I'm just wrongly interpreting its
>>> value as a wall clock time?
>>>
>>> I've found that the syncComplete.queue is empty, meaning the
>>> WaitQueue object believes that there's nothing waiting for the
>>> signal, yet the "read-hotness-tracker:1" thread is clearly waiting
>>> for it.
>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>> I think I'm getting really close now. This seems to have something
>>>> to do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324) <------
>>>> here
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>> at
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>> local variable:
>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>> <------ and here
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>> at
>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>> at
>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>> local variable: java.lang.String#42806
>>>> local variable: java.lang.String#47659
>>>> local variable:
>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>>> at
>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>> at
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>> local variable:
>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> local variable:
>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> at
>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>> at java.lang.Thread.run(Thread.java:829)
>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>
>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>> public WriteContext beginWrite(Mutation mutation, boolean
>>>> makeDurable) throws RequestExecutionException
>>>> {
>>>> OpOrder.Group group = null;
>>>> try
>>>> {
>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>
>>>> // write the mutation to the commitlog and memtables
>>>> CommitLogPosition position = null;
>>>> if (makeDurable)
>>>> {
>>>> position = addToCommitLog(mutation); // <--- and here
>>>> }
>>>> return new CassandraWriteContext(group, position);
>>>> }
>>>> catch (Throwable t)
>>>> {
>>>> if (group != null)
>>>> {
>>>> group.close();
>>>> }
>>>> throw t;
>>>> }
>>>> }
>>>> The Keyspace.writeOrder.start() method call blocks the
>>>> MemtableFlushWriter thread, and then it
>>>> calls addToCommitLog(mutation) which indirectly
>>>> calls AbstractCommitLogService.awaitSyncAt()
>>>> void awaitSyncAt(long syncTime, Context context)
>>>> {
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ?
>>>> syncComplete.register(context, Context::stop) :
>>>> syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly(); // <--- here
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>> }
>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>> Now I know what is blocking the memtable flushing, but I haven't
>>>> been able to figure out is why it got stuck on waiting for that signal.
>>>> I would appreciate it if anyone can offer some insight here.
>>>>
>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>> I will give it a try and see what I can find. I plan to go down
>>>>> the rabbit hole tomorrow. Will keep you updated.
>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>
>>>>>>
>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>>>> <us...@cassandra.apache.org> wrote:
>>>>>>>
>>>>>>> Writes on this node starts to timeout and fail. But if left
>>>>>>> untouched, it's only gonna get worse, and eventually lead to JVM
>>>>>>> OOM and crash.
>>>>>>> By inspecting the heap dump created at OOM, we can see that both
>>>>>>> of the MemtableFlushWriter threads are stuck on line 1190 in the
>>>>>>> ColumnFamilyStore.java:
>>>>>>> // mark writes older than the barrier as blocking
>>>>>>> progress, permitting them to exceed our memory limit
>>>>>>> // if they are stuck waiting on it, then wait for
>>>>>>> them all to complete
>>>>>>> writeBarrier.markBlocking();
>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>> And the MemtablePostFlush thread is stuck on line 1094 in the
>>>>>>> same file.
>>>>>>> try
>>>>>>> {
>>>>>>> // we wait on the latch for the
>>>>>>> commitLogUpperBound to be set, and so that waiters
>>>>>>> // on this task can rely on all prior flushes
>>>>>>> being complete
>>>>>>> latch.await(); // <----------- stuck here
>>>>>>> }
>>>>>>> Our top suspect is CDC interacting with repair, since this
>>>>>>> started to happen shortly after we enabled CDC on the nodes, and
>>>>>>> each time repair was running. But we have not been able to
>>>>>>> reproduce this in a testing cluster, and don't know what's the
>>>>>>> next step to troubleshoot this issue. So I'm posting it in the
>>>>>>> mailing lists and hoping someone may know something about it or
>>>>>>> point me to the right direction.
>>>>>>
>>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>>>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long
>>>>>> running read commands” that have a barrier on the memtable that
>>>>>> prevent release.
>>>>>>
>>>>>> You’ve got the heap (great, way better than most people
>>>>>> debugging), are you able to navigate through it and look for
>>>>>> references to that memtable or other things holding a barrier?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>> On Nov 8, 2024, at 12:22, Dmitry Konstantinov <ne...@gmail.com> wrote:
>>
>> Hi Bowen,
>>
>> >> protected volatile long lastSyncedAt = currentTimeMillis();". It
>> looks like the code is inconsistent on whether lastSyncedAt is a
>> relative monotonic clock or absolute epoch time.
>> Yes, I think so, I am going to create a defect regarding it.
>>
>> >> I've also attempted to search for all RegisteredSignal objects in
>> the heap, and found 15 of them, but none of them is related to the
>> read-hotness-tracker:1 thread.
>> It is very strange because the stack trace for this thread waits
>> within the object and read-hotness-tracker:1 thread has active
>> "WaitQueue.Signal signal" local variable..
>> What tool do you use for the heap dump inspection? I use Eclipse
>> Memory Analyzer. It has "threads_overview" view and you can check the
>> values for local variables within a thread stack.
>> Like this:
>> <image.png>
>>
>> Regards,
>> Dmitry
>>
>>
>>
>> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Hi Dmitry,
>>
>> Thank you for helping me.
>>
>> You are right, that's a monotonic clock. I was definitely wrongly
>> interpreting its value. I got misled bythis line of code
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
>> "protected volatile long lastSyncedAt = currentTimeMillis();". It
>> looks like the code is inconsistent on whether lastSyncedAt is a
>> relative monotonic clock or absolute epoch time.
>>
>> Please see the attached cassandra.yaml with sensitive information
>> masked out with asterisks.
>>
>> There's no ERROR level log in the system.log other than some
>> guardrail violations (items_per_collection, if it matters) and
>> random and rare network issues such as connection timeout in the
>> few days run up to and during the event.
>>
>> We have the stdout and stderr captured in the systemd journal,
>> and they have been quiet in during that period as well, not a
>> single line got logged.
>>
>> There's 196 CommitLogSegment (more specifically,
>> MemoryMappedSegment) objects in the heap dump. Of these, 194 has
>> cdcState = CONTAINS lastSyncedOffset = 33554432 lastMarkerOffset
>> = 33554432. The other two has cdcState = CONTAINS
>> lastSyncedOffset = 27007818 lastMarkerOffset = 27007818 and
>> cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
>> respectively.
>>
>> The CDC index files have been deleted by the CDC consumer, and we
>> don't keep a copy of these files elsewhere, so I cannot check the
>> offsets in them unfortunately.
>>
>> Regarding the WaitQueue, I found the queue to be empty in the
>> heap dump, which means there's no RegisteredSignal objects in the
>> queue at all. The queue is a ConcurrentLinkedQueue object, which
>> has a head and a tail. The head has item=null and next=null, and
>> the tail has item=null and next=head. I've also attempted to
>> search for all RegisteredSignal objects in the heap, and found 15
>> of them, but none of them is related to the
>> read-hotness-tracker:1 thread. They are for the following threads:
>>
>> * MemtablePostFlush:1
>> * AntiEntropyStage:1
>> * MemtableFlushWriter:910
>> * COMMIT-LOG-ALLOCATOR
>> * NativePoolCleaner
>> * MemtableFlushWriter:911
>> * Native-Transport-Requests-135
>> * Native-Transport-Requests-139
>> * Native-Transport-Requests-132
>> * Native-Transport-Requests-143
>> * Native-Transport-Requests-129
>> * Native-Transport-Requests-140
>> * Native-Transport-Requests-153
>> * Native-Transport-Requests-141
>> * Native-Transport-Requests-137
>>
>> This looks rather suspicious to me, as the read-hotness-tracker:1
>> thread is blocked and waiting for a non-existence registered signal.
>>
>> Cheers,
>> Bowen
>>
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>> Hi Bowen,
>>>
>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt =
>>> pollStarted;
>>> where long pollStarted = clock.now();
>>> the logic uses clock from: SyncRunnable sync = new
>>> SyncRunnable(preciseTime)
>>> preciseTime by default is SystemClock
>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime
>>> -> System.nanoTime()
>>> so, the field is populated from System.nanoTime() which returns
>>> a relative timestamp, which cannot be treated as an absolute
>>> time like linux epoch (it is calculated like this:
>>> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301)
>>>
>>>
>>> Can you share your cassandra.yaml file? (with removed IPs, paths
>>> and other security sensitive things, of course)
>>> Do you have any exceptions in your system.log related to commit log?
>>> Do you have JVM stdout/stderr captured to a file? Does it have
>>> any exceptions? (which can be present there if Cassandra does
>>> not catch them properly by itself)
>>>
>>> ---
>>> How many CommitLogSegment objects do you have in your heap dump?
>>> What are values for the following fields of CommitLogSegment
>>> objects?
>>> lastSyncedOffset
>>> lastMarkerOffset
>>> cdcState
>>>
>>> Do you have CDC index files written by
>>> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile
>>> ?
>>> If you have them - what are the last offsets written there?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> The syncComplete is an instance of WaitQueue, and the
>>> syncComplete.queue is an instance of ConcurrentLinkedQueue.
>>> Surprisingly, the queue is empty. There is no item in the
>>> queue's linked list, only the head and tail nodes, each has
>>> item=null.
>>>
>>> The usage of the WaitQueue within the
>>> AbstractCommitLogService looks correct to me:
>>>
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) :
>>> syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly();
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>>
>>> This matches the example given in the WaitQueue's
>>> documentation (which is definitely missing a pair of curly
>>> brackets and has bad indentation):
>>>
>>> while (!conditionMet())
>>> Signal s = q.register();
>>> if (!conditionMet()) // or, perhaps more correctly,
>>> !conditionChanged()
>>> s.await();
>>> else
>>> s.cancel();
>>>
>>> So, why is the queue empty while the thread is blocked on
>>> waiting for the signal?
>>>
>>> Further digging showed that
>>> PeriodicCommitLogService.lastSyncedAt = 1573038452910796,
>>> which translates to 6 November 2019 11:07:32.910796, about 5
>>> years ago.
>>>
>>> Is this suppose to be a wall clock time in milliseconds
>>> since 1 January 1970 00:00? Or is it a monotonic clock time
>>> that has no meaning out of the context, and I'm just wrongly
>>> interpreting its value as a wall clock time?
>>>
>>>
>>> I've found that the syncComplete.queue is empty, meaning the
>>> WaitQueue object believes that there's nothing waiting for
>>> the signal, yet the "read-hotness-tracker:1" thread is
>>> clearly waiting for it.
>>>
>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>
>>>> I think I'm getting really close now. This seems to have
>>>> something to do with the "read-hotness-tracker:1" thread.
>>>> The thread dump is:
>>>>
>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>> at
>>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)<------
>>>> here
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>> at
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>> local variable:
>>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>> local variable:
>>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>> at
>>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>>> <------ and here
>>>> local variable:
>>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>> at
>>>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>> at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>> at
>>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>> at
>>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>> local variable: java.lang.String#42806
>>>> local variable: java.lang.String#47659
>>>> local variable:
>>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>> local variable:
>>>> org.apache.cassandra.metrics.RestorableMeter#752
>>>> at
>>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>> at
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>> local variable:
>>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> local variable:
>>>> java.util.concurrent.Executors$RunnableAdapter#934
>>>> at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> local variable:
>>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>> local variable:
>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>> local variable:
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> local variable:
>>>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>> at
>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>> at java.lang.Thread.run(Thread.java:829)
>>>> local variable:
>>>> io.netty.util.concurrent.FastThreadLocalThread#7
>>>>
>>>>
>>>> In theCassandraKeyspaceWriteHandler.beginWrite()method:
>>>>
>>>> public WriteContext beginWrite(Mutation mutation, boolean
>>>> makeDurable) throws RequestExecutionException
>>>> {
>>>> OpOrder.Group group = null;
>>>> try
>>>> {
>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>
>>>> // write the mutation to the commitlog and memtables
>>>> CommitLogPosition position = null;
>>>> if (makeDurable)
>>>> {
>>>> position = addToCommitLog(mutation);// <--- and here
>>>> }
>>>> return new CassandraWriteContext(group, position);
>>>> }
>>>> catch (Throwable t)
>>>> {
>>>> if (group != null)
>>>> {
>>>> group.close();
>>>> }
>>>> throw t;
>>>> }
>>>> }
>>>>
>>>> TheKeyspace.writeOrder.start()method call blocks the
>>>> MemtableFlushWriter thread, and then it
>>>> callsaddToCommitLog(mutation)which indirectly
>>>> callsAbstractCommitLogService.awaitSyncAt()
>>>>
>>>> void awaitSyncAt(long syncTime, Context context)
>>>> {
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ?
>>>> syncComplete.register(context, Context::stop) :
>>>> syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly(); // <--- here
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>> }
>>>>
>>>> This then got stuck on thesignal.awaitUninterruptibly()
>>>>
>>>> Now I know what is blocking the memtable flushing, but I
>>>> haven't been able to figure out is why it got stuck on
>>>> waiting for that signal.
>>>>
>>>> I would appreciate it if anyone can offer some insight here.
>>>>
>>>>
>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>
>>>>> I will give it a try and see what I can find. I plan to go
>>>>> down the rabbit hole tomorrow. Will keep you updated.
>>>>>
>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>
>>>>>>
>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via
>>>>>>> user<us...@cassandra.apache.org>
>>>>>>> <ma...@cassandra.apache.org>wrote:
>>>>>>>
>>>>>>> Writes on this node starts to timeout and fail. But if
>>>>>>> left untouched, it's only gonna get worse, and
>>>>>>> eventually lead to JVM OOM and crash.
>>>>>>>
>>>>>>> By inspecting the heap dump created at OOM, we can see
>>>>>>> that both of the MemtableFlushWriter threads are stuck
>>>>>>> online 1190
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>in
>>>>>>> the ColumnFamilyStore.java:
>>>>>>>
>>>>>>> // mark writes older than the barrier as blocking
>>>>>>> progress, permitting them to exceed our memory limit
>>>>>>> // if they are stuck waiting on it, then wait for them
>>>>>>> all to complete
>>>>>>> writeBarrier.markBlocking();
>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>>
>>>>>>> And the MemtablePostFlush thread is stuck online 1094
>>>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>in
>>>>>>> the same file.
>>>>>>>
>>>>>>> try
>>>>>>> {
>>>>>>> // we wait on the latch for the commitLogUpperBound to
>>>>>>> be set, and so that waiters
>>>>>>> // on this task can rely on all prior flushes being complete
>>>>>>> latch.await();// <----------- stuck here
>>>>>>> }
>>>>>>> Our top suspect is CDC interacting with repair, since
>>>>>>> this started to happen shortly after we enabled CDC on
>>>>>>> the nodes, and each time repair was running. But we have
>>>>>>> not been able to reproduce this in a testing cluster,
>>>>>>> and don't know what's the next step to troubleshoot this
>>>>>>> issue. So I'm posting it in the mailing lists and hoping
>>>>>>> someone may know something about it or point me to the
>>>>>>> right direction.
>>>>>>>
>>>>>>
>>>>>> Wouldn’t be completely surprised if CDC or repair
>>>>>> somehow has a barrier, I’ve also seen similar behavior
>>>>>> pre-3.0 with “very long running read commands” that have
>>>>>> a barrier on the memtable that prevent release.
>>>>>>
>>>>>> You’ve got the heap (great, way better than most people
>>>>>> debugging), are you able to navigate through it and look
>>>>>> for references to that memtable or other things holding a
>>>>>> barrier?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
Re: Unexplained stuck memtable flush
Posted by Abe Ratnofsky <ab...@aber.io>.
Hey Bowen,
Are you able to reproduce the issue without running com.datastax.oss.cdc.agent? I don't see any glaringly obvious bugs there indicated by the thread dump, but it would be useful to rule that out, particularly because it runs as an agent and manages new instances of C*-defined classes, potentially on executors that aren't expecting to run those classes.
I'm curious what threads like this one are waiting on:
"pool-5-thread-1" prio=5 tid=37 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
local variable: java.lang.Thread#25
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#16
local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#62
at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:546)
local variable: java.util.concurrent.PriorityBlockingQueue#1
local variable: java.util.concurrent.locks.ReentrantLock#24
at com.datastax.oss.cdc.agent.CommitLogReaderService.run(CommitLogReaderService.java:95)
local variable: com.datastax.oss.cdc.agent.CommitLogReaderServiceImpl#1
> On Nov 8, 2024, at 12:22, Dmitry Konstantinov <ne...@gmail.com> wrote:
>
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();". It looks like the code is inconsistent on whether lastSyncedAt is a relative monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects in the heap, and found 15 of them, but none of them is related to the read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits within the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse Memory Analyzer. It has "threads_overview" view and you can check the values for local variables within a thread stack.
> Like this:<image.png>
>
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <us...@cassandra.apache.org> wrote:
> Hi Dmitry,
> Thank you for helping me.
> You are right, that's a monotonic clock. I was definitely wrongly interpreting its value. I got misled by this line of code: "protected volatile long lastSyncedAt = currentTimeMillis();". It looks like the code is inconsistent on whether lastSyncedAt is a relative monotonic clock or absolute epoch time.
> Please see the attached cassandra.yaml with sensitive information masked out with asterisks.
> There's no ERROR level log in the system.log other than some guardrail violations (items_per_collection, if it matters) and random and rare network issues such as connection timeout in the few days run up to and during the event.
> We have the stdout and stderr captured in the systemd journal, and they have been quiet in during that period as well, not a single line got logged.
> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment) objects in the heap dump. Of these, 194 has cdcState = CONTAINS lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818 and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20 respectively.
> The CDC index files have been deleted by the CDC consumer, and we don't keep a copy of these files elsewhere, so I cannot check the offsets in them unfortunately.
> Regarding the WaitQueue, I found the queue to be empty in the heap dump, which means there's no RegisteredSignal objects in the queue at all. The queue is a ConcurrentLinkedQueue object, which has a head and a tail. The head has item=null and next=null, and the tail has item=null and next=head. I've also attempted to search for all RegisteredSignal objects in the heap, and found 15 of them, but none of them is related to the read-hotness-tracker:1 thread. They are for the following threads:
> • MemtablePostFlush:1
> • AntiEntropyStage:1
> • MemtableFlushWriter:910
> • COMMIT-LOG-ALLOCATOR
> • NativePoolCleaner
> • MemtableFlushWriter:911
> • Native-Transport-Requests-135
> • Native-Transport-Requests-139
> • Native-Transport-Requests-132
> • Native-Transport-Requests-143
> • Native-Transport-Requests-129
> • Native-Transport-Requests-140
> • Native-Transport-Requests-153
> • Native-Transport-Requests-141
> • Native-Transport-Requests-137
> This looks rather suspicious to me, as the read-hotness-tracker:1 thread is blocked and waiting for a non-existence registered signal.
> Cheers,
> Bowen
> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>> Hi Bowen,
>>
>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
>> where long pollStarted = clock.now();
>> the logic uses clock from: SyncRunnable sync = new SyncRunnable(preciseTime)
>> preciseTime by default is SystemClock
>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime -> System.nanoTime()
>> so, the field is populated from System.nanoTime() which returns a relative timestamp, which cannot be treated as an absolute time like linux epoch (it is calculated like this: https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301 )
>>
>> Can you share your cassandra.yaml file? (with removed IPs, paths and other security sensitive things, of course)
>> Do you have any exceptions in your system.log related to commit log?
>> Do you have JVM stdout/stderr captured to a file? Does it have any exceptions? (which can be present there if Cassandra does not catch them properly by itself)
>>
>> ---
>> How many CommitLogSegment objects do you have in your heap dump?
>> What are values for the following fields of CommitLogSegment objects?
>> lastSyncedOffset
>> lastMarkerOffset
>> cdcState
>>
>> Do you have CDC index files written by org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>> If you have them - what are the last offsets written there?
>>
>> Regards,
>> Dmitry
>>
>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <us...@cassandra.apache.org> wrote:
>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty. There is no item in the queue's linked list, only the head and tail nodes, each has item=null.
>> The usage of the WaitQueue within the AbstractCommitLogService looks correct to me:
>> do
>> {
>> WaitQueue.Signal signal = context != null ? syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly();
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> This matches the example given in the WaitQueue's documentation (which is definitely missing a pair of curly brackets and has bad indentation):
>> while (!conditionMet())
>> Signal s = q.register();
>> if (!conditionMet()) // or, perhaps more correctly, !conditionChanged()
>> s.await();
>> else
>> s.cancel();
>> So, why is the queue empty while the thread is blocked on waiting for the signal?
>> Further digging showed that PeriodicCommitLogService.lastSyncedAt = 1573038452910796, which translates to 6 November 2019 11:07:32.910796, about 5 years ago.
>> Is this suppose to be a wall clock time in milliseconds since 1 January 1970 00:00? Or is it a monotonic clock time that has no meaning out of the context, and I'm just wrongly interpreting its value as a wall clock time?
>>
>> I've found that the syncComplete.queue is empty, meaning the WaitQueue object believes that there's nothing waiting for the signal, yet the "read-hotness-tracker:1" thread is clearly waiting for it.
>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>> I think I'm getting really close now. This seems to have something to do with the "read-hotness-tracker:1" thread. The thread dump is:
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324) <------ here
>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>> at org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable: org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54) <------ and here
>>> local variable: org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable: org.apache.cassandra.db.Mutation#54491
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable: org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>> at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable: org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable: org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable: org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable) throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); // <--- and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>> The Keyspace.writeOrder.start() method call blocks the MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which indirectly calls AbstractCommitLogService.awaitSyncAt()
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ? syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>> This then got stuck on the signal.awaitUninterruptibly()
>>> Now I know what is blocking the memtable flushing, but I haven't been able to figure out is why it got stuck on waiting for that signal.
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>> I will give it a try and see what I can find. I plan to go down the rabbit hole tomorrow. Will keep you updated.
>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>
>>>>>
>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user <us...@cassandra.apache.org> wrote:
>>>>>>
>>>>>> Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>>>> By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190 in the ColumnFamilyStore.java:
>>>>>> // mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
>>>>>> // if they are stuck waiting on it, then wait for them all to complete
>>>>>> writeBarrier.markBlocking();
>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>> And the MemtablePostFlush thread is stuck on line 1094 in the same file.
>>>>>> try
>>>>>> {
>>>>>> // we wait on the latch for the commitLogUpperBound to be set, and so that waiters
>>>>>> // on this task can rely on all prior flushes being complete
>>>>>> latch.await(); // <----------- stuck here
>>>>>> }
>>>>>> Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
>>>>>
>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a barrier, I’ve also seen similar behavior pre-3.0 with “very long running read commands” that have a barrier on the memtable that prevent release.
>>>>>
>>>>> You’ve got the heap (great, way better than most people debugging), are you able to navigate through it and look for references to that memtable or other things holding a barrier?
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
> On Nov 8, 2024, at 12:22, Dmitry Konstantinov <ne...@gmail.com> wrote:
>
> Hi Bowen,
>
> >> protected volatile long lastSyncedAt = currentTimeMillis();". It looks like the code is inconsistent on whether lastSyncedAt is a relative monotonic clock or absolute epoch time.
> Yes, I think so, I am going to create a defect regarding it.
>
> >> I've also attempted to search for all RegisteredSignal objects in the heap, and found 15 of them, but none of them is related to the read-hotness-tracker:1 thread.
> It is very strange because the stack trace for this thread waits within the object and read-hotness-tracker:1 thread has active "WaitQueue.Signal signal" local variable..
> What tool do you use for the heap dump inspection? I use Eclipse Memory Analyzer. It has "threads_overview" view and you can check the values for local variables within a thread stack.
> Like this:
> <image.png>
>
> Regards,
> Dmitry
>
>
>
> On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <user@cassandra.apache.org <ma...@cassandra.apache.org>> wrote:
>> Hi Dmitry,
>>
>> Thank you for helping me.
>>
>> You are right, that's a monotonic clock. I was definitely wrongly interpreting its value. I got misled by this line of code <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>: "protected volatile long lastSyncedAt = currentTimeMillis();". It looks like the code is inconsistent on whether lastSyncedAt is a relative monotonic clock or absolute epoch time.
>>
>> Please see the attached cassandra.yaml with sensitive information masked out with asterisks.
>>
>> There's no ERROR level log in the system.log other than some guardrail violations (items_per_collection, if it matters) and random and rare network issues such as connection timeout in the few days run up to and during the event.
>>
>> We have the stdout and stderr captured in the systemd journal, and they have been quiet in during that period as well, not a single line got logged.
>>
>> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment) objects in the heap dump. Of these, 194 has cdcState = CONTAINS lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818 and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20 respectively.
>>
>> The CDC index files have been deleted by the CDC consumer, and we don't keep a copy of these files elsewhere, so I cannot check the offsets in them unfortunately.
>>
>> Regarding the WaitQueue, I found the queue to be empty in the heap dump, which means there's no RegisteredSignal objects in the queue at all. The queue is a ConcurrentLinkedQueue object, which has a head and a tail. The head has item=null and next=null, and the tail has item=null and next=head. I've also attempted to search for all RegisteredSignal objects in the heap, and found 15 of them, but none of them is related to the read-hotness-tracker:1 thread. They are for the following threads:
>>
>> MemtablePostFlush:1
>> AntiEntropyStage:1
>> MemtableFlushWriter:910
>> COMMIT-LOG-ALLOCATOR
>> NativePoolCleaner
>> MemtableFlushWriter:911
>> Native-Transport-Requests-135
>> Native-Transport-Requests-139
>> Native-Transport-Requests-132
>> Native-Transport-Requests-143
>> Native-Transport-Requests-129
>> Native-Transport-Requests-140
>> Native-Transport-Requests-153
>> Native-Transport-Requests-141
>> Native-Transport-Requests-137
>> This looks rather suspicious to me, as the read-hotness-tracker:1 thread is blocked and waiting for a non-existence registered signal.
>>
>> Cheers,
>> Bowen
>>
>> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>>> Hi Bowen,
>>>
>>> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
>>> where long pollStarted = clock.now();
>>> the logic uses clock from: SyncRunnable sync = new SyncRunnable(preciseTime)
>>> preciseTime by default is SystemClock
>>> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime -> System.nanoTime()
>>> so, the field is populated from System.nanoTime() which returns a relative timestamp, which cannot be treated as an absolute time like linux epoch (it is calculated like this: https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301 )
>>>
>>> Can you share your cassandra.yaml file? (with removed IPs, paths and other security sensitive things, of course)
>>> Do you have any exceptions in your system.log related to commit log?
>>> Do you have JVM stdout/stderr captured to a file? Does it have any exceptions? (which can be present there if Cassandra does not catch them properly by itself)
>>>
>>> ---
>>> How many CommitLogSegment objects do you have in your heap dump?
>>> What are values for the following fields of CommitLogSegment objects?
>>> lastSyncedOffset
>>> lastMarkerOffset
>>> cdcState
>>>
>>> Do you have CDC index files written by org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
>>> If you have them - what are the last offsets written there?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <user@cassandra.apache.org <ma...@cassandra.apache.org>> wrote:
>>>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty. There is no item in the queue's linked list, only the head and tail nodes, each has item=null.
>>>>
>>>> The usage of the WaitQueue within the AbstractCommitLogService looks correct to me:
>>>>
>>>> do
>>>> {
>>>> WaitQueue.Signal signal = context != null ? syncComplete.register(context, Context::stop) : syncComplete.register();
>>>> if (lastSyncedAt < syncTime)
>>>> signal.awaitUninterruptibly();
>>>> else
>>>> signal.cancel();
>>>> }
>>>> while (lastSyncedAt < syncTime);
>>>>
>>>> This matches the example given in the WaitQueue's documentation (which is definitely missing a pair of curly brackets and has bad indentation):
>>>>
>>>> while (!conditionMet())
>>>> Signal s = q.register();
>>>> if (!conditionMet()) // or, perhaps more correctly, !conditionChanged()
>>>> s.await();
>>>> else
>>>> s.cancel();
>>>>
>>>> So, why is the queue empty while the thread is blocked on waiting for the signal?
>>>>
>>>> Further digging showed that PeriodicCommitLogService.lastSyncedAt = 1573038452910796, which translates to 6 November 2019 11:07:32.910796, about 5 years ago.
>>>>
>>>> Is this suppose to be a wall clock time in milliseconds since 1 January 1970 00:00? Or is it a monotonic clock time that has no meaning out of the context, and I'm just wrongly interpreting its value as a wall clock time?
>>>>
>>>>
>>>>
>>>> I've found that the syncComplete.queue is empty, meaning the WaitQueue object believes that there's nothing waiting for the signal, yet the "read-hotness-tracker:1" thread is clearly waiting for it.
>>>>
>>>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>>>> I think I'm getting really close now. This seems to have something to do with the "read-hotness-tracker:1" thread. The thread dump is:
>>>>>
>>>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>>>> at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>>>> local variable: org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>>>> at org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>>>> at org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324) <------ here
>>>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>>>> at org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>>>> local variable: org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>>>> local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>>>> local variable: org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>>>> at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>>>> at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54) <------ and here
>>>>> local variable: org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>>>> local variable: org.apache.cassandra.db.Keyspace#8
>>>>> local variable: org.apache.cassandra.db.Mutation#54491
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>>>> at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>>>> at org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>>>> at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>>>> at org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>>>> local variable: java.lang.String#42806
>>>>> local variable: java.lang.String#47659
>>>>> local variable: org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>>>> at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>>>> at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>>>> local variable: org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>>>> local variable: org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>> local variable: org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>> local variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>>>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>>>>> at java.lang.Thread.run(Thread.java:829)
>>>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>>>
>>>>>
>>>>>
>>>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>>>
>>>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable) throws RequestExecutionException
>>>>> {
>>>>> OpOrder.Group group = null;
>>>>> try
>>>>> {
>>>>> group = Keyspace.writeOrder.start(); // <--- here
>>>>>
>>>>> // write the mutation to the commitlog and memtables
>>>>> CommitLogPosition position = null;
>>>>> if (makeDurable)
>>>>> {
>>>>> position = addToCommitLog(mutation); // <--- and here
>>>>> }
>>>>> return new CassandraWriteContext(group, position);
>>>>> }
>>>>> catch (Throwable t)
>>>>> {
>>>>> if (group != null)
>>>>> {
>>>>> group.close();
>>>>> }
>>>>> throw t;
>>>>> }
>>>>> }
>>>>>
>>>>> The Keyspace.writeOrder.start() method call blocks the MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>>>
>>>>> void awaitSyncAt(long syncTime, Context context)
>>>>> {
>>>>> do
>>>>> {
>>>>> WaitQueue.Signal signal = context != null ? syncComplete.register(context, Context::stop) : syncComplete.register();
>>>>> if (lastSyncedAt < syncTime)
>>>>> signal.awaitUninterruptibly(); // <--- here
>>>>> else
>>>>> signal.cancel();
>>>>> }
>>>>> while (lastSyncedAt < syncTime);
>>>>> }
>>>>>
>>>>> This then got stuck on the signal.awaitUninterruptibly()
>>>>>
>>>>> Now I know what is blocking the memtable flushing, but I haven't been able to figure out is why it got stuck on waiting for that signal.
>>>>>
>>>>> I would appreciate it if anyone can offer some insight here.
>>>>>
>>>>>
>>>>>
>>>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>>>> I will give it a try and see what I can find. I plan to go down the rabbit hole tomorrow. Will keep you updated.
>>>>>>
>>>>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user <us...@cassandra.apache.org> <ma...@cassandra.apache.org> wrote:
>>>>>>>>
>>>>>>>> Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>>>>>>
>>>>>>>> By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190> in the ColumnFamilyStore.java:
>>>>>>>>
>>>>>>>> // mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
>>>>>>>> // if they are stuck waiting on it, then wait for them all to complete
>>>>>>>> writeBarrier.markBlocking();
>>>>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>>>>
>>>>>>>> And the MemtablePostFlush thread is stuck on line 1094 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094> in the same file.
>>>>>>>>
>>>>>>>> try
>>>>>>>> {
>>>>>>>> // we wait on the latch for the commitLogUpperBound to be set, and so that waiters
>>>>>>>> // on this task can rely on all prior flushes being complete
>>>>>>>> latch.await(); // <----------- stuck here
>>>>>>>> }
>>>>>>>> Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
>>>>>>>>
>>>>>>>
>>>>>>> Wouldn’t be completely surprised if CDC or repair somehow has a barrier, I’ve also seen similar behavior pre-3.0 with “very long running read commands” that have a barrier on the memtable that prevent release.
>>>>>>>
>>>>>>> You’ve got the heap (great, way better than most people debugging), are you able to navigate through it and look for references to that memtable or other things holding a barrier?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Hi Bowen,
>> protected volatile long lastSyncedAt = currentTimeMillis();". It looks
like the code is inconsistent on whether lastSyncedAt is a relative
monotonic clock or absolute epoch time.
Yes, I think so, I am going to create a defect regarding it.
>> I've also attempted to search for all RegisteredSignal objects in the
heap, and found 15 of them, but none of them is related to the
read-hotness-tracker:1 thread.
It is very strange because the stack trace for this thread waits within the
object and read-hotness-tracker:1 thread has active "WaitQueue.Signal
signal" local variable..
What tool do you use for the heap dump inspection? I use Eclipse Memory
Analyzer. It has "threads_overview" view and you can check the values for
local variables within a thread stack.
Like this:
[image: image.png]
Regards,
Dmitry
On Fri, 8 Nov 2024 at 11:01, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Hi Dmitry,
>
> Thank you for helping me.
>
> You are right, that's a monotonic clock. I was definitely wrongly
> interpreting its value. I got misled by this line of code
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
> "protected volatile long lastSyncedAt = currentTimeMillis();". It looks
> like the code is inconsistent on whether lastSyncedAt is a relative
> monotonic clock or absolute epoch time.
>
> Please see the attached cassandra.yaml with sensitive information masked
> out with asterisks.
>
> There's no ERROR level log in the system.log other than some guardrail
> violations (items_per_collection, if it matters) and random and rare
> network issues such as connection timeout in the few days run up to and
> during the event.
>
> We have the stdout and stderr captured in the systemd journal, and they
> have been quiet in during that period as well, not a single line got logged.
>
> There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
> objects in the heap dump. Of these, 194 has cdcState = CONTAINS
> lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two has
> cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset = 27007818
> and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset = 20
> respectively.
>
> The CDC index files have been deleted by the CDC consumer, and we don't
> keep a copy of these files elsewhere, so I cannot check the offsets in them
> unfortunately.
>
> Regarding the WaitQueue, I found the queue to be empty in the heap dump,
> which means there's no RegisteredSignal objects in the queue at all. The
> queue is a ConcurrentLinkedQueue object, which has a head and a tail. The
> head has item=null and next=null, and the tail has item=null and next=head.
> I've also attempted to search for all RegisteredSignal objects in the heap,
> and found 15 of them, but none of them is related to the
> read-hotness-tracker:1 thread. They are for the following threads:
>
> - MemtablePostFlush:1
> - AntiEntropyStage:1
> - MemtableFlushWriter:910
> - COMMIT-LOG-ALLOCATOR
> - NativePoolCleaner
> - MemtableFlushWriter:911
> - Native-Transport-Requests-135
> - Native-Transport-Requests-139
> - Native-Transport-Requests-132
> - Native-Transport-Requests-143
> - Native-Transport-Requests-129
> - Native-Transport-Requests-140
> - Native-Transport-Requests-153
> - Native-Transport-Requests-141
> - Native-Transport-Requests-137
>
> This looks rather suspicious to me, as the read-hotness-tracker:1 thread
> is blocked and waiting for a non-existence registered signal.
>
> Cheers,
> Bowen
> On 07/11/2024 21:30, Dmitry Konstantinov wrote:
>
> Hi Bowen,
>
> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
> where long pollStarted = clock.now();
> the logic uses clock from: SyncRunnable sync = new
> SyncRunnable(preciseTime)
> preciseTime by default is SystemClock
> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
> System.nanoTime()
> so, the field is populated from System.nanoTime() which returns a relative
> timestamp, which cannot be treated as an absolute time like linux epoch (it
> is calculated like this:
> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
> )
>
> Can you share your cassandra.yaml file? (with removed IPs, paths and other
> security sensitive things, of course)
> Do you have any exceptions in your system.log related to commit log?
> Do you have JVM stdout/stderr captured to a file? Does it have any
> exceptions? (which can be present there if Cassandra does not catch them
> properly by itself)
>
> ---
> How many CommitLogSegment objects do you have in your heap dump?
> What are values for the following fields of CommitLogSegment objects?
> lastSyncedOffset
> lastMarkerOffset
> cdcState
>
> Do you have CDC index files written by
> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
> If you have them - what are the last offsets written there?
>
> Regards,
> Dmitry
>
> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> The syncComplete is an instance of WaitQueue, and the syncComplete.queue
>> is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty.
>> There is no item in the queue's linked list, only the head and tail nodes,
>> each has item=null.
>>
>> The usage of the WaitQueue within the AbstractCommitLogService looks
>> correct to me:
>>
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly();
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>>
>> This matches the example given in the WaitQueue's documentation (which is
>> definitely missing a pair of curly brackets and has bad indentation):
>>
>> while (!conditionMet())
>> Signal s = q.register();
>> if (!conditionMet()) // or, perhaps more correctly,
>> !conditionChanged()
>> s.await();
>> else
>> s.cancel();
>>
>> So, why is the queue empty while the thread is blocked on waiting for the
>> signal?
>>
>> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
>> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
>> about 5 years ago.
>>
>> Is this suppose to be a wall clock time in milliseconds since 1 January
>> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
>> context, and I'm just wrongly interpreting its value as a wall clock time?
>>
>>
>> I've found that the syncComplete.queue is empty, meaning the WaitQueue
>> object believes that there's nothing waiting for the signal, yet the
>> "read-hotness-tracker:1" thread is clearly waiting for it.
>> On 06/11/2024 13:49, Bowen Song via user wrote:
>>
>> I think I'm getting really close now. This seems to have something to do
>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> <------ here
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> <------ and here
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>> at
>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>> local variable:
>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>> local variable:
>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> local variable:
>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>> at java.lang.Thread.run(Thread.java:829)
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>
>>
>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>
>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>> throws RequestExecutionException
>> {
>> OpOrder.Group group = null;
>> try
>> {
>> group = Keyspace.writeOrder.start(); // <--- here
>>
>> // write the mutation to the commitlog and memtables
>> CommitLogPosition position = null;
>> if (makeDurable)
>> {
>> position = addToCommitLog(mutation); // <--- and here
>> }
>> return new CassandraWriteContext(group, position);
>> }
>> catch (Throwable t)
>> {
>> if (group != null)
>> {
>> group.close();
>> }
>> throw t;
>> }
>> }
>>
>> The Keyspace.writeOrder.start() method call blocks the
>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>
>> void awaitSyncAt(long syncTime, Context context)
>> {
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly(); // <--- here
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> }
>>
>> This then got stuck on the signal.awaitUninterruptibly()
>>
>> Now I know what is blocking the memtable flushing, but I haven't been
>> able to figure out is why it got stuck on waiting for that signal.
>>
>> I would appreciate it if anyone can offer some insight here.
>>
>>
>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go down the
>> rabbit hole tomorrow. Will keep you updated.
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>
>>
>>
>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>
>> Writes on this node starts to timeout and fail. But if left untouched,
>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that both of the
>> MemtableFlushWriter threads are stuck on line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking progress,
>> permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for them all to
>> complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the commitLogUpperBound to be
>> set, and so that waiters
>> // on this task can rely on all prior flushes being
>> complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this started to
>> happen shortly after we enabled CDC on the nodes, and each time repair was
>> running. But we have not been able to reproduce this in a testing cluster,
>> and don't know what's the next step to troubleshoot this issue. So I'm
>> posting it in the mailing lists and hoping someone may know something about
>> it or point me to the right direction.
>>
>>
>> Wouldn’t be completely surprised if CDC or repair somehow has a
>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>> read commands” that have a barrier on the memtable that prevent release.
>>
>> You’ve got the heap (great, way better than most people debugging), are
>> you able to navigate through it and look for references to that memtable or
>> other things holding a barrier?
>>
>>
>>
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Dmitry,
Thank you for helping me.
You are right, that's a monotonic clock. I was definitely wrongly
interpreting its value. I got misled by this line of code
<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java#L63>:
"protected volatile long lastSyncedAt = currentTimeMillis();". It looks
like the code is inconsistent on whether lastSyncedAt is a relative
monotonic clock or absolute epoch time.
Please see the attached cassandra.yaml with sensitive information masked
out with asterisks.
There's no ERROR level log in the system.log other than some guardrail
violations (items_per_collection, if it matters) and random and rare
network issues such as connection timeout in the few days run up to and
during the event.
We have the stdout and stderr captured in the systemd journal, and they
have been quiet in during that period as well, not a single line got logged.
There's 196 CommitLogSegment (more specifically, MemoryMappedSegment)
objects in the heap dump. Of these, 194 has cdcState = CONTAINS
lastSyncedOffset = 33554432 lastMarkerOffset = 33554432. The other two
has cdcState = CONTAINS lastSyncedOffset = 27007818 lastMarkerOffset =
27007818 and cdcState = PERMITTED lastSyncedOffset = 20 lastMarkerOffset
= 20 respectively.
The CDC index files have been deleted by the CDC consumer, and we don't
keep a copy of these files elsewhere, so I cannot check the offsets in
them unfortunately.
Regarding the WaitQueue, I found the queue to be empty in the heap dump,
which means there's no RegisteredSignal objects in the queue at all. The
queue is a ConcurrentLinkedQueue object, which has a head and a tail.
The head has item=null and next=null, and the tail has item=null and
next=head. I've also attempted to search for all RegisteredSignal
objects in the heap, and found 15 of them, but none of them is related
to the read-hotness-tracker:1 thread. They are for the following threads:
* MemtablePostFlush:1
* AntiEntropyStage:1
* MemtableFlushWriter:910
* COMMIT-LOG-ALLOCATOR
* NativePoolCleaner
* MemtableFlushWriter:911
* Native-Transport-Requests-135
* Native-Transport-Requests-139
* Native-Transport-Requests-132
* Native-Transport-Requests-143
* Native-Transport-Requests-129
* Native-Transport-Requests-140
* Native-Transport-Requests-153
* Native-Transport-Requests-141
* Native-Transport-Requests-137
This looks rather suspicious to me, as the read-hotness-tracker:1 thread
is blocked and waiting for a non-existence registered signal.
Cheers,
Bowen
On 07/11/2024 21:30, Dmitry Konstantinov wrote:
> Hi Bowen,
>
> lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
> where long pollStarted = clock.now();
> the logic uses clock from: SyncRunnable sync = new
> SyncRunnable(preciseTime)
> preciseTime by default is SystemClock
> clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
> System.nanoTime()
> so, the field is populated from System.nanoTime() which returns a
> relative timestamp, which cannot be treated as an absolute time like
> linux epoch (it is calculated like this:
> https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
> )
>
> Can you share your cassandra.yaml file? (with removed IPs, paths and
> other security sensitive things, of course)
> Do you have any exceptions in your system.log related to commit log?
> Do you have JVM stdout/stderr captured to a file? Does it have any
> exceptions? (which can be present there if Cassandra does not catch
> them properly by itself)
>
> ---
> How many CommitLogSegment objects do you have in your heap dump?
> What are values for the following fields of CommitLogSegment objects?
> lastSyncedOffset
> lastMarkerOffset
> cdcState
>
> Do you have CDC index files written by
> org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
> If you have them - what are the last offsets written there?
>
> Regards,
> Dmitry
>
> On Thu, 7 Nov 2024 at 16:56, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> The syncComplete is an instance of WaitQueue, and the
> syncComplete.queue is an instance of ConcurrentLinkedQueue.
> Surprisingly, the queue is empty. There is no item in the queue's
> linked list, only the head and tail nodes, each has item=null.
>
> The usage of the WaitQueue within the AbstractCommitLogService
> looks correct to me:
>
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) :
> syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly();
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
>
> This matches the example given in the WaitQueue's documentation
> (which is definitely missing a pair of curly brackets and has bad
> indentation):
>
> while (!conditionMet())
> Signal s = q.register();
> if (!conditionMet()) // or, perhaps more correctly,
> !conditionChanged()
> s.await();
> else
> s.cancel();
>
> So, why is the queue empty while the thread is blocked on waiting
> for the signal?
>
> Further digging showed that PeriodicCommitLogService.lastSyncedAt
> = 1573038452910796, which translates to 6 November 2019
> 11:07:32.910796, about 5 years ago.
>
> Is this suppose to be a wall clock time in milliseconds since 1
> January 1970 00:00? Or is it a monotonic clock time that has no
> meaning out of the context, and I'm just wrongly interpreting its
> value as a wall clock time?
>
>
> I've found that the syncComplete.queue is empty, meaning the
> WaitQueue object believes that there's nothing waiting for the
> signal, yet the "read-hotness-tracker:1" thread is clearly waiting
> for it.
>
> On 06/11/2024 13:49, Bowen Song via user wrote:
>>
>> I think I'm getting really close now. This seems to have
>> something to do with the "read-hotness-tracker:1" thread. The
>> thread dump is:
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> <------ here
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> <------ and here
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at
>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable:
>> org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>> at
>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>> local variable:
>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>> local variable:
>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> local variable:
>> java.util.concurrent.Executors$RunnableAdapter#934
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> local variable:
>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>> local variable:
>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalThread#7
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> local variable:
>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>> at java.lang.Thread.run(Thread.java:829)
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalThread#7
>>
>>
>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>
>> public WriteContext beginWrite(Mutation mutation, boolean
>> makeDurable) throws RequestExecutionException
>> {
>> OpOrder.Group group = null;
>> try
>> {
>> group = Keyspace.writeOrder.start(); // <--- here
>>
>> // write the mutation to the commitlog and memtables
>> CommitLogPosition position = null;
>> if (makeDurable)
>> {
>> position = addToCommitLog(mutation); // <--- and here
>> }
>> return new CassandraWriteContext(group, position);
>> }
>> catch (Throwable t)
>> {
>> if (group != null)
>> {
>> group.close();
>> }
>> throw t;
>> }
>> }
>>
>> The Keyspace.writeOrder.start() method call blocks the
>> MemtableFlushWriter thread, and then it calls
>> addToCommitLog(mutation) which indirectly calls
>> AbstractCommitLogService.awaitSyncAt()
>>
>> void awaitSyncAt(long syncTime, Context context)
>> {
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) :
>> syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly(); // <--- here
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> }
>>
>> This then got stuck on the signal.awaitUninterruptibly()
>>
>> Now I know what is blocking the memtable flushing, but I haven't
>> been able to figure out is why it got stuck on waiting for that
>> signal.
>>
>> I would appreciate it if anyone can offer some insight here.
>>
>>
>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>
>>> I will give it a try and see what I can find. I plan to go down
>>> the rabbit hole tomorrow. Will keep you updated.
>>>
>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>
>>>>
>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>> <us...@cassandra.apache.org> <ma...@cassandra.apache.org>
>>>>> wrote:
>>>>>
>>>>> Writes on this node starts to timeout and fail. But if left
>>>>> untouched, it's only gonna get worse, and eventually lead to
>>>>> JVM OOM and crash.
>>>>>
>>>>> By inspecting the heap dump created at OOM, we can see that
>>>>> both of the MemtableFlushWriter threads are stuck on line 1190
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>> in the ColumnFamilyStore.java:
>>>>>
>>>>> // mark writes older than the barrier as blocking
>>>>> progress, permitting them to exceed our memory limit
>>>>> // if they are stuck waiting on it, then wait for
>>>>> them all to complete
>>>>> writeBarrier.markBlocking();
>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>
>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>> in the same file.
>>>>>
>>>>> try
>>>>> {
>>>>> // we wait on the latch for the
>>>>> commitLogUpperBound to be set, and so that waiters
>>>>> // on this task can rely on all prior flushes
>>>>> being complete
>>>>> latch.await(); // <----------- stuck here
>>>>> }
>>>>> Our top suspect is CDC interacting with repair, since this
>>>>> started to happen shortly after we enabled CDC on the nodes,
>>>>> and each time repair was running. But we have not been able to
>>>>> reproduce this in a testing cluster, and don't know what's the
>>>>> next step to troubleshoot this issue. So I'm posting it in the
>>>>> mailing lists and hoping someone may know something about it
>>>>> or point me to the right direction.
>>>>>
>>>>
>>>> Wouldn’t be completely surprised if CDC or repair somehow has
>>>> a barrier, I’ve also seen similar behavior pre-3.0 with “very
>>>> long running read commands” that have a barrier on the memtable
>>>> that prevent release.
>>>>
>>>> You’ve got the heap (great, way better than most people
>>>> debugging), are you able to navigate through it and look for
>>>> references to that memtable or other things holding a barrier?
>>>>
>>>>
>>>>
>>>>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Hi Bowen,
lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted;
where long pollStarted = clock.now();
the logic uses clock from: SyncRunnable sync = new SyncRunnable(preciseTime)
preciseTime by default is SystemClock
clock.now() -> org.apache.cassandra.utils.Clock.Default#nanoTime ->
System.nanoTime()
so, the field is populated from System.nanoTime() which returns a relative
timestamp, which cannot be treated as an absolute time like linux epoch (it
is calculated like this:
https://github.com/openjdk/jdk/blob/jdk-11%2B20/src/hotspot/os/linux/os_linux.cpp#L1301
)
Can you share your cassandra.yaml file? (with removed IPs, paths and other
security sensitive things, of course)
Do you have any exceptions in your system.log related to commit log?
Do you have JVM stdout/stderr captured to a file? Does it have any
exceptions? (which can be present there if Cassandra does not catch them
properly by itself)
---
How many CommitLogSegment objects do you have in your heap dump?
What are values for the following fields of CommitLogSegment objects?
lastSyncedOffset
lastMarkerOffset
cdcState
Do you have CDC index files written by
org.apache.cassandra.db.commitlog.CommitLogSegment#writeCDCIndexFile ?
If you have them - what are the last offsets written there?
Regards,
Dmitry
On Thu, 7 Nov 2024 at 16:56, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> The syncComplete is an instance of WaitQueue, and the syncComplete.queue
> is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty.
> There is no item in the queue's linked list, only the head and tail nodes,
> each has item=null.
>
> The usage of the WaitQueue within the AbstractCommitLogService looks
> correct to me:
>
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) : syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly();
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
>
> This matches the example given in the WaitQueue's documentation (which is
> definitely missing a pair of curly brackets and has bad indentation):
>
> while (!conditionMet())
> Signal s = q.register();
> if (!conditionMet()) // or, perhaps more correctly,
> !conditionChanged()
> s.await();
> else
> s.cancel();
>
> So, why is the queue empty while the thread is blocked on waiting for the
> signal?
>
> Further digging showed that PeriodicCommitLogService.lastSyncedAt =
> 1573038452910796, which translates to 6 November 2019 11:07:32.910796,
> about 5 years ago.
>
> Is this suppose to be a wall clock time in milliseconds since 1 January
> 1970 00:00? Or is it a monotonic clock time that has no meaning out of the
> context, and I'm just wrongly interpreting its value as a wall clock time?
>
>
> I've found that the syncComplete.queue is empty, meaning the WaitQueue
> object believes that there's nothing waiting for the signal, yet the
> "read-hotness-tracker:1" thread is clearly waiting for it.
> On 06/11/2024 13:49, Bowen Song via user wrote:
>
> I think I'm getting really close now. This seems to have something to do
> with the "read-hotness-tracker:1" thread. The thread dump is:
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> <------ here
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> <------ and here
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable: org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
> local variable:
> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
> local variable:
> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> local variable: java.util.concurrent.Executors$RunnableAdapter#934
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
> at java.lang.Thread.run(Thread.java:829)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>
>
> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>
> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
> throws RequestExecutionException
> {
> OpOrder.Group group = null;
> try
> {
> group = Keyspace.writeOrder.start(); // <--- here
>
> // write the mutation to the commitlog and memtables
> CommitLogPosition position = null;
> if (makeDurable)
> {
> position = addToCommitLog(mutation); // <--- and here
> }
> return new CassandraWriteContext(group, position);
> }
> catch (Throwable t)
> {
> if (group != null)
> {
> group.close();
> }
> throw t;
> }
> }
>
> The Keyspace.writeOrder.start() method call blocks the
> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
> indirectly calls AbstractCommitLogService.awaitSyncAt()
>
> void awaitSyncAt(long syncTime, Context context)
> {
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) : syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly(); // <--- here
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
> }
>
> This then got stuck on the signal.awaitUninterruptibly()
>
> Now I know what is blocking the memtable flushing, but I haven't been able
> to figure out is why it got stuck on waiting for that signal.
>
> I would appreciate it if anyone can offer some insight here.
>
>
> On 05/11/2024 17:48, Bowen Song via user wrote:
>
> I will give it a try and see what I can find. I plan to go down the rabbit
> hole tomorrow. Will keep you updated.
> On 05/11/2024 17:34, Jeff Jirsa wrote:
>
>
>
> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>
> Writes on this node starts to timeout and fail. But if left untouched,
> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of the
> MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress,
> permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all to
> complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to be
> set, and so that waiters
> // on this task can rely on all prior flushes being
> complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to
> happen shortly after we enabled CDC on the nodes, and each time repair was
> running. But we have not been able to reproduce this in a testing cluster,
> and don't know what's the next step to troubleshoot this issue. So I'm
> posting it in the mailing lists and hoping someone may know something about
> it or point me to the right direction.
>
>
> Wouldn’t be completely surprised if CDC or repair somehow has a barrier,
> I’ve also seen similar behavior pre-3.0 with “very long running read
> commands” that have a barrier on the memtable that prevent release.
>
> You’ve got the heap (great, way better than most people debugging), are
> you able to navigate through it and look for references to that memtable or
> other things holding a barrier?
>
>
>
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
The syncComplete is an instance of WaitQueue, and the syncComplete.queue
is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is
empty. There is no item in the queue's linked list, only the head and
tail nodes, each has item=null.
The usage of the WaitQueue within the AbstractCommitLogService looks
correct to me:
do
{
WaitQueue.Signal signal = context != null ?
syncComplete.register(context, Context::stop) : syncComplete.register();
if (lastSyncedAt < syncTime)
signal.awaitUninterruptibly();
else
signal.cancel();
}
while (lastSyncedAt < syncTime);
This matches the example given in the WaitQueue's documentation (which
is definitely missing a pair of curly brackets and has bad indentation):
while (!conditionMet())
Signal s = q.register();
if (!conditionMet()) // or, perhaps more correctly,
!conditionChanged()
s.await();
else
s.cancel();
So, why is the queue empty while the thread is blocked on waiting for
the signal?
Further digging showed that PeriodicCommitLogService.lastSyncedAt =
1573038452910796, which translates to 6 November 2019 11:07:32.910796,
about 5 years ago.
Is this suppose to be a wall clock time in milliseconds since 1 January
1970 00:00? Or is it a monotonic clock time that has no meaning out of
the context, and I'm just wrongly interpreting its value as a wall clock
time?
I've found that the syncComplete.queue is empty, meaning the WaitQueue
object believes that there's nothing waiting for the signal, yet the
"read-hotness-tracker:1" thread is clearly waiting for it.
On 06/11/2024 13:49, Bowen Song via user wrote:
>
> I think I'm getting really close now. This seems to have something to
> do with the "read-hotness-tracker:1" thread. The thread dump is:
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> <------ here
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable:
> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> <------ and here
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable: org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
> local variable:
> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
> local variable:
> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> local variable: java.util.concurrent.Executors$RunnableAdapter#934
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
> at java.lang.Thread.run(Thread.java:829)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>
>
> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>
> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
> throws RequestExecutionException
> {
> OpOrder.Group group = null;
> try
> {
> group = Keyspace.writeOrder.start(); // <--- here
>
> // write the mutation to the commitlog and memtables
> CommitLogPosition position = null;
> if (makeDurable)
> {
> position = addToCommitLog(mutation); // <--- and here
> }
> return new CassandraWriteContext(group, position);
> }
> catch (Throwable t)
> {
> if (group != null)
> {
> group.close();
> }
> throw t;
> }
> }
>
> The Keyspace.writeOrder.start() method call blocks the
> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation)
> which indirectly calls AbstractCommitLogService.awaitSyncAt()
>
> void awaitSyncAt(long syncTime, Context context)
> {
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) : syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly(); // <--- here
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
> }
>
> This then got stuck on the signal.awaitUninterruptibly()
>
> Now I know what is blocking the memtable flushing, but I haven't been
> able to figure out is why it got stuck on waiting for that signal.
>
> I would appreciate it if anyone can offer some insight here.
>
>
> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go down the
>> rabbit hole tomorrow. Will keep you updated.
>>
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>> <us...@cassandra.apache.org> wrote:
>>>>
>>>> Writes on this node starts to timeout and fail. But if left
>>>> untouched, it's only gonna get worse, and eventually lead to JVM
>>>> OOM and crash.
>>>>
>>>> By inspecting the heap dump created at OOM, we can see that both of
>>>> the MemtableFlushWriter threads are stuck on line 1190
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>> in the ColumnFamilyStore.java:
>>>>
>>>> // mark writes older than the barrier as blocking
>>>> progress, permitting them to exceed our memory limit
>>>> // if they are stuck waiting on it, then wait for them
>>>> all to complete
>>>> writeBarrier.markBlocking();
>>>> writeBarrier.await(); // <----------- stuck here
>>>>
>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>> in the same file.
>>>>
>>>> try
>>>> {
>>>> // we wait on the latch for the commitLogUpperBound
>>>> to be set, and so that waiters
>>>> // on this task can rely on all prior flushes being
>>>> complete
>>>> latch.await(); // <----------- stuck here
>>>> }
>>>> Our top suspect is CDC interacting with repair, since this started
>>>> to happen shortly after we enabled CDC on the nodes, and each time
>>>> repair was running. But we have not been able to reproduce this in
>>>> a testing cluster, and don't know what's the next step to
>>>> troubleshoot this issue. So I'm posting it in the mailing lists and
>>>> hoping someone may know something about it or point me to the right
>>>> direction.
>>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long
>>> running read commands” that have a barrier on the memtable that
>>> prevent release.
>>>
>>> You’ve got the heap (great, way better than most people debugging),
>>> are you able to navigate through it and look for references to that
>>> memtable or other things holding a barrier?
>>>
>>>
>>>
>>>
Re: Unexplained stuck memtable flush
Posted by Jon Haddad <jo...@rustyrazorblade.com>.
I think this is the correct explanation. It's very similar to
CASSANDRA-19576, where compaction is unable to finish because we can't
insert into compaction history.
Really good analysis, Jaydeep.
Jon
On Sun, Nov 10, 2024 at 1:51 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Upon further studying the thread dump, I think there is a deadlock between
> CompactionExecutor (Thread1) & MemtableReclaimMemory (Thread2) in the stack
> trace I have mentioned in my email below.
>
> Please find the deadlock details here:
>
> 1. *Thread1:* It has invoked *readOrdering.start()* on *baseCf* and
> *indexCf* as part of the "*ReadExecutionController controller =
> cmd.executionController()*" at a line no
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L921>
> 2. *Thread1:* It wants to apply the change locally to Memtable as part
> of indexer.insertRow(row) at a line no
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L962>
> 3. *Thread1:* There is not enough space available in Memtable; hence,
> Thread1 is blocked on space to become available
> 4. *Thread2:* It wants to free up the space in Memtable, but before it
> does that, it is waiting
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1350>
> on all the *readOrdering*for a given Cf to finish.
> 5. *Thread2**:* But Thread2's indefinite wait cannot succeed because
> Thread1 has already invoked "*.start()*" on it, so unless and until
> Thread1 calls *".close()"* Threa2 cannot proceed. And here is the
> *deadlock*.
>
> Here are the stack traces for Thread1 and Thread2 to prove the deadlock,
> as mentioned above.
>
> Thread1 (*CompactionExecutor*)
>
> "CompactionExecutor:5" - Thread t@118
> java.lang.Thread.State: WAITING
> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
> at java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289) at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282) at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131) at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235) at app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:195) at app//org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89) at app//org.apache.cassandra.utils.memory.MemtableBufferAllocator$1.allocate(MemtableBufferAllocator.java:40) at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:77) at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:63) at app//org.apache.cassandra.db.Clustering.clone(Clustering.java:54) at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:52) at app//org.apache.cassandra.db.rows.BTreeRow.clone(BTreeRow.java:517) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:375) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:351) at app//org.apache.cassandra.utils.btree.BTree.updateLeaves(BTree.java:461) at app//org.apache.cassandra.utils.btree.BTree.update(BTree.java:368) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:146) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:190) at app//org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:134) at app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1431) at app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40) at app//org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:529) at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCell(CassandraIndex.java:445) at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCells(CassandraIndex.java:437) at app//org.apache.cassandra.index.internal.CassandraIndex$1.insertRow(CassandraIndex.java:387) at app//org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$25(SecondaryIndexManager.java:985) at app//org.apache.cassandra.index.SecondaryIndexManager$$Lambda$1396/0x00007f00861e2cb0.accept(Unknown Source) at java.base@11.0.20.1/java.lang.Iterable.forEach(Iterable.java:75)
> at app//org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:985) at app//org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:78) at app//org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1888) at app//org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141) at app//org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at app//org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
>
>
>
> Thread2 (*MemtableReclaimMemory*)
>
> "MemtableReclaimMemory:1" - Thread t@56
> java.lang.Thread.State: WAITING
> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
> at java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289) at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282) at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186) at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259) at app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267) at app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425) at app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357) at app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133) at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
>
> Jaydeep
>
> On Wed, Nov 6, 2024 at 8:29 AM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> We have also seen this issue a few times in our production (4.1). My
>> teammates added a thread dump here
>> <https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>
>> .
>>
>> One of my theories is that the MemtableReclaimMemory thread has been
>> unable to make progress:
>>
>> "MemtableReclaimMemory:1" - Thread t@56
>>
>> java.lang.Thread.State: WAITING
>>
>> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native
>> Method)
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
>>
>> at
>> app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
>>
>> at
>> app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>
>> at
>> app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>
>> at
>> app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>
>> at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
>>
>>
>>
>> One of the possible bugs I could see is a line
>> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
>> * indexController = new ReadExecutionController(command,
>> indexCfs.readOrdering.start(), indexCfs.metadata(), null, null,
>> NO_SAMPLING, false);*
>> If "*indexCfs.readOrdering.start()*" succeeded but the constructor "*new
>> ReadExecutionController*", then we are not closing "
>> *indexCfs.readOrdering*", which means it will remain
>> inaccurate forever.
>>
>>
>> Jaydeep
>>
>> On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> I think I'm getting really close now. This seems to have something to do
>>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>>
>>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>>> at jdk.internal.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>>> at
>>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>>> <------ here
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> local variable: com.codahale.metrics.Timer$Context#2086795
>>> at
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>>> local variable:
>>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>>> local variable:
>>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>>> local variable:
>>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>>> at
>>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>>> <------ and here
>>> local variable:
>>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>>> local variable: org.apache.cassandra.db.Keyspace#8
>>> local variable: org.apache.cassandra.db.Mutation#54491
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>>> at
>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>>> at
>>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>>> local variable: java.lang.String#42806
>>> local variable: java.lang.String#47659
>>> local variable:
>>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>>> at
>>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>>> local variable:
>>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>>> local variable:
>>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> local variable:
>>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>> local variable:
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>>> at
>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>>> at java.lang.Thread.run(Thread.java:829)
>>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>>
>>>
>>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>>
>>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>>> throws RequestExecutionException
>>> {
>>> OpOrder.Group group = null;
>>> try
>>> {
>>> group = Keyspace.writeOrder.start(); // <--- here
>>>
>>> // write the mutation to the commitlog and memtables
>>> CommitLogPosition position = null;
>>> if (makeDurable)
>>> {
>>> position = addToCommitLog(mutation); // <--- and here
>>> }
>>> return new CassandraWriteContext(group, position);
>>> }
>>> catch (Throwable t)
>>> {
>>> if (group != null)
>>> {
>>> group.close();
>>> }
>>> throw t;
>>> }
>>> }
>>>
>>> The Keyspace.writeOrder.start() method call blocks the
>>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>>
>>> void awaitSyncAt(long syncTime, Context context)
>>> {
>>> do
>>> {
>>> WaitQueue.Signal signal = context != null ?
>>> syncComplete.register(context, Context::stop) : syncComplete.register();
>>> if (lastSyncedAt < syncTime)
>>> signal.awaitUninterruptibly(); // <--- here
>>> else
>>> signal.cancel();
>>> }
>>> while (lastSyncedAt < syncTime);
>>> }
>>>
>>> This then got stuck on the signal.awaitUninterruptibly()
>>>
>>> Now I know what is blocking the memtable flushing, but I haven't been
>>> able to figure out is why it got stuck on waiting for that signal.
>>>
>>> I would appreciate it if anyone can offer some insight here.
>>>
>>>
>>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>
>>> I will give it a try and see what I can find. I plan to go down the
>>> rabbit hole tomorrow. Will keep you updated.
>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>
>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>>
>>> Writes on this node starts to timeout and fail. But if left untouched,
>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can see that both of the
>>> MemtableFlushWriter threads are stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as blocking progress,
>>> permitting them to exceed our memory limit
>>> // if they are stuck waiting on it, then wait for them all
>>> to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <----------- stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the commitLogUpperBound to
>>> be set, and so that waiters
>>> // on this task can rely on all prior flushes being
>>> complete
>>> latch.await(); // <----------- stuck here
>>> }
>>> Our top suspect is CDC interacting with repair, since this started to
>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>> running. But we have not been able to reproduce this in a testing cluster,
>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>> posting it in the mailing lists and hoping someone may know something about
>>> it or point me to the right direction.
>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow has a
>>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>>> read commands” that have a barrier on the memtable that prevent release.
>>>
>>> You’ve got the heap (great, way better than most people debugging), are
>>> you able to navigate through it and look for references to that memtable or
>>> other things holding a barrier?
>>>
>>>
>>>
>>>
>>>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Jaydeep,
Great work! I believe you are spot on. You should create a Jira ticket
detail the findings.
Cheers,
Bowen
On 10/11/2024 21:49, Jaydeep Chovatia wrote:
> Upon further studying the thread dump, I think there is a deadlock
> between CompactionExecutor (Thread1) & MemtableReclaimMemory (Thread2)
> in the stack trace I have mentioned in my email below.
>
> Please find the deadlock details here:
>
> 1. *Thread1:*It has
> invoked/readOrdering.start()/on/baseCf/and/indexCf/as part of the
> "/ReadExecutionController controller = cmd.executionController()/"
> at aline no
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L921>
> 2. *Thread1:*It wants to apply the change locally to Memtable as part
> of indexer.insertRow(row) at aline no
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L962>
> 3. *Thread1:*There is not enough space available in Memtable; hence,
> Thread1 is blocked on space to become available
> 4. *Thread2:*It wants to free up the space in Memtable, but before it
> does that, it iswaiting
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1350>on
> all the/readOrdering/for a given Cf to finish.
> 5. *Thread2**:*But Thread2's indefinite wait cannot succeed because
> Thread1 has already invoked "/.start()/" on it, so unless and
> until Thread1 calls/".close()"/Threa2 cannot proceed. And here is
> the*deadlock*.
>
> Here are the stack traces for Thread1 and Thread2 to prove the
> deadlock, as mentioned above.
>
> Thread1 (/CompactionExecutor/)
>
> "CompactionExecutor:5" -Thread t@118
> java.lang.Thread.State: WAITING
> atjava.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native <http://java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native> Method)
> atjava.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) <http://java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)>
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131)
> at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235)
> at app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:195)
> at app//org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
> at app//org.apache.cassandra.utils.memory.MemtableBufferAllocator$1.allocate(MemtableBufferAllocator.java:40)
> at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:77)
> at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:63)
> at app//org.apache.cassandra.db.Clustering.clone(Clustering.java:54) at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:52)
> at app//org.apache.cassandra.db.rows.BTreeRow.clone(BTreeRow.java:517) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:375)
> at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:351)
> at app//org.apache.cassandra.utils.btree.BTree.updateLeaves(BTree.java:461) at app//org.apache.cassandra.utils.btree.BTree.update(BTree.java:368) at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:146)
> at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:190)
> at app//org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:134)
> at app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1431)
> at app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
> at app//org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:529)
> at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCell(CassandraIndex.java:445)
> at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCells(CassandraIndex.java:437)
> at app//org.apache.cassandra.index.internal.CassandraIndex$1.insertRow(CassandraIndex.java:387)
> at app//org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$25(SecondaryIndexManager.java:985)
> at app//org.apache.cassandra.index.SecondaryIndexManager$$Lambda$1396/0x00007f00861e2cb0.accept(Unknown
> Source) atjava.base@11.0.20.1/java.lang.Iterable.forEach(Iterable.java:75) <http://java.base@11.0.20.1/java.lang.Iterable.forEach(Iterable.java:75)>
> at app//org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:985)
> at app//org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:78)
> at app//org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1888)
> at app//org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141) at app//org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at app//org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) atjava.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
> atjava.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
> at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> atjava.base@11.0.20.1/java.lang <http://java.base@11.0.20.1/java.lang>.Thread.run(Thread.java:829)
>
> Thread2 (/MemtableReclaimMemory/)
>
> "MemtableReclaimMemory:1" -Thread t@56
> java.lang.Thread.State: WAITING
> atjava.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native <http://java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native> Method)
> atjava.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) <http://java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)>
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
> at app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
> at app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
> at app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
> atjava.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
> atjava.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
> at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> atjava.base@11.0.20.1/java.lang <http://java.base@11.0.20.1/java.lang>.Thread.run(Thread.java:829)
> Jaydeep
>
> On Wed, Nov 6, 2024 at 8:29 AM Jaydeep Chovatia
> <ch...@gmail.com> wrote:
>
> We have also seen this issue a few times in our production (4.1).
> My teammates added a thread dump here
> <https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>.
>
> One of my theories is that the MemtableReclaimMemory thread has
> been unable to make progress:
>
> "MemtableReclaimMemory:1" - Thread t@56
>
> java.lang.Thread.State: WAITING
>
> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native
> <http://java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native>
> Method)
>
> at
> java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> <http://java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)>
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
>
> at
> app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
>
> at
> app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
>
> at
> app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
> <http://java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)>
>
>
> One of the possible bugs I could see is a line
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
> / indexController = new
> ReadExecutionController(command, indexCfs.readOrdering.start(),
> indexCfs.metadata(), null, null, NO_SAMPLING, false);/
> If "/indexCfs.readOrdering.start()/" succeeded but the constructor
> "/new ReadExecutionController/", then we are not closing
> "/indexCfs.readOrdering/", which means it will remain
> inaccurate forever.
>
>
> Jaydeep
>
> On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> I think I'm getting really close now. This seems to have
> something to do with the "read-hotness-tracker:1" thread. The
> thread dump is:
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> <------ here
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable:
> org.apache.cassandra.db.commitlog.CommitLog#1
> local variable:
> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> <------ and here
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at
> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable:
> org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
> local variable:
> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
> local variable:
> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> local variable:
> java.util.concurrent.Executors$RunnableAdapter#934
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#7
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#7
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#7
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable:
> io.netty.util.concurrent.FastThreadLocalRunnable#7
> at java.lang.Thread.run(Thread.java:829)
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#7
>
>
> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>
> public WriteContext beginWrite(Mutation mutation, boolean
> makeDurable) throws RequestExecutionException
> {
> OpOrder.Group group = null;
> try
> {
> group = Keyspace.writeOrder.start(); // <--- here
>
> // write the mutation to the commitlog and memtables
> CommitLogPosition position = null;
> if (makeDurable)
> {
> position = addToCommitLog(mutation); // <--- and here
> }
> return new CassandraWriteContext(group, position);
> }
> catch (Throwable t)
> {
> if (group != null)
> {
> group.close();
> }
> throw t;
> }
> }
>
> The Keyspace.writeOrder.start() method call blocks the
> MemtableFlushWriter thread, and then it calls
> addToCommitLog(mutation) which indirectly calls
> AbstractCommitLogService.awaitSyncAt()
>
> void awaitSyncAt(long syncTime, Context context)
> {
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) :
> syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly(); // <--- here
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
> }
>
> This then got stuck on the signal.awaitUninterruptibly()
>
> Now I know what is blocking the memtable flushing, but I
> haven't been able to figure out is why it got stuck on waiting
> for that signal.
>
> I would appreciate it if anyone can offer some insight here.
>
>
> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go
>> down the rabbit hole tomorrow. Will keep you updated.
>>
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>> <us...@cassandra.apache.org>
>>>> <ma...@cassandra.apache.org> wrote:
>>>>
>>>> Writes on this node starts to timeout and fail. But if left
>>>> untouched, it's only gonna get worse, and eventually lead
>>>> to JVM OOM and crash.
>>>>
>>>> By inspecting the heap dump created at OOM, we can see that
>>>> both of the MemtableFlushWriter threads are stuck on line
>>>> 1190
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>> in the ColumnFamilyStore.java:
>>>>
>>>> // mark writes older than the barrier as
>>>> blocking progress, permitting them to exceed our memory limit
>>>> // if they are stuck waiting on it, then wait
>>>> for them all to complete
>>>> writeBarrier.markBlocking();
>>>> writeBarrier.await(); // <----------- stuck here
>>>>
>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>> in the same file.
>>>>
>>>> try
>>>> {
>>>> // we wait on the latch for the
>>>> commitLogUpperBound to be set, and so that waiters
>>>> // on this task can rely on all prior
>>>> flushes being complete
>>>> latch.await(); // <----------- stuck here
>>>> }
>>>> Our top suspect is CDC interacting with repair, since this
>>>> started to happen shortly after we enabled CDC on the
>>>> nodes, and each time repair was running. But we have not
>>>> been able to reproduce this in a testing cluster, and don't
>>>> know what's the next step to troubleshoot this issue. So
>>>> I'm posting it in the mailing lists and hoping someone may
>>>> know something about it or point me to the right direction.
>>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow
>>> has a barrier, I’ve also seen similar behavior pre-3.0 with
>>> “very long running read commands” that have a barrier on the
>>> memtable that prevent release.
>>>
>>> You’ve got the heap (great, way better than most people
>>> debugging), are you able to navigate through it and look for
>>> references to that memtable or other things holding a barrier?
>>>
>>>
>>>
>>>
Re: Unexplained stuck memtable flush
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Upon further studying the thread dump, I think there is a deadlock between
CompactionExecutor (Thread1) & MemtableReclaimMemory (Thread2) in the stack
trace I have mentioned in my email below.
Please find the deadlock details here:
1. *Thread1:* It has invoked *readOrdering.start()* on *baseCf* and
*indexCf* as part of the "*ReadExecutionController controller =
cmd.executionController()*" at a line no
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L921>
2. *Thread1:* It wants to apply the change locally to Memtable as part
of indexer.insertRow(row) at a line no
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L962>
3. *Thread1:* There is not enough space available in Memtable; hence,
Thread1 is blocked on space to become available
4. *Thread2:* It wants to free up the space in Memtable, but before it
does that, it is waiting
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1350>
on all the *readOrdering*for a given Cf to finish.
5. *Thread2**:* But Thread2's indefinite wait cannot succeed because
Thread1 has already invoked "*.start()*" on it, so unless and until
Thread1 calls *".close()"* Threa2 cannot proceed. And here is the
*deadlock*.
Here are the stack traces for Thread1 and Thread2 to prove the deadlock, as
mentioned above.
Thread1 (*CompactionExecutor*)
"CompactionExecutor:5" - Thread t@118
java.lang.Thread.State: WAITING
at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131)
at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235)
at app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:195)
at app//org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
at app//org.apache.cassandra.utils.memory.MemtableBufferAllocator$1.allocate(MemtableBufferAllocator.java:40)
at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:77)
at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:63)
at app//org.apache.cassandra.db.Clustering.clone(Clustering.java:54)
at app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:52)
at app//org.apache.cassandra.db.rows.BTreeRow.clone(BTreeRow.java:517)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:375)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.insert(AtomicBTreePartition.java:351)
at app//org.apache.cassandra.utils.btree.BTree.updateLeaves(BTree.java:461)
at app//org.apache.cassandra.utils.btree.BTree.update(BTree.java:368)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:146)
at app//org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:190)
at app//org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:134)
at app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1431)
at app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
at app//org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:529)
at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCell(CassandraIndex.java:445)
at app//org.apache.cassandra.index.internal.CassandraIndex$1.indexCells(CassandraIndex.java:437)
at app//org.apache.cassandra.index.internal.CassandraIndex$1.insertRow(CassandraIndex.java:387)
at app//org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$25(SecondaryIndexManager.java:985)
at app//org.apache.cassandra.index.SecondaryIndexManager$$Lambda$1396/0x00007f00861e2cb0.accept(Unknown
Source) at
java.base@11.0.20.1/java.lang.Iterable.forEach(Iterable.java:75)
at app//org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:985)
at app//org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:78)
at app//org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1888)
at app//org.apache.cassandra.concurrent.FutureTask$3.call(FutureTask.java:141)
at app//org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at app//org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
Thread2 (*MemtableReclaimMemory*)
"MemtableReclaimMemory:1" - Thread t@56
java.lang.Thread.State: WAITING
at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
at app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
at app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
at app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
at app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
at app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
Jaydeep
On Wed, Nov 6, 2024 at 8:29 AM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> We have also seen this issue a few times in our production (4.1). My
> teammates added a thread dump here
> <https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>
> .
>
> One of my theories is that the MemtableReclaimMemory thread has been
> unable to make progress:
>
> "MemtableReclaimMemory:1" - Thread t@56
>
> java.lang.Thread.State: WAITING
>
> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native
> Method)
>
> at
> java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
>
> at
> app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
>
> at
> app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at
> app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
>
>
>
> One of the possible bugs I could see is a line
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
> * indexController = new ReadExecutionController(command,
> indexCfs.readOrdering.start(), indexCfs.metadata(), null, null,
> NO_SAMPLING, false);*
> If "*indexCfs.readOrdering.start()*" succeeded but the constructor "*new
> ReadExecutionController*", then we are not closing "
> *indexCfs.readOrdering*", which means it will remain
> inaccurate forever.
>
>
> Jaydeep
>
> On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> I think I'm getting really close now. This seems to have something to do
>> with the "read-hotness-tracker:1" thread. The thread dump is:
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> <------ here
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> <------ and here
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable: org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>> at
>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>> local variable:
>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>> local variable:
>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> local variable: java.util.concurrent.Executors$RunnableAdapter#934
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> local variable:
>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
>> at java.lang.Thread.run(Thread.java:829)
>> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>>
>>
>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>
>> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
>> throws RequestExecutionException
>> {
>> OpOrder.Group group = null;
>> try
>> {
>> group = Keyspace.writeOrder.start(); // <--- here
>>
>> // write the mutation to the commitlog and memtables
>> CommitLogPosition position = null;
>> if (makeDurable)
>> {
>> position = addToCommitLog(mutation); // <--- and here
>> }
>> return new CassandraWriteContext(group, position);
>> }
>> catch (Throwable t)
>> {
>> if (group != null)
>> {
>> group.close();
>> }
>> throw t;
>> }
>> }
>>
>> The Keyspace.writeOrder.start() method call blocks the
>> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
>> indirectly calls AbstractCommitLogService.awaitSyncAt()
>>
>> void awaitSyncAt(long syncTime, Context context)
>> {
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) : syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly(); // <--- here
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> }
>>
>> This then got stuck on the signal.awaitUninterruptibly()
>>
>> Now I know what is blocking the memtable flushing, but I haven't been
>> able to figure out is why it got stuck on waiting for that signal.
>>
>> I would appreciate it if anyone can offer some insight here.
>>
>>
>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go down the
>> rabbit hole tomorrow. Will keep you updated.
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>
>>
>>
>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>>
>> Writes on this node starts to timeout and fail. But if left untouched,
>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that both of the
>> MemtableFlushWriter threads are stuck on line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking progress,
>> permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for them all to
>> complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the commitLogUpperBound to be
>> set, and so that waiters
>> // on this task can rely on all prior flushes being
>> complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this started to
>> happen shortly after we enabled CDC on the nodes, and each time repair was
>> running. But we have not been able to reproduce this in a testing cluster,
>> and don't know what's the next step to troubleshoot this issue. So I'm
>> posting it in the mailing lists and hoping someone may know something about
>> it or point me to the right direction.
>>
>>
>> Wouldn’t be completely surprised if CDC or repair somehow has a
>> barrier, I’ve also seen similar behavior pre-3.0 with “very long running
>> read commands” that have a barrier on the memtable that prevent release.
>>
>> You’ve got the heap (great, way better than most people debugging), are
>> you able to navigate through it and look for references to that memtable or
>> other things holding a barrier?
>>
>>
>>
>>
>>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Sorry, I mistaken CassandraTableWriteHandler for
CassandraKeyspaceWriteHandler, and thought the code has changed. I was
wrong, the code has not changed, they are two different files.
Your read-hotness-tracker thread dump is:
"read-hotness-tracker:1" - Thread t@113
java.lang.Thread.State: WAITING
at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at
app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
at
app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at
app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131)
at
app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235)
at
app//org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:195)
at
app//org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
at
app//org.apache.cassandra.utils.memory.MemtableBufferAllocator$1.allocate(MemtableBufferAllocator.java:40)
at
app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:77)
at
app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:63)
at
app//org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:46)
at
app//org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:119)
at
app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1431)
at
app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
at app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:783)
at app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:616)
at app//org.apache.cassandra.db.Mutation.apply(Mutation.java:256)
at app//org.apache.cassandra.db.Mutation.apply(Mutation.java:276)
at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:696)
at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:687)
at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:488)
at
app//org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1509)
at
app//org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2176)
at
app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at
java.base@11.0.20.1/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base@11.0.20.1/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base@11.0.20.1/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
Locked ownable synchronizers:
- Locked <76304b81> (a java.util.concurrent.ThreadPoolExecutor$Worker)
It went from Keyspace.applyInternal() into
CassandraTableWriteHandler.write(), that is
after CassandraKeyspaceWriteHandler.beginWrite() has returned
successfully, and it has yet to closed the barrier. This is unlike our
situation in which the CassandraKeyspaceWriteHandler.beginWrite() got
stuck on waiting for a signal.
Base on my understanding of it, these two issues have different causes.
On 06/11/2024 18:36, Bowen Song via user wrote:
>
> I can see some similarities and some differences between your thread
> dump and ours.
>
> In your thread dump:
>
> * no MemtableFlushWriter thread
> * the MemtablePostFlush thread is idle
> * the MemtableReclaimMemory thread is waiting for a barrier,
> possibly stuck
> * the read-hotness-tracker thread is waiting for memtable memory
> allocation. I'm not sure whether it's blocking a barrier, the
> relevant part of the source code has changed between 4.1.1 and 4.1.3.
>
> In our thread dump:
>
> * two MemtableFlushWriter threads are stuck, each waiting for a barrier
> * the MemtablePostFlush thread is stuck, waiting for a latch
> * the MemtableReclaimMemory thread is idle
> * the read-hotness-tracker thread is waiting for a signal, and it's
> blocking the barrier one of the MemtableFlushWriter threads is
> waiting on
>
> It is possible that in both cases the read-hotness-tracker thread was
> blocking the barrier, which caused the MemtableFlushWriter or
> MemtableReclaimMemory thread to get stuck. It is also totally possible
> that your case has nothing to do with the read-hotness-tracker thread.
>
> I would recommend you try to find out what was blocking the barrier
> MemtableReclaimMemory thread is waiting on.
>
>
> On 06/11/2024 16:29, Jaydeep Chovatia wrote:
>> We have also seen this issue a few times in our production (4.1). My
>> teammates added a thread dump here
>> <https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>.
>>
>> One of my theories is that the MemtableReclaimMemory thread has been
>> unable to make progress:
>>
>> "MemtableReclaimMemory:1" - Thread t@56
>>
>> java.lang.Thread.State: WAITING
>>
>> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native
>> <http://java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native> Method)
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> <http://java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)>
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
>>
>> at
>> app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
>>
>> at
>> app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
>>
>> at
>> app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>
>> at
>> app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
>>
>> at
>> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
>>
>> at
>> app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>>
>> at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
>> <http://java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)>
>>
>>
>> One of the possible bugs I could see is a line
>> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
>> / indexController = new ReadExecutionController(command,
>> indexCfs.readOrdering.start(), indexCfs.metadata(), null, null,
>> NO_SAMPLING, false);/
>> If "/indexCfs.readOrdering.start()/" succeeded but the constructor
>> "/new ReadExecutionController/", then we are not closing
>> "/indexCfs.readOrdering/", which means it will remain
>> inaccurate forever.
>>
>>
>> Jaydeep
>>
>> On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> I think I'm getting really close now. This seems to have
>> something to do with the "read-hotness-tracker:1" thread. The
>> thread dump is:
>>
>> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
>> at jdk.internal.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>> local variable:
>> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
>> at
>> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
>> <------ here
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> local variable: com.codahale.metrics.Timer$Context#2086795
>> at
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
>> local variable:
>> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
>> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
>> local variable:
>> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
>> local variable:
>> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
>> at
>> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
>> <------ and here
>> local variable:
>> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
>> at
>> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
>> local variable: org.apache.cassandra.db.Keyspace#8
>> local variable: org.apache.cassandra.db.Mutation#54491
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
>> at
>> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
>> at
>> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
>> at
>> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
>> local variable: java.lang.String#42806
>> local variable: java.lang.String#47659
>> local variable:
>> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
>> local variable:
>> org.apache.cassandra.metrics.RestorableMeter#752
>> at
>> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
>> at
>> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
>> local variable:
>> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
>> local variable:
>> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> local variable:
>> java.util.concurrent.Executors$RunnableAdapter#934
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> local variable:
>> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
>> local variable:
>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalThread#7
>> local variable:
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> local variable:
>> java.util.concurrent.ThreadPoolExecutor$Worker#7
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalRunnable#7
>> at java.lang.Thread.run(Thread.java:829)
>> local variable:
>> io.netty.util.concurrent.FastThreadLocalThread#7
>>
>>
>> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>>
>> public WriteContext beginWrite(Mutation mutation, boolean
>> makeDurable) throws RequestExecutionException
>> {
>> OpOrder.Group group = null;
>> try
>> {
>> group = Keyspace.writeOrder.start(); // <--- here
>>
>> // write the mutation to the commitlog and memtables
>> CommitLogPosition position = null;
>> if (makeDurable)
>> {
>> position = addToCommitLog(mutation); // <--- and here
>> }
>> return new CassandraWriteContext(group, position);
>> }
>> catch (Throwable t)
>> {
>> if (group != null)
>> {
>> group.close();
>> }
>> throw t;
>> }
>> }
>>
>> The Keyspace.writeOrder.start() method call blocks the
>> MemtableFlushWriter thread, and then it calls
>> addToCommitLog(mutation) which indirectly calls
>> AbstractCommitLogService.awaitSyncAt()
>>
>> void awaitSyncAt(long syncTime, Context context)
>> {
>> do
>> {
>> WaitQueue.Signal signal = context != null ?
>> syncComplete.register(context, Context::stop) :
>> syncComplete.register();
>> if (lastSyncedAt < syncTime)
>> signal.awaitUninterruptibly(); // <--- here
>> else
>> signal.cancel();
>> }
>> while (lastSyncedAt < syncTime);
>> }
>>
>> This then got stuck on the signal.awaitUninterruptibly()
>>
>> Now I know what is blocking the memtable flushing, but I haven't
>> been able to figure out is why it got stuck on waiting for that
>> signal.
>>
>> I would appreciate it if anyone can offer some insight here.
>>
>>
>> On 05/11/2024 17:48, Bowen Song via user wrote:
>>>
>>> I will give it a try and see what I can find. I plan to go down
>>> the rabbit hole tomorrow. Will keep you updated.
>>>
>>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>>
>>>>
>>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>>> <us...@cassandra.apache.org> <ma...@cassandra.apache.org>
>>>>> wrote:
>>>>>
>>>>> Writes on this node starts to timeout and fail. But if left
>>>>> untouched, it's only gonna get worse, and eventually lead to
>>>>> JVM OOM and crash.
>>>>>
>>>>> By inspecting the heap dump created at OOM, we can see that
>>>>> both of the MemtableFlushWriter threads are stuck on line 1190
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>>> in the ColumnFamilyStore.java:
>>>>>
>>>>> // mark writes older than the barrier as blocking
>>>>> progress, permitting them to exceed our memory limit
>>>>> // if they are stuck waiting on it, then wait for
>>>>> them all to complete
>>>>> writeBarrier.markBlocking();
>>>>> writeBarrier.await(); // <----------- stuck here
>>>>>
>>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>>> in the same file.
>>>>>
>>>>> try
>>>>> {
>>>>> // we wait on the latch for the
>>>>> commitLogUpperBound to be set, and so that waiters
>>>>> // on this task can rely on all prior flushes
>>>>> being complete
>>>>> latch.await(); // <----------- stuck here
>>>>> }
>>>>> Our top suspect is CDC interacting with repair, since this
>>>>> started to happen shortly after we enabled CDC on the nodes,
>>>>> and each time repair was running. But we have not been able to
>>>>> reproduce this in a testing cluster, and don't know what's the
>>>>> next step to troubleshoot this issue. So I'm posting it in the
>>>>> mailing lists and hoping someone may know something about it
>>>>> or point me to the right direction.
>>>>>
>>>>
>>>> Wouldn’t be completely surprised if CDC or repair somehow has
>>>> a barrier, I’ve also seen similar behavior pre-3.0 with “very
>>>> long running read commands” that have a barrier on the memtable
>>>> that prevent release.
>>>>
>>>> You’ve got the heap (great, way better than most people
>>>> debugging), are you able to navigate through it and look for
>>>> references to that memtable or other things holding a barrier?
>>>>
>>>>
>>>>
>>>>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
I can see some similarities and some differences between your thread
dump and ours.
In your thread dump:
* no MemtableFlushWriter thread
* the MemtablePostFlush thread is idle
* the MemtableReclaimMemory thread is waiting for a barrier, possibly
stuck
* the read-hotness-tracker thread is waiting for memtable memory
allocation. I'm not sure whether it's blocking a barrier, the
relevant part of the source code has changed between 4.1.1 and 4.1.3.
In our thread dump:
* two MemtableFlushWriter threads are stuck, each waiting for a barrier
* the MemtablePostFlush thread is stuck, waiting for a latch
* the MemtableReclaimMemory thread is idle
* the read-hotness-tracker thread is waiting for a signal, and it's
blocking the barrier one of the MemtableFlushWriter threads is
waiting on
It is possible that in both cases the read-hotness-tracker thread was
blocking the barrier, which caused the MemtableFlushWriter or
MemtableReclaimMemory thread to get stuck. It is also totally possible
that your case has nothing to do with the read-hotness-tracker thread.
I would recommend you try to find out what was blocking the barrier
MemtableReclaimMemory thread is waiting on.
On 06/11/2024 16:29, Jaydeep Chovatia wrote:
> We have also seen this issue a few times in our production (4.1). My
> teammates added a thread dump here
> <https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>.
>
> One of my theories is that the MemtableReclaimMemory thread has been
> unable to make progress:
>
> "MemtableReclaimMemory:1" - Thread t@56
>
> java.lang.Thread.State: WAITING
>
> at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native
> <http://java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native> Method)
>
> at
> java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> <http://java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)>
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
>
> at
> app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
>
> at
> app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
>
> at
> app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
>
> at
> app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
>
> at
> app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
> at
> app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)>
>
> at
> java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> <http://java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)>
>
> at
> app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
> at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
> <http://java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)>
>
>
> One of the possible bugs I could see is a line
> <https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
> / indexController = new ReadExecutionController(command,
> indexCfs.readOrdering.start(), indexCfs.metadata(), null, null,
> NO_SAMPLING, false);/
> If "/indexCfs.readOrdering.start()/" succeeded but the constructor
> "/new ReadExecutionController/", then we are not closing
> "/indexCfs.readOrdering/", which means it will remain
> inaccurate forever.
>
>
> Jaydeep
>
> On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> I think I'm getting really close now. This seems to have something
> to do with the "read-hotness-tracker:1" thread. The thread dump is:
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> <------ here
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable:
> org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> <------ and here
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at
> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable:
> org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
> local variable:
> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
> local variable:
> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> local variable:
> java.util.concurrent.Executors$RunnableAdapter#934
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#7
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#7
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#7
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable:
> io.netty.util.concurrent.FastThreadLocalRunnable#7
> at java.lang.Thread.run(Thread.java:829)
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#7
>
>
> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>
> public WriteContext beginWrite(Mutation mutation, boolean
> makeDurable) throws RequestExecutionException
> {
> OpOrder.Group group = null;
> try
> {
> group = Keyspace.writeOrder.start(); // <--- here
>
> // write the mutation to the commitlog and memtables
> CommitLogPosition position = null;
> if (makeDurable)
> {
> position = addToCommitLog(mutation); // <--- and here
> }
> return new CassandraWriteContext(group, position);
> }
> catch (Throwable t)
> {
> if (group != null)
> {
> group.close();
> }
> throw t;
> }
> }
>
> The Keyspace.writeOrder.start() method call blocks the
> MemtableFlushWriter thread, and then it calls
> addToCommitLog(mutation) which indirectly calls
> AbstractCommitLogService.awaitSyncAt()
>
> void awaitSyncAt(long syncTime, Context context)
> {
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) :
> syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly(); // <--- here
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
> }
>
> This then got stuck on the signal.awaitUninterruptibly()
>
> Now I know what is blocking the memtable flushing, but I haven't
> been able to figure out is why it got stuck on waiting for that
> signal.
>
> I would appreciate it if anyone can offer some insight here.
>
>
> On 05/11/2024 17:48, Bowen Song via user wrote:
>>
>> I will give it a try and see what I can find. I plan to go down
>> the rabbit hole tomorrow. Will keep you updated.
>>
>> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>>
>>>
>>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>>> <us...@cassandra.apache.org> <ma...@cassandra.apache.org>
>>>> wrote:
>>>>
>>>> Writes on this node starts to timeout and fail. But if left
>>>> untouched, it's only gonna get worse, and eventually lead to
>>>> JVM OOM and crash.
>>>>
>>>> By inspecting the heap dump created at OOM, we can see that
>>>> both of the MemtableFlushWriter threads are stuck on line 1190
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>> in the ColumnFamilyStore.java:
>>>>
>>>> // mark writes older than the barrier as blocking
>>>> progress, permitting them to exceed our memory limit
>>>> // if they are stuck waiting on it, then wait for
>>>> them all to complete
>>>> writeBarrier.markBlocking();
>>>> writeBarrier.await(); // <----------- stuck here
>>>>
>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>> in the same file.
>>>>
>>>> try
>>>> {
>>>> // we wait on the latch for the
>>>> commitLogUpperBound to be set, and so that waiters
>>>> // on this task can rely on all prior flushes
>>>> being complete
>>>> latch.await(); // <----------- stuck here
>>>> }
>>>> Our top suspect is CDC interacting with repair, since this
>>>> started to happen shortly after we enabled CDC on the nodes,
>>>> and each time repair was running. But we have not been able to
>>>> reproduce this in a testing cluster, and don't know what's the
>>>> next step to troubleshoot this issue. So I'm posting it in the
>>>> mailing lists and hoping someone may know something about it or
>>>> point me to the right direction.
>>>>
>>>
>>> Wouldn’t be completely surprised if CDC or repair somehow has
>>> a barrier, I’ve also seen similar behavior pre-3.0 with “very
>>> long running read commands” that have a barrier on the memtable
>>> that prevent release.
>>>
>>> You’ve got the heap (great, way better than most people
>>> debugging), are you able to navigate through it and look for
>>> references to that memtable or other things holding a barrier?
>>>
>>>
>>>
>>>
Re: Unexplained stuck memtable flush
Posted by Jaydeep Chovatia <ch...@gmail.com>.
We have also seen this issue a few times in our production (4.1). My
teammates added a thread dump here
<https://issues.apache.org/jira/browse/CASSANDRA-19564?focusedCommentId=17893987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17893987>
.
One of my theories is that the MemtableReclaimMemory thread has been unable
to make progress:
"MemtableReclaimMemory:1" - Thread t@56
java.lang.Thread.State: WAITING
at java.base@11.0.20.1/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@11.0.20.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at
app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
at
app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at
app//org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
at
app//org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
at
app//org.apache.cassandra.utils.concurrent.OpOrder$Group.await(OpOrder.java:267)
at
app//org.apache.cassandra.utils.concurrent.OpOrder$Barrier.await(OpOrder.java:425)
at
app//org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1357)
at
app//org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at
java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)
One of the possible bugs I could see is a line
<https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/ReadExecutionController.java#L141C77-L141C97>
* indexController = new ReadExecutionController(command,
indexCfs.readOrdering.start(), indexCfs.metadata(), null, null,
NO_SAMPLING, false);*
If "*indexCfs.readOrdering.start()*" succeeded but the constructor "*new
ReadExecutionController*", then we are not closing "*indexCfs.readOrdering*",
which means it will remain
inaccurate forever.
Jaydeep
On Wed, Nov 6, 2024 at 5:51 AM Bowen Song via user <
user@cassandra.apache.org> wrote:
> I think I'm getting really close now. This seems to have something to do
> with the "read-hotness-tracker:1" thread. The thread dump is:
>
> "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
> <------ here
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> local variable: com.codahale.metrics.Timer$Context#2086795
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
> local variable:
> org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
> local variable: org.apache.cassandra.db.commitlog.CommitLog#1
> local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
> local variable:
> org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
> at
> org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
> <------ and here
> local variable:
> org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
> local variable: org.apache.cassandra.db.Keyspace#8
> local variable: org.apache.cassandra.db.Mutation#54491
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
> at
> org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
> local variable: java.lang.String#42806
> local variable: java.lang.String#47659
> local variable:
> org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
> local variable: org.apache.cassandra.metrics.RestorableMeter#752
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
> local variable:
> org.apache.cassandra.concurrent.ExecutionFailure$1#122029
> local variable:
> org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> local variable: java.util.concurrent.Executors$RunnableAdapter#934
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
> local variable:
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
> at java.lang.Thread.run(Thread.java:829)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#7
>
>
> In the CassandraKeyspaceWriteHandler.beginWrite() method:
>
> public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
> throws RequestExecutionException
> {
> OpOrder.Group group = null;
> try
> {
> group = Keyspace.writeOrder.start(); // <--- here
>
> // write the mutation to the commitlog and memtables
> CommitLogPosition position = null;
> if (makeDurable)
> {
> position = addToCommitLog(mutation); // <--- and here
> }
> return new CassandraWriteContext(group, position);
> }
> catch (Throwable t)
> {
> if (group != null)
> {
> group.close();
> }
> throw t;
> }
> }
>
> The Keyspace.writeOrder.start() method call blocks the
> MemtableFlushWriter thread, and then it calls addToCommitLog(mutation) which
> indirectly calls AbstractCommitLogService.awaitSyncAt()
>
> void awaitSyncAt(long syncTime, Context context)
> {
> do
> {
> WaitQueue.Signal signal = context != null ?
> syncComplete.register(context, Context::stop) : syncComplete.register();
> if (lastSyncedAt < syncTime)
> signal.awaitUninterruptibly(); // <--- here
> else
> signal.cancel();
> }
> while (lastSyncedAt < syncTime);
> }
>
> This then got stuck on the signal.awaitUninterruptibly()
>
> Now I know what is blocking the memtable flushing, but I haven't been able
> to figure out is why it got stuck on waiting for that signal.
>
> I would appreciate it if anyone can offer some insight here.
>
>
> On 05/11/2024 17:48, Bowen Song via user wrote:
>
> I will give it a try and see what I can find. I plan to go down the rabbit
> hole tomorrow. Will keep you updated.
> On 05/11/2024 17:34, Jeff Jirsa wrote:
>
>
>
> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
> <us...@cassandra.apache.org> <us...@cassandra.apache.org> wrote:
>
> Writes on this node starts to timeout and fail. But if left untouched,
> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of the
> MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress,
> permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all to
> complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to be
> set, and so that waiters
> // on this task can rely on all prior flushes being
> complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to
> happen shortly after we enabled CDC on the nodes, and each time repair was
> running. But we have not been able to reproduce this in a testing cluster,
> and don't know what's the next step to troubleshoot this issue. So I'm
> posting it in the mailing lists and hoping someone may know something about
> it or point me to the right direction.
>
>
> Wouldn’t be completely surprised if CDC or repair somehow has a barrier,
> I’ve also seen similar behavior pre-3.0 with “very long running read
> commands” that have a barrier on the memtable that prevent release.
>
> You’ve got the heap (great, way better than most people debugging), are
> you able to navigate through it and look for references to that memtable or
> other things holding a barrier?
>
>
>
>
>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
I think I'm getting really close now. This seems to have something to do
with the "read-hotness-tracker:1" thread. The thread dump is:
"read-hotness-tracker:1" daemon prio=5 tid=93 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitUninterruptibly(Awaitable.java:186)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$SignalWithListener#2086139
at
org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitUninterruptibly(Awaitable.java:259)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService.awaitSyncAt(AbstractCommitLogService.java:324)
<------ here
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
local variable: com.codahale.metrics.Timer$Context#2086795
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogService.maybeWaitForSync(PeriodicCommitLogService.java:42)
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
at
org.apache.cassandra.db.commitlog.AbstractCommitLogService.finishWriteFor(AbstractCommitLogService.java:284)
local variable:
org.apache.cassandra.db.commitlog.PeriodicCommitLogService#1
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
local variable: org.apache.cassandra.db.commitlog.CommitLog#1
local variable: org.apache.cassandra.io.util.DataOutputBuffer$1$1#61
local variable:
org.apache.cassandra.db.commitlog.CommitLogSegment$Allocation#1
at
org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:100)
at
org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:54)
<------ and here
local variable:
org.apache.cassandra.utils.concurrent.OpOrder$Group#8162
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:628)
local variable: org.apache.cassandra.db.Keyspace#8
local variable: org.apache.cassandra.db.Mutation#54491
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:512)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:228)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:248)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:675)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:666)
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:447)
at
org.apache.cassandra.db.SystemKeyspace.persistSSTableReadMeter(SystemKeyspace.java:1488)
local variable: java.lang.String#42806
local variable: java.lang.String#47659
local variable:
org.apache.cassandra.io.sstable.UUIDBasedSSTableId#1389
local variable: org.apache.cassandra.metrics.RestorableMeter#752
at
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy$1.run(SSTableReader.java:2170)
at
org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
local variable:
org.apache.cassandra.concurrent.ExecutionFailure$1#122029
local variable:
org.apache.cassandra.utils.WithResources$None$$Lambda$223#1
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
local variable: java.util.concurrent.Executors$RunnableAdapter#934
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
local variable:
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
local variable:
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
local variable:
org.apache.cassandra.concurrent.ScheduledThreadPoolExecutorPlus#6
local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
local variable: io.netty.util.concurrent.FastThreadLocalThread#7
local variable:
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#931
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
local variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
local variable: io.netty.util.concurrent.FastThreadLocalRunnable#7
at java.lang.Thread.run(Thread.java:829)
local variable: io.netty.util.concurrent.FastThreadLocalThread#7
In the CassandraKeyspaceWriteHandler.beginWrite() method:
public WriteContext beginWrite(Mutation mutation, boolean makeDurable)
throws RequestExecutionException
{
OpOrder.Group group = null;
try
{
group = Keyspace.writeOrder.start(); // <--- here
// write the mutation to the commitlog and memtables
CommitLogPosition position = null;
if (makeDurable)
{
position = addToCommitLog(mutation); // <--- and here
}
return new CassandraWriteContext(group, position);
}
catch (Throwable t)
{
if (group != null)
{
group.close();
}
throw t;
}
}
The Keyspace.writeOrder.start() method call blocks the
MemtableFlushWriter thread, and then it calls addToCommitLog(mutation)
which indirectly calls AbstractCommitLogService.awaitSyncAt()
void awaitSyncAt(long syncTime, Context context)
{
do
{
WaitQueue.Signal signal = context != null ?
syncComplete.register(context, Context::stop) : syncComplete.register();
if (lastSyncedAt < syncTime)
signal.awaitUninterruptibly(); // <--- here
else
signal.cancel();
}
while (lastSyncedAt < syncTime);
}
This then got stuck on the signal.awaitUninterruptibly()
Now I know what is blocking the memtable flushing, but I haven't been
able to figure out is why it got stuck on waiting for that signal.
I would appreciate it if anyone can offer some insight here.
On 05/11/2024 17:48, Bowen Song via user wrote:
>
> I will give it a try and see what I can find. I plan to go down the
> rabbit hole tomorrow. Will keep you updated.
>
> On 05/11/2024 17:34, Jeff Jirsa wrote:
>>
>>
>>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> Writes on this node starts to timeout and fail. But if left
>>> untouched, it's only gonna get worse, and eventually lead to JVM OOM
>>> and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can see that both of
>>> the MemtableFlushWriter threads are stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as blocking
>>> progress, permitting them to exceed our memory limit
>>> // if they are stuck waiting on it, then wait for them
>>> all to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <----------- stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the commitLogUpperBound
>>> to be set, and so that waiters
>>> // on this task can rely on all prior flushes being
>>> complete
>>> latch.await(); // <----------- stuck here
>>> }
>>> Our top suspect is CDC interacting with repair, since this started
>>> to happen shortly after we enabled CDC on the nodes, and each time
>>> repair was running. But we have not been able to reproduce this in a
>>> testing cluster, and don't know what's the next step to troubleshoot
>>> this issue. So I'm posting it in the mailing lists and hoping
>>> someone may know something about it or point me to the right direction.
>>>
>>
>> Wouldn’t be completely surprised if CDC or repair somehow has a
>> barrier, I’ve also seen similar behavior pre-3.0 with “very long
>> running read commands” that have a barrier on the memtable that
>> prevent release.
>>
>> You’ve got the heap (great, way better than most people debugging),
>> are you able to navigate through it and look for references to that
>> memtable or other things holding a barrier?
>>
>>
>>
>>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
I will give it a try and see what I can find. I plan to go down the
rabbit hole tomorrow. Will keep you updated.
On 05/11/2024 17:34, Jeff Jirsa wrote:
>
>
>> On Nov 5, 2024, at 4:12 AM, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Writes on this node starts to timeout and fail. But if left
>> untouched, it's only gonna get worse, and eventually lead to JVM OOM
>> and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that both of
>> the MemtableFlushWriter threads are stuck on line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking
>> progress, permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for them
>> all to complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the commitLogUpperBound
>> to be set, and so that waiters
>> // on this task can rely on all prior flushes being
>> complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this started to
>> happen shortly after we enabled CDC on the nodes, and each time
>> repair was running. But we have not been able to reproduce this in a
>> testing cluster, and don't know what's the next step to troubleshoot
>> this issue. So I'm posting it in the mailing lists and hoping someone
>> may know something about it or point me to the right direction.
>>
>
> Wouldn’t be completely surprised if CDC or repair somehow has a
> barrier, I’ve also seen similar behavior pre-3.0 with “very long
> running read commands” that have a barrier on the memtable that
> prevent release.
>
> You’ve got the heap (great, way better than most people debugging),
> are you able to navigate through it and look for references to that
> memtable or other things holding a barrier?
>
>
>
>
Re: Unexplained stuck memtable flush
Posted by Jeff Jirsa <jj...@gmail.com>.
> On Nov 5, 2024, at 4:12 AM, Bowen Song via user <us...@cassandra.apache.org> wrote:
>
> Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all to complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to be set, and so that waiters
> // on this task can rely on all prior flushes being complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
>
Wouldn’t be completely surprised if CDC or repair somehow has a barrier, I’ve also seen similar behavior pre-3.0 with “very long running read commands” that have a barrier on the memtable that prevent release.
You’ve got the heap (great, way better than most people debugging), are you able to navigate through it and look for references to that memtable or other things holding a barrier?
Re: [External]Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
If it is not related to the memtable flush issue, can you please post in
a different mailing list thread instead?
By replying to this thread, everyone reading it would initially assume
it is somehow related, which neither good for them (wasting their time
to try to understand it) nor you (your message may get ignored because
it is off-topic).
On 05/11/2024 12:50, Jiri Steuer (EIT) wrote:
>
> Of cause, let me explain the situation. I have a common question
> without direct relation the problem with “Unexplained stuck memtable
> flush”. I would like to know, how can I identify situation that all
> nodes cross all data centers will be synch.
>
> * It is little tricky to wait e.g. 1 day, 2 days or longer time for
> the synchronization
>
> My question is. Do you see easy way, how can I check that expected
> huge synchronization (e.g. after extension cluster – add new data
> center or add half of new nodes) fill be finish from perspective of
> data synch? Thx for sharing you best practices, regards
>
> Jiri
>
> *
> *
>
> *
>
> This item's classification is Internal. It was created by and is in
> property of the EmbedIT. Do not distribute outside of the organization.
>
> From:* Bowen Song via user <us...@cassandra.apache.org>
> *Sent:* Tuesday, November 5, 2024 1:39 PM
> *To:* user@cassandra.apache.org
> *Cc:* Bowen Song <bo...@bso.ng>
> *Subject:* Re: [External]Unexplained stuck memtable flush
>
> This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly
> with links and attachments.
> Please report all suspicious e-mails to helpdesk@embedit.com
>
> ------------------------------------------------------------------------
>
> Hi Jiri,
>
> Thank you for taking a look at this issue. But I'm sorry, I don't
> really understand your message. Can you please elaborate?
>
> Cheers,
> Bowen
>
> On 05/11/2024 12:34, Jiri Steuer (EIT) wrote:
>
> Hi all,
>
> It is possible easy to check the moment/milestone, when the data
> cross more data centers will by synch (in case that other
> applications and user access will be disabled)? I think about
> monitoring of throughput or …? Thx for feedback
>
> J. Steuer
>
> This item's classification is Internal. It was created by and is
> in property of the EmbedIT. Do not distribute outside of the
> organization.
>
> *From:* Bowen Song via user <us...@cassandra.apache.org>
> <ma...@cassandra.apache.org>
> *Sent:* Tuesday, November 5, 2024 1:12 PM
> *To:* dev@cassandra.apache.org; user@cassandra.apache.org
> *Cc:* Bowen Song <bo...@bso.ng> <ma...@bso.ng>
> *Subject:* [External]Unexplained stuck memtable flush
>
> This message is from an EXTERNAL SENDER - be CAUTIOUS,
> particularly with links and attachments.
> Please report all suspicious e-mails to helpdesk@embedit.com
>
> ------------------------------------------------------------------------
>
> Hi all,
>
> We have a cluster running Cassandra 4.1.1. We are seeing
> the memtable flush randomly getting stuck. This has happened twice
> in the last 10 days, to two different nodes in the same cluster.
> This started to happen after we enabled CDC, and each time it got
> stuck, there was at least one repair running involving the
> affected node.
>
> The signs of the stuck memtable flush is most obvious from the
> "StatusLogger" logs.
>
> At the beginning, the MemtablePostFlush and MemtableFlushWriter
> got stuck, they have 1 and 2 active tasks each, and a small number
> of pending tasks.
>
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:65 - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ReadStage
> 0 0 34052333 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - CompactionExecutor 0
> 0 1019777 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MutationStage
> 0 0 14930764 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - GossipStage 0
> 0 889452 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - HintsDispatcher 0
> 0 19 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Repair-Task 0
> 0 65 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - RequestResponseStage
> 0 0 44447834 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Native-Transport-Requests 0
> 0 8967921 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtableFlushWriter 2
> 10 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtablePostFlush 1
> 11 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - InternalResponseStage 0
> 0 240 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - AntiEntropyStage 1
> 1773 120067 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
>
> The number of pending tasks slowly grows larger over time, and the
> number of completed tasks do not increase at all.
>
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:65 - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - ReadStage
> 0 0 39905462 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - CompactionExecutor 0
> 0 1170100 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MutationStage
> 0 0 16976992 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#76 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#74 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#72 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - GossipStage 0
> 0 1026877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair-Task 0
> 0 83 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - RequestResponseStage
> 0 0 51122191 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtablePostFlush 1
> 54 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtableFlushWriter 2
> 53 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#67 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#82 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - HintsDispatcher 0
> 0 20 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Native-Transport-Requests
> 0 0 10145020 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - InternalResponseStage 0
> 0 996 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - AntiEntropyStage 1
> 9694 120067 0 0
>
> Then, things start to get worse, the MutationStage also got stuck.
>
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:65 - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ReadStage
> 0 0 40186426 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - CompactionExecutor 0
> 0 1171479 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MutationStage 128
> 201414 16988212 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#76 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#74 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#72 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - GossipStage 0
> 0 1032988 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair-Task 0
> 0 85 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - RequestResponseStage
> 0 0 51307497 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtablePostFlush 1
> 54 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtableFlushWriter 2
> 53 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#67 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#84 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#82 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - HintsDispatcher 0
> 0 20 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Native-Transport-Requests
> 0 0 10194549 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - InternalResponseStage 0
> 0 1087 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - AntiEntropyStage 1
> 10364 120067 0 0
>
> Writes on this node starts to timeout and fail. But if left
> untouched, it's only gonna get worse, and eventually lead to JVM
> OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both
> of the MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress,
> permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them
> all to complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the
> commitLogUpperBound to be set, and so that waiters
> // on this task can rely on all prior flushes
> being complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started
> to happen shortly after we enabled CDC on the nodes, and each time
> repair was running. But we have not been able to reproduce this in
> a testing cluster, and don't know what's the next step to
> troubleshoot this issue. So I'm posting it in the mailing lists
> and hoping someone may know something about it or point me to the
> right direction.
>
> p.s.: sorry about posting this to both the user & dev mailing
> lists. It's an end-user related issue but involves Cassandra
> internals, so I can't decide which one is best suited.
>
> Cheers,
> Bowen
>
RE: [External]Unexplained stuck memtable flush
Posted by "Jiri Steuer (EIT)" <Ji...@embedit.com>.
Of cause, let me explain the situation. I have a common question without direct relation the problem with “Unexplained stuck memtable flush”. I would like to know, how can I identify situation that all nodes cross all data centers will be synch.
* It is little tricky to wait e.g. 1 day, 2 days or longer time for the synchronization
My question is. Do you see easy way, how can I check that expected huge synchronization (e.g. after extension cluster – add new data center or add half of new nodes) fill be finish from perspective of data synch? Thx for sharing you best practices, regards
Jiri
This item's classification is Internal. It was created by and is in property of the EmbedIT. Do not distribute outside of the organization.
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Tuesday, November 5, 2024 1:39 PM
To: user@cassandra.apache.org
Cc: Bowen Song <bo...@bso.ng>
Subject: Re: [External]Unexplained stuck memtable flush
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
Please report all suspicious e-mails to helpdesk@embedit.com<ma...@embedit.com>
________________________________
Hi Jiri,
Thank you for taking a look at this issue. But I'm sorry, I don't really understand your message. Can you please elaborate?
Cheers,
Bowen
On 05/11/2024 12:34, Jiri Steuer (EIT) wrote:
Hi all,
It is possible easy to check the moment/milestone, when the data cross more data centers will by synch (in case that other applications and user access will be disabled)? I think about monitoring of throughput or …? Thx for feedback
J. Steuer
This item's classification is Internal. It was created by and is in property of the EmbedIT. Do not distribute outside of the organization.
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Tuesday, November 5, 2024 1:12 PM
To: dev@cassandra.apache.org<ma...@cassandra.apache.org>; user@cassandra.apache.org<ma...@cassandra.apache.org>
Cc: Bowen Song <bo...@bso.ng>
Subject: [External]Unexplained stuck memtable flush
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
Please report all suspicious e-mails to helpdesk@embedit.com<ma...@embedit.com>
________________________________
Hi all,
We have a cluster running Cassandra 4.1.1. We are seeing the memtable flush randomly getting stuck. This has happened twice in the last 10 days, to two different nodes in the same cluster. This started to happen after we enabled CDC, and each time it got stuck, there was at least one repair running involving the affected node.
The signs of the stuck memtable flush is most obvious from the "StatusLogger" logs.
At the beginning, the MemtablePostFlush and MemtableFlushWriter got stuck, they have 1 and 2 active tasks each, and a small number of pending tasks.
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ReadStage 0 0 34052333 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - CompactionExecutor 0 0 1019777 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MutationStage 0 0 14930764 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - GossipStage 0 0 889452 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - HintsDispatcher 0 0 19 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Repair-Task 0 0 65 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - RequestResponseStage 0 0 44447834 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Native-Transport-Requests 0 0 8967921 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtableFlushWriter 2 10 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtablePostFlush 1 11 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - InternalResponseStage 0 0 240 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - AntiEntropyStage 1 1773 120067 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
The number of pending tasks slowly grows larger over time, and the number of completed tasks do not increase at all.
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - ReadStage 0 0 39905462 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - CompactionExecutor 0 0 1170100 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MutationStage 0 0 16976992 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - GossipStage 0 0 1026877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair-Task 0 0 83 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - RequestResponseStage 0 0 51122191 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Native-Transport-Requests 0 0 10145020 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - InternalResponseStage 0 0 996 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - AntiEntropyStage 1 9694 120067 0 0
Then, things start to get worse, the MutationStage also got stuck.
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ReadStage 0 0 40186426 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - CompactionExecutor 0 0 1171479 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MutationStage 128 201414 16988212 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - GossipStage 0 0 1032988 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair-Task 0 0 85 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - RequestResponseStage 0 0 51307497 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#84 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Native-Transport-Requests 0 0 10194549 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - InternalResponseStage 0 0 1087 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - AntiEntropyStage 1 10364 120067 0 0
Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190> in the ColumnFamilyStore.java:
// mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
// if they are stuck waiting on it, then wait for them all to complete
writeBarrier.markBlocking();
writeBarrier.await(); // <----------- stuck here
And the MemtablePostFlush thread is stuck on line 1094<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094> in the same file.
try
{
// we wait on the latch for the commitLogUpperBound to be set, and so that waiters
// on this task can rely on all prior flushes being complete
latch.await(); // <----------- stuck here
}
Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
p.s.: sorry about posting this to both the user & dev mailing lists. It's an end-user related issue but involves Cassandra internals, so I can't decide which one is best suited.
Cheers,
Bowen
Re: [External]Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Jiri,
Thank you for taking a look at this issue. But I'm sorry, I don't really
understand your message. Can you please elaborate?
Cheers,
Bowen
On 05/11/2024 12:34, Jiri Steuer (EIT) wrote:
>
> Hi all,
>
> It is possible easy to check the moment/milestone, when the data cross
> more data centers will by synch (in case that other applications and
> user access will be disabled)? I think about monitoring of throughput
> or …? Thx for feedback
>
> J. Steuer
>
> *
> *
>
> *
>
> This item's classification is Internal. It was created by and is in
> property of the EmbedIT. Do not distribute outside of the organization.
>
> From:* Bowen Song via user <us...@cassandra.apache.org>
> *Sent:* Tuesday, November 5, 2024 1:12 PM
> *To:* dev@cassandra.apache.org; user@cassandra.apache.org
> *Cc:* Bowen Song <bo...@bso.ng>
> *Subject:* [External]Unexplained stuck memtable flush
>
> This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly
> with links and attachments.
> Please report all suspicious e-mails to helpdesk@embedit.com
>
> ------------------------------------------------------------------------
>
> Hi all,
>
> We have a cluster running Cassandra 4.1.1. We are seeing the memtable
> flush randomly getting stuck. This has happened twice in the last 10
> days, to two different nodes in the same cluster. This started to
> happen after we enabled CDC, and each time it got stuck, there was at
> least one repair running involving the affected node.
>
> The signs of the stuck memtable flush is most obvious from the
> "StatusLogger" logs.
>
> At the beginning, the MemtablePostFlush and MemtableFlushWriter got
> stuck, they have 1 and 2 active tasks each, and a small number of
> pending tasks.
>
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - ReadStage 0 0 34052333
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - CompactionExecutor 0 0 1019777
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - MutationStage 0 0 14930764
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - GossipStage 0 0 889452
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - HintsDispatcher 0 0 19
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - Repair-Task 0 0 65
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - RequestResponseStage 0 0 44447834
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - Native-Transport-Requests 0 0 8967921
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - MemtableFlushWriter 2 10 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - MemtablePostFlush 1 11 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - InternalResponseStage 0 0 240
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - AntiEntropyStage 1 1773 120067
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
>
> The number of pending tasks slowly grows larger over time, and the
> number of completed tasks do not increase at all.
>
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - ReadStage 0 0 39905462
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - CompactionExecutor 0 0 1170100
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - MutationStage 0 0 16976992
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#76 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#74 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#72 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - GossipStage 0 0 1026877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair-Task 0 0 83
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - RequestResponseStage 0 0 51122191
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - MemtablePostFlush 1 54 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - MemtableFlushWriter 2 53 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#67 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#82 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - HintsDispatcher 0 0 20
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Native-Transport-Requests 0 0 10145020
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69
> - InternalResponseStage 0 0 996
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69
> - AntiEntropyStage 1 9694 120067
> 0 0
>
> Then, things start to get worse, the MutationStage also got stuck.
>
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - ReadStage 0 0 40186426
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - CompactionExecutor 0 0 1171479
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - MutationStage 128 201414 16988212 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#76 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#74 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#72 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - GossipStage 0 0 1032988
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair-Task 0 0 85
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - RequestResponseStage 0 0 51307497
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - MemtablePostFlush 1 54 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - MemtableFlushWriter 2 53 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#67 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#84 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#82 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - HintsDispatcher 0 0 20
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Native-Transport-Requests 0 0 10194549
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - InternalResponseStage 0 0 1087
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69
> - AntiEntropyStage 1 10364 120067
> 0 0
>
> Writes on this node starts to timeout and fail. But if left untouched,
> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of
> the MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress, permitting
> them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all
> to complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to
> be set, and so that waiters
> // on this task can rely on all prior flushes being
> complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to
> happen shortly after we enabled CDC on the nodes, and each time repair
> was running. But we have not been able to reproduce this in a testing
> cluster, and don't know what's the next step to troubleshoot this
> issue. So I'm posting it in the mailing lists and hoping someone may
> know something about it or point me to the right direction.
>
> p.s.: sorry about posting this to both the user & dev mailing lists.
> It's an end-user related issue but involves Cassandra internals, so I
> can't decide which one is best suited.
>
> Cheers,
> Bowen
>
RE: [External]Unexplained stuck memtable flush
Posted by "Jiri Steuer (EIT)" <Ji...@embedit.com>.
Hi all,
It is possible easy to check the moment/milestone, when the data cross more data centers will by synch (in case that other applications and user access will be disabled)? I think about monitoring of throughput or …? Thx for feedback
J. Steuer
This item's classification is Internal. It was created by and is in property of the EmbedIT. Do not distribute outside of the organization.
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Tuesday, November 5, 2024 1:12 PM
To: dev@cassandra.apache.org; user@cassandra.apache.org
Cc: Bowen Song <bo...@bso.ng>
Subject: [External]Unexplained stuck memtable flush
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
Please report all suspicious e-mails to helpdesk@embedit.com<ma...@embedit.com>
________________________________
Hi all,
We have a cluster running Cassandra 4.1.1. We are seeing the memtable flush randomly getting stuck. This has happened twice in the last 10 days, to two different nodes in the same cluster. This started to happen after we enabled CDC, and each time it got stuck, there was at least one repair running involving the affected node.
The signs of the stuck memtable flush is most obvious from the "StatusLogger" logs.
At the beginning, the MemtablePostFlush and MemtableFlushWriter got stuck, they have 1 and 2 active tasks each, and a small number of pending tasks.
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ReadStage 0 0 34052333 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - CompactionExecutor 0 0 1019777 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MutationStage 0 0 14930764 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - GossipStage 0 0 889452 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - HintsDispatcher 0 0 19 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Repair-Task 0 0 65 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - RequestResponseStage 0 0 44447834 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Native-Transport-Requests 0 0 8967921 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtableFlushWriter 2 10 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - MemtablePostFlush 1 11 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - InternalResponseStage 0 0 240 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - AntiEntropyStage 1 1773 120067 0 0
INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
The number of pending tasks slowly grows larger over time, and the number of completed tasks do not increase at all.
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - ReadStage 0 0 39905462 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - CompactionExecutor 0 0 1170100 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MutationStage 0 0 16976992 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - GossipStage 0 0 1026877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair-Task 0 0 83 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - RequestResponseStage 0 0 51122191 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Native-Transport-Requests 0 0 10145020 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - InternalResponseStage 0 0 996 0 0
INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 - AntiEntropyStage 1 9694 120067 0 0
Then, things start to get worse, the MutationStage also got stuck.
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ReadStage 0 0 40186426 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - CompactionExecutor 0 0 1171479 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MutationStage 128 201414 16988212 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#76 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#74 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#72 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - GossipStage 0 0 1032988 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair-Task 0 0 85 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - RequestResponseStage 0 0 51307497 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ValidationExecutor 0 0 36651 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtablePostFlush 1 54 47856 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtableFlushWriter 2 53 21781 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#67 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MemtableReclaimMemory 0 0 21877 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#84 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - PendingRangeCalculator 0 0 177 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#82 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Repair#61 0 0 1344 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - SecondaryIndexManagement 0 0 1 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - HintsDispatcher 0 0 20 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Native-Transport-Requests 0 0 10194549 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - MigrationStage 0 0 5 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 21769 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - InternalResponseStage 0 0 1087 0 0
INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 - AntiEntropyStage 1 10364 120067 0 0
Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190> in the ColumnFamilyStore.java:
// mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
// if they are stuck waiting on it, then wait for them all to complete
writeBarrier.markBlocking();
writeBarrier.await(); // <----------- stuck here
And the MemtablePostFlush thread is stuck on line 1094<https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094> in the same file.
try
{
// we wait on the latch for the commitLogUpperBound to be set, and so that waiters
// on this task can rely on all prior flushes being complete
latch.await(); // <----------- stuck here
}
Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
p.s.: sorry about posting this to both the user & dev mailing lists. It's an end-user related issue but involves Cassandra internals, so I can't decide which one is best suited.
Cheers,
Bowen
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Sorry, I must have misread it. The full thread dump is attached. I
compressed it with gzip because the text file is over 1 MB in size.
On 05/11/2024 14:04, Dmitry Konstantinov wrote:
> I am speaking about a thread dump (stack traces for all threads), not
> a heap dump. The heap dump should contain thread stacks info.
> Thread dump (stack traces) is small and does not have sensitive info.
>
> Regards,
> Dmitry
>
> On Tue, 5 Nov 2024 at 13:53, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> It's about 18GB in size and may contain a huge amount of sensitive
> data (e.g. all the pending writes), so I can't share it. However,
> if there's any particular piece of information you would like to
> have, I'm more than happy to extract the info from the dump and
> and share it here.
>
> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>> Hi Bowen, would it be possible to share a full thread dump?
>>
>> Regards,
>> Dmitry
>>
>> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Hi all,
>>
>> We have a cluster running Cassandra 4.1.1. We are seeing
>> the memtable flush randomly getting stuck. This has happened
>> twice in the last 10 days, to two different nodes in the same
>> cluster. This started to happen after we enabled CDC, and
>> each time it got stuck, there was at least one repair running
>> involving the affected node.
>>
>> The signs of the stuck memtable flush is most obvious from
>> the "StatusLogger" logs.
>>
>> At the beginning, the MemtablePostFlush and
>> MemtableFlushWriter got stuck, they have 1 and 2 active tasks
>> each, and a small number of pending tasks.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:65 - Pool Name Active
>> Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ReadStage 0
>> 0 34052333 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1019777 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MutationStage 0
>> 0 14930764 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Repair#61 0
>> 0 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - GossipStage 0
>> 0 889452 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 19 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Repair-Task 0
>> 0 65 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 44447834 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Native-Transport-Requests 0
>> 0 8967921 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 10 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 11 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
>> 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Sampler 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 240 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - AntiEntropyStage 1
>> 1773 120067 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>>
>> The number of pending tasks slowly grows larger over time,
>> and the number of completed tasks do not increase at all.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:65 - Pool Name Active
>> Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - ReadStage 0
>> 0 39905462 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1170100 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MutationStage 0
>> 0 16976992 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#76 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#74 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#72 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - GossipStage 0
>> 0 1026877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair-Task 0
>> 0 83 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 51122191 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 54 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 53 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#67 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#82 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#61 0
>> 0 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 20 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Native-Transport-Requests 0
>> 0 10145020 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Sampler 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
>> 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 996 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 - AntiEntropyStage 1
>> 9694 120067 0 0
>>
>> Then, things start to get worse, the MutationStage also got
>> stuck.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:65 - Pool Name Active
>> Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ReadStage 0
>> 0 40186426 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1171479 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MutationStage 128
>> 201414 16988212 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#76 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#74 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#72 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - GossipStage 0
>> 0 1032988 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair-Task 0
>> 0 85 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 51307497 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 54 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 53 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#67 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#84 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#82 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#61 0
>> 0 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 20 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Native-Transport-Requests 0
>> 0 10194549 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Sampler 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
>> 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 1087 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - AntiEntropyStage 1
>> 10364 120067 0 0
>>
>> Writes on this node starts to timeout and fail. But if left
>> untouched, it's only gonna get worse, and eventually lead to
>> JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that
>> both of the MemtableFlushWriter threads are stuck on line
>> 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking
>> progress, permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for
>> them all to complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the
>> commitLogUpperBound to be set, and so that waiters
>> // on this task can rely on all prior flushes
>> being complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this
>> started to happen shortly after we enabled CDC on the nodes,
>> and each time repair was running. But we have not been able
>> to reproduce this in a testing cluster, and don't know what's
>> the next step to troubleshoot this issue. So I'm posting it
>> in the mailing lists and hoping someone may know something
>> about it or point me to the right direction.
>>
>> p.s.: sorry about posting this to both the user & dev mailing
>> lists. It's an end-user related issue but involves Cassandra
>> internals, so I can't decide which one is best suited.
>>
>> Cheers,
>> Bowen
>>
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Funny enough, we used to run on ext4 and XFS on mdarray RAID1, but the
crappy disks we had (and still have) randomly spitting out garbage data
every once in a while. We suspected it's a firmware bug but unable to
confirm or reliably reproduce it. Other than this behaviour, those disks
work fine. Since mdarray can't tell which copy of the data is good, it
has a 50% chance overwriting the good data with the garbage, usually
resulting in SSTable corruptions. Since we switched to ZFS, never had an
issue with that again. ZFS saved countless hours of replacing nodes, and
saved the replacement cost of all the crappy disks (we had many hundreds
of them). So I'm fairly happy with that outcome. Do I wish we had better
disks? Yes, definitely. But we don't, so ZFS will have to do it.
I will try to find out what is blocking the barrier, and will post
updates here if I find anything.
On 05/11/2024 17:22, Jon Haddad wrote:
> Yeah, I looked through your stack trace and saw it wasn't the same
> thing, but the steps to identify the root cause should be the same.
>
> I nuked ZFS from orbit :) This was happening across all the machines
> at various times in the cluster, and we haven't seen a single issue
> since switching to XFS.
>
> Thanks for the advice though, I'll keep it in mind if I encounter it
> again.
>
> Jon
>
> On Tue, Nov 5, 2024 at 9:18 AM Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Hi Jon,
>
> That is interesting. We happen to be running Cassandra on ZFS.
> However we have not had any incident for years with this setup,
> the only change is the recent addition of CDC.
>
> I can see that in CASSANDRA-19564, the MemtablePostFlush thread
> was stuck on the unlink() syscall. But in our case, it was stuck here:
>
> "MemtablePostFlush:1" daemon prio=5 tid=237 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$RegisteredSignal#1
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:306)
> local variable:
> org.apache.cassandra.utils.concurrent.CountDownLatch$Async#7
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:338)
> at
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1094)
> local variable:
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush#7
> at
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1077)
> at
> org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
> at
> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.SingleThreadExecutorPlus#5
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#9
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#11
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable:
> java.util.concurrent.ThreadPoolExecutor$Worker#9
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable:
> io.netty.util.concurrent.FastThreadLocalRunnable#11
> at java.lang.Thread.run(Thread.java:829)
> local variable:
> io.netty.util.concurrent.FastThreadLocalThread#11
>
> Which made me believe that they are not the same issue.
>
> Thanks for the info though.
>
> BTW, if you still have access to that oddly behaving ZFS
> filesystem, have a look at the free space (should have at least
> 20% free) and the fragmentation ratio in "zpool list". If neither
> is an issue, check whether deduplication is enabled or there's a
> large number of snapshots. These can have significant impact on
> file deletion performance on ZFS. Also worth checking the disks, I
> have seen broken disks that stuck on some operations, e.g. when a
> specific sector is being read, and this will certainly affect the
> filesystem behaviour.
>
> Cheers,
> Bowen
>
> On 05/11/2024 16:41, Jon Haddad wrote:
>> I ran into this a few months ago, and in my case I tracked it
>> down to an issue with ZFS not unlinking commitlogs properly.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-19564
>>
>> On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov
>> <ne...@gmail.com> wrote:
>>
>> I am speaking about a thread dump (stack traces for all
>> threads), not a heap dump. The heap dump should contain
>> thread stacks info.
>> Thread dump (stack traces) is small and does not have
>> sensitive info.
>>
>> Regards,
>> Dmitry
>>
>> On Tue, 5 Nov 2024 at 13:53, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> It's about 18GB in size and may contain a huge amount of
>> sensitive data (e.g. all the pending writes), so I can't
>> share it. However, if there's any particular piece of
>> information you would like to have, I'm more than happy
>> to extract the info from the dump and and share it here.
>>
>> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>>> Hi Bowen, would it be possible to share a full thread dump?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user
>>> <us...@cassandra.apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> We have a cluster running Cassandra 4.1.1. We are
>>> seeing the memtable flush randomly getting stuck.
>>> This has happened twice in the last 10 days, to two
>>> different nodes in the same cluster. This started to
>>> happen after we enabled CDC, and each time it got
>>> stuck, there was at least one repair running
>>> involving the affected node.
>>>
>>> The signs of the stuck memtable flush is most
>>> obvious from the "StatusLogger" logs.
>>>
>>> At the beginning, the MemtablePostFlush and
>>> MemtableFlushWriter got stuck, they have 1 and 2
>>> active tasks each, and a small number of pending tasks.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:65 - Pool Name Active
>>> Pending Completed Blocked All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - ReadStage 0 0
>>> 34052333 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - CompactionExecutor 0
>>> 0 1019777 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - MutationStage 0
>>> 0 14930764 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - MemtableReclaimMemory
>>> 0 0 21877 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - PendingRangeCalculator
>>> 0 0 177 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - Repair#61 0
>>> 0 1344 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - GossipStage 0
>>> 0 889452 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - SecondaryIndexManagement
>>> 0 0 1 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - HintsDispatcher 0
>>> 0 19 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - Repair-Task 0
>>> 0 65 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - RequestResponseStage
>>> 0 0 44447834 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - Native-Transport-Requests
>>> 0 0 8967921 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - MigrationStage 0
>>> 0 5 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - MemtableFlushWriter 2
>>> 10 21781 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - MemtablePostFlush 1
>>> 11 47856 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>>> 0 0 21769 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - Sampler 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - ValidationExecutor 0
>>> 0 36651 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - ViewBuildExecutor 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - InternalResponseStage
>>> 0 0 240 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - AntiEntropyStage 1
>>> 1773 120067 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>>> StatusLogger.java:69 - CacheCleanupExecutor
>>> 0 0 0 0 0
>>>
>>> The number of pending tasks slowly grows larger over
>>> time, and the number of completed tasks do not
>>> increase at all.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:65 - Pool Name Active
>>> Pending Completed Blocked All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - ReadStage 0 0
>>> 39905462 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - CompactionExecutor 0
>>> 0 1170100 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - MutationStage 0
>>> 0 16976992 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#76 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#74 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#72 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - GossipStage 0
>>> 0 1026877 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair-Task 0
>>> 0 83 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - RequestResponseStage
>>> 0 0 51122191 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - ValidationExecutor 0
>>> 0 36651 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - MemtablePostFlush 1
>>> 54 47856 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - MemtableFlushWriter 2
>>> 53 21781 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - CacheCleanupExecutor
>>> 0 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#67 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - MemtableReclaimMemory
>>> 0 0 21877 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - PendingRangeCalculator
>>> 0 0 177 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#82 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Repair#61 0
>>> 0 1344 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - SecondaryIndexManagement
>>> 0 0 1 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - HintsDispatcher 0
>>> 0 20 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Native-Transport-Requests
>>> 0 0 10145020 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - MigrationStage 0
>>> 0 5 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - Sampler 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>>> 0 0 21769 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>>> StatusLogger.java:69 - ViewBuildExecutor 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>>> StatusLogger.java:69 - InternalResponseStage
>>> 0 0 996 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>>> StatusLogger.java:69 - AntiEntropyStage 1
>>> 9694 120067 0 0
>>>
>>> Then, things start to get worse, the MutationStage
>>> also got stuck.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:65 - Pool Name Active
>>> Pending Completed Blocked All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - ReadStage 0 0
>>> 40186426 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - CompactionExecutor 0
>>> 0 1171479 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - MutationStage 128
>>> 201414 16988212 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#76 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#74 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#72 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - GossipStage 0
>>> 0 1032988 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair-Task 0
>>> 0 85 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - RequestResponseStage
>>> 0 0 51307497 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - ValidationExecutor 0
>>> 0 36651 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - MemtablePostFlush 1
>>> 54 47856 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - MemtableFlushWriter 2
>>> 53 21781 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - CacheCleanupExecutor
>>> 0 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#67 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - MemtableReclaimMemory
>>> 0 0 21877 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#84 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - PendingRangeCalculator
>>> 0 0 177 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#82 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Repair#61 0
>>> 0 1344 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - SecondaryIndexManagement
>>> 0 0 1 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - HintsDispatcher 0
>>> 0 20 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Native-Transport-Requests
>>> 0 0 10194549 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - MigrationStage 0
>>> 0 5 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - Sampler 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>>> 0 0 21769 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - ViewBuildExecutor 0
>>> 0 0 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - InternalResponseStage
>>> 0 0 1087 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>>> StatusLogger.java:69 - AntiEntropyStage 1
>>> 10364 120067 0 0
>>>
>>> Writes on this node starts to timeout and fail. But
>>> if left untouched, it's only gonna get worse, and
>>> eventually lead to JVM OOM and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can
>>> see that both of the MemtableFlushWriter threads are
>>> stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as
>>> blocking progress, permitting them to exceed our
>>> memory limit
>>> // if they are stuck waiting on it, then
>>> wait for them all to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <-----------
>>> stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line
>>> 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the
>>> commitLogUpperBound to be set, and so that waiters
>>> // on this task can rely on all
>>> prior flushes being complete
>>> latch.await(); // <-----------
>>> stuck here
>>> }
>>> Our top suspect is CDC interacting with repair,
>>> since this started to happen shortly after we
>>> enabled CDC on the nodes, and each time repair was
>>> running. But we have not been able to reproduce this
>>> in a testing cluster, and don't know what's the next
>>> step to troubleshoot this issue. So I'm posting it
>>> in the mailing lists and hoping someone may know
>>> something about it or point me to the right direction.
>>>
>>> p.s.: sorry about posting this to both the user &
>>> dev mailing lists. It's an end-user related issue
>>> but involves Cassandra internals, so I can't decide
>>> which one is best suited.
>>>
>>> Cheers,
>>> Bowen
>>>
>>>
>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>
>>
>>
>> --
>> Dmitry Konstantinov
>>
Re: Unexplained stuck memtable flush
Posted by Jon Haddad <jo...@rustyrazorblade.com>.
Yeah, I looked through your stack trace and saw it wasn't the same thing,
but the steps to identify the root cause should be the same.
I nuked ZFS from orbit :) This was happening across all the machines at
various times in the cluster, and we haven't seen a single issue since
switching to XFS.
Thanks for the advice though, I'll keep it in mind if I encounter it again.
Jon
On Tue, Nov 5, 2024 at 9:18 AM Bowen Song via user <
user@cassandra.apache.org> wrote:
> Hi Jon,
>
> That is interesting. We happen to be running Cassandra on ZFS. However we
> have not had any incident for years with this setup, the only change is the
> recent addition of CDC.
>
> I can see that in CASSANDRA-19564, the MemtablePostFlush thread was stuck
> on the unlink() syscall. But in our case, it was stuck here:
>
> "MemtablePostFlush:1" daemon prio=5 tid=237 WAITING
> at jdk.internal.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
> local variable:
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$RegisteredSignal#1
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:306)
> local variable:
> org.apache.cassandra.utils.concurrent.CountDownLatch$Async#7
> at
> org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:338)
> at
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1094)
> local variable:
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush#7
> at
> org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1077)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> local variable:
> org.apache.cassandra.concurrent.SingleThreadExecutorPlus#5
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
> local variable: io.netty.util.concurrent.FastThreadLocalThread#11
> local variable: org.apache.cassandra.concurrent.FutureTask#2555
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> local variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> local variable: io.netty.util.concurrent.FastThreadLocalRunnable#11
> at java.lang.Thread.run(Thread.java:829)
> local variable: io.netty.util.concurrent.FastThreadLocalThread#11
>
> Which made me believe that they are not the same issue.
>
> Thanks for the info though.
>
> BTW, if you still have access to that oddly behaving ZFS filesystem, have
> a look at the free space (should have at least 20% free) and the
> fragmentation ratio in "zpool list". If neither is an issue, check whether
> deduplication is enabled or there's a large number of snapshots. These can
> have significant impact on file deletion performance on ZFS. Also worth
> checking the disks, I have seen broken disks that stuck on some operations,
> e.g. when a specific sector is being read, and this will certainly affect
> the filesystem behaviour.
>
> Cheers,
> Bowen
> On 05/11/2024 16:41, Jon Haddad wrote:
>
> I ran into this a few months ago, and in my case I tracked it down to an
> issue with ZFS not unlinking commitlogs properly.
>
> https://issues.apache.org/jira/browse/CASSANDRA-19564
>
> On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov <ne...@gmail.com>
> wrote:
>
>> I am speaking about a thread dump (stack traces for all threads), not a
>> heap dump. The heap dump should contain thread stacks info.
>> Thread dump (stack traces) is small and does not have sensitive info.
>>
>> Regards,
>> Dmitry
>>
>> On Tue, 5 Nov 2024 at 13:53, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> It's about 18GB in size and may contain a huge amount of sensitive data
>>> (e.g. all the pending writes), so I can't share it. However, if there's any
>>> particular piece of information you would like to have, I'm more than happy
>>> to extract the info from the dump and and share it here.
>>> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>>>
>>> Hi Bowen, would it be possible to share a full thread dump?
>>>
>>> Regards,
>>> Dmitry
>>>
>>> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We have a cluster running Cassandra 4.1.1. We are seeing the memtable
>>>> flush randomly getting stuck. This has happened twice in the last 10 days,
>>>> to two different nodes in the same cluster. This started to happen after we
>>>> enabled CDC, and each time it got stuck, there was at least one repair
>>>> running involving the affected node.
>>>>
>>>> The signs of the stuck memtable flush is most obvious from the
>>>> "StatusLogger" logs.
>>>>
>>>> At the beginning, the MemtablePostFlush and MemtableFlushWriter got
>>>> stuck, they have 1 and 2 active tasks each, and a small number of pending
>>>> tasks.
>>>>
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
>>>> Pool Name Active Pending Completed Blocked
>>>> All Time Blocked
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> ReadStage 0 0 34052333
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> CompactionExecutor 0 0 1019777
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> MutationStage 0 0 14930764
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> MemtableReclaimMemory 0 0 21877
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> PendingRangeCalculator 0 0 177
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> Repair#61 0 0 1344
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> GossipStage 0 0 889452
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> SecondaryIndexManagement 0 0 1
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> HintsDispatcher 0 0 19
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> Repair-Task 0 0 65
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> RequestResponseStage 0 0 44447834
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> Native-Transport-Requests 0 0 8967921
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> MigrationStage 0 0 5
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> MemtableFlushWriter 2 10 21781
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> MemtablePostFlush 1 11 47856
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> Sampler 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> ValidationExecutor 0 0 36651
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> ViewBuildExecutor 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> InternalResponseStage 0 0 240
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> AntiEntropyStage 1 1773 120067
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>>> CacheCleanupExecutor 0 0 0
>>>> 0 0
>>>>
>>>> The number of pending tasks slowly grows larger over time, and the
>>>> number of completed tasks do not increase at all.
>>>>
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
>>>> Pool Name Active Pending Completed Blocked
>>>> All Time Blocked
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> ReadStage 0 0 39905462
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> CompactionExecutor 0 0 1170100
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> MutationStage 0 0 16976992
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#76 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#74 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#72 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> GossipStage 0 0 1026877
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair-Task 0 0 83
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> RequestResponseStage 0 0 51122191
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> ValidationExecutor 0 0 36651
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> MemtablePostFlush 1 54 47856
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> MemtableFlushWriter 2 53 21781
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> CacheCleanupExecutor 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#67 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> MemtableReclaimMemory 0 0 21877
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> PendingRangeCalculator 0 0 177
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#82 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Repair#61 0 0 1344
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> SecondaryIndexManagement 0 0 1
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> HintsDispatcher 0 0 20
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Native-Transport-Requests 0 0 10145020
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> MigrationStage 0 0 5
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> Sampler 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>>> ViewBuildExecutor 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>>> InternalResponseStage 0 0 996
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>>> AntiEntropyStage 1 9694 120067
>>>> 0 0
>>>>
>>>> Then, things start to get worse, the MutationStage also got stuck.
>>>>
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
>>>> Pool Name Active Pending Completed Blocked
>>>> All Time Blocked
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> ReadStage 0 0 40186426
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> CompactionExecutor 0 0 1171479
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> MutationStage 128 201414 16988212
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#76 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#74 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#72 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> GossipStage 0 0 1032988
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair-Task 0 0 85
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> RequestResponseStage 0 0 51307497
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> ValidationExecutor 0 0 36651
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> MemtablePostFlush 1 54 47856
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> MemtableFlushWriter 2 53 21781
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> CacheCleanupExecutor 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#67 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> MemtableReclaimMemory 0 0 21877
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#84 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> PendingRangeCalculator 0 0 177
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#82 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Repair#61 0 0 1344
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> SecondaryIndexManagement 0 0 1
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> HintsDispatcher 0 0 20
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Native-Transport-Requests 0 0 10194549
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> MigrationStage 0 0 5
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> Sampler 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> ViewBuildExecutor 0 0 0
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> InternalResponseStage 0 0 1087
>>>> 0 0
>>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>>> AntiEntropyStage 1 10364 120067
>>>> 0 0
>>>>
>>>> Writes on this node starts to timeout and fail. But if left untouched,
>>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>>
>>>> By inspecting the heap dump created at OOM, we can see that both of the
>>>> MemtableFlushWriter threads are stuck on line 1190
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>>> in the ColumnFamilyStore.java:
>>>>
>>>> // mark writes older than the barrier as blocking progress,
>>>> permitting them to exceed our memory limit
>>>> // if they are stuck waiting on it, then wait for them all
>>>> to complete
>>>> writeBarrier.markBlocking();
>>>> writeBarrier.await(); // <----------- stuck here
>>>>
>>>> And the MemtablePostFlush thread is stuck on line 1094
>>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>>> in the same file.
>>>>
>>>> try
>>>> {
>>>> // we wait on the latch for the commitLogUpperBound to
>>>> be set, and so that waiters
>>>> // on this task can rely on all prior flushes being
>>>> complete
>>>> latch.await(); // <----------- stuck here
>>>> }
>>>> Our top suspect is CDC interacting with repair, since this started to
>>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>>> running. But we have not been able to reproduce this in a testing cluster,
>>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>>> posting it in the mailing lists and hoping someone may know something about
>>>> it or point me to the right direction.
>>>>
>>>> p.s.: sorry about posting this to both the user & dev mailing lists.
>>>> It's an end-user related issue but involves Cassandra internals, so I can't
>>>> decide which one is best suited.
>>>>
>>>> Cheers,
>>>> Bowen
>>>>
>>>>
>>>>
>>>
>>> --
>>> Dmitry Konstantinov
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi Jon,
That is interesting. We happen to be running Cassandra on ZFS. However
we have not had any incident for years with this setup, the only change
is the recent addition of CDC.
I can see that in CASSANDRA-19564, the MemtablePostFlush thread was
stuck on the unlink() syscall. But in our case, it was stuck here:
"MemtablePostFlush:1" daemon prio=5 tid=237 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:289)
local variable:
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$RegisteredSignal#1
at
org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:282)
at
org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:306)
local variable:
org.apache.cassandra.utils.concurrent.CountDownLatch$Async#7
at
org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:338)
at
org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1094)
local variable:
org.apache.cassandra.db.ColumnFamilyStore$PostFlush#7
at
org.apache.cassandra.db.ColumnFamilyStore$PostFlush.call(ColumnFamilyStore.java:1077)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
local variable: org.apache.cassandra.concurrent.FutureTask#2555
local variable: org.apache.cassandra.concurrent.FutureTask#2555
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
local variable:
org.apache.cassandra.concurrent.SingleThreadExecutorPlus#5
local variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
local variable: io.netty.util.concurrent.FastThreadLocalThread#11
local variable: org.apache.cassandra.concurrent.FutureTask#2555
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
local variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
local variable: io.netty.util.concurrent.FastThreadLocalRunnable#11
at java.lang.Thread.run(Thread.java:829)
local variable: io.netty.util.concurrent.FastThreadLocalThread#11
Which made me believe that they are not the same issue.
Thanks for the info though.
BTW, if you still have access to that oddly behaving ZFS filesystem,
have a look at the free space (should have at least 20% free) and the
fragmentation ratio in "zpool list". If neither is an issue, check
whether deduplication is enabled or there's a large number of snapshots.
These can have significant impact on file deletion performance on ZFS.
Also worth checking the disks, I have seen broken disks that stuck on
some operations, e.g. when a specific sector is being read, and this
will certainly affect the filesystem behaviour.
Cheers,
Bowen
On 05/11/2024 16:41, Jon Haddad wrote:
> I ran into this a few months ago, and in my case I tracked it down to
> an issue with ZFS not unlinking commitlogs properly.
>
> https://issues.apache.org/jira/browse/CASSANDRA-19564
>
> On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov
> <ne...@gmail.com> wrote:
>
> I am speaking about a thread dump (stack traces for all threads),
> not a heap dump. The heap dump should contain thread stacks info.
> Thread dump (stack traces) is small and does not have sensitive info.
>
> Regards,
> Dmitry
>
> On Tue, 5 Nov 2024 at 13:53, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> It's about 18GB in size and may contain a huge amount of
> sensitive data (e.g. all the pending writes), so I can't share
> it. However, if there's any particular piece of information
> you would like to have, I'm more than happy to extract the
> info from the dump and and share it here.
>
> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>> Hi Bowen, would it be possible to share a full thread dump?
>>
>> Regards,
>> Dmitry
>>
>> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user
>> <us...@cassandra.apache.org> wrote:
>>
>> Hi all,
>>
>> We have a cluster running Cassandra 4.1.1. We are seeing
>> the memtable flush randomly getting stuck. This has
>> happened twice in the last 10 days, to two different
>> nodes in the same cluster. This started to happen after
>> we enabled CDC, and each time it got stuck, there was at
>> least one repair running involving the affected node.
>>
>> The signs of the stuck memtable flush is most obvious
>> from the "StatusLogger" logs.
>>
>> At the beginning, the MemtablePostFlush and
>> MemtableFlushWriter got stuck, they have 1 and 2 active
>> tasks each, and a small number of pending tasks.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:65 - Pool Name
>> Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ReadStage 0 0
>> 34052333 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1019777 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MutationStage 0 0
>> 14930764 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Repair#61 0 0
>> 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - GossipStage 0 0
>> 889452 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 19 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Repair-Task 0
>> 0 65 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 44447834 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Native-Transport-Requests
>> 0 0 8967921 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 10 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 11 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>> 0 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 240 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 -
>> AntiEntropyStage 1 1773
>> 120067 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>>
>> The number of pending tasks slowly grows larger over
>> time, and the number of completed tasks do not increase
>> at all.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:65 - Pool Name
>> Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - ReadStage 0 0
>> 39905462 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1170100 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MutationStage 0 0
>> 16976992 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#76 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#74 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#72 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - GossipStage 0 0
>> 1026877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair-Task 0
>> 0 83 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 51122191 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 54 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 53 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#67 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#82 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Repair#61 0 0
>> 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 20 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Native-Transport-Requests
>> 0 0 10145020 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>> 0 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 996 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
>> StatusLogger.java:69 -
>> AntiEntropyStage 1 9694
>> 120067 0 0
>>
>> Then, things start to get worse, the MutationStage also
>> got stuck.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:65 - Pool Name
>> Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ReadStage 0 0
>> 40186426 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - CompactionExecutor 0
>> 0 1171479 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MutationStage
>> 128 201414 16988212 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#76 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#74 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#72 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - GossipStage 0 0
>> 1032988 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair-Task 0
>> 0 85 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - RequestResponseStage 0
>> 0 51307497 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ValidationExecutor 0
>> 0 36651 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtablePostFlush 1
>> 54 47856 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtableFlushWriter 2
>> 53 21781 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - CacheCleanupExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#67 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MemtableReclaimMemory 0
>> 0 21877 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#84 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - PendingRangeCalculator 0
>> 0 177 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#82 0 0
>> 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Repair#61 0 0
>> 1344 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - SecondaryIndexManagement 0
>> 0 1 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - HintsDispatcher 0
>> 0 20 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Native-Transport-Requests
>> 0 0 10194549 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - MigrationStage 0
>> 0 5 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0
>> 0 0 21769 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - ViewBuildExecutor 0
>> 0 0 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 - InternalResponseStage 0
>> 0 1087 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
>> StatusLogger.java:69 -
>> AntiEntropyStage 1 10364
>> 120067 0 0
>>
>> Writes on this node starts to timeout and fail. But if
>> left untouched, it's only gonna get worse, and eventually
>> lead to JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see
>> that both of the MemtableFlushWriter threads are stuck on
>> line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as
>> blocking progress, permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait
>> for them all to complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck
>> here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the
>> commitLogUpperBound to be set, and so that waiters
>> // on this task can rely on all prior
>> flushes being complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since
>> this started to happen shortly after we enabled CDC on
>> the nodes, and each time repair was running. But we have
>> not been able to reproduce this in a testing cluster, and
>> don't know what's the next step to troubleshoot this
>> issue. So I'm posting it in the mailing lists and hoping
>> someone may know something about it or point me to the
>> right direction.
>>
>> p.s.: sorry about posting this to both the user & dev
>> mailing lists. It's an end-user related issue but
>> involves Cassandra internals, so I can't decide which one
>> is best suited.
>>
>> Cheers,
>> Bowen
>>
>>
>>
>>
>> --
>> Dmitry Konstantinov
>
>
>
> --
> Dmitry Konstantinov
>
Re: Unexplained stuck memtable flush
Posted by Jon Haddad <jo...@rustyrazorblade.com>.
I ran into this a few months ago, and in my case I tracked it down to an
issue with ZFS not unlinking commitlogs properly.
https://issues.apache.org/jira/browse/CASSANDRA-19564
On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov <ne...@gmail.com>
wrote:
> I am speaking about a thread dump (stack traces for all threads), not a
> heap dump. The heap dump should contain thread stacks info.
> Thread dump (stack traces) is small and does not have sensitive info.
>
> Regards,
> Dmitry
>
> On Tue, 5 Nov 2024 at 13:53, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> It's about 18GB in size and may contain a huge amount of sensitive data
>> (e.g. all the pending writes), so I can't share it. However, if there's any
>> particular piece of information you would like to have, I'm more than happy
>> to extract the info from the dump and and share it here.
>> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>>
>> Hi Bowen, would it be possible to share a full thread dump?
>>
>> Regards,
>> Dmitry
>>
>> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> We have a cluster running Cassandra 4.1.1. We are seeing the memtable
>>> flush randomly getting stuck. This has happened twice in the last 10 days,
>>> to two different nodes in the same cluster. This started to happen after we
>>> enabled CDC, and each time it got stuck, there was at least one repair
>>> running involving the affected node.
>>>
>>> The signs of the stuck memtable flush is most obvious from the
>>> "StatusLogger" logs.
>>>
>>> At the beginning, the MemtablePostFlush and MemtableFlushWriter got
>>> stuck, they have 1 and 2 active tasks each, and a small number of pending
>>> tasks.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
>>> Pool Name Active Pending Completed Blocked
>>> All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> ReadStage 0 0 34052333
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> CompactionExecutor 0 0 1019777
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> MutationStage 0 0 14930764
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> MemtableReclaimMemory 0 0 21877
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> PendingRangeCalculator 0 0 177
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> Repair#61 0 0 1344
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> GossipStage 0 0 889452
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> SecondaryIndexManagement 0 0 1
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> HintsDispatcher 0 0 19
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> Repair-Task 0 0 65
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> RequestResponseStage 0 0 44447834
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> Native-Transport-Requests 0 0 8967921
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> MigrationStage 0 0 5
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> MemtableFlushWriter 2 10 21781
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> MemtablePostFlush 1 11 47856
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> Sampler 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> ValidationExecutor 0 0 36651
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> ViewBuildExecutor 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> InternalResponseStage 0 0 240
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> AntiEntropyStage 1 1773 120067
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>>> CacheCleanupExecutor 0 0 0
>>> 0 0
>>>
>>> The number of pending tasks slowly grows larger over time, and the
>>> number of completed tasks do not increase at all.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
>>> Pool Name Active Pending Completed Blocked
>>> All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> ReadStage 0 0 39905462
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> CompactionExecutor 0 0 1170100
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> MutationStage 0 0 16976992
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#76 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#74 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#72 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> GossipStage 0 0 1026877
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair-Task 0 0 83
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> RequestResponseStage 0 0 51122191
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> ValidationExecutor 0 0 36651
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> MemtablePostFlush 1 54 47856
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> MemtableFlushWriter 2 53 21781
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> CacheCleanupExecutor 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#67 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> MemtableReclaimMemory 0 0 21877
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> PendingRangeCalculator 0 0 177
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#82 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Repair#61 0 0 1344
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> SecondaryIndexManagement 0 0 1
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> HintsDispatcher 0 0 20
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Native-Transport-Requests 0 0 10145020
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> MigrationStage 0 0 5
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> Sampler 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>> ViewBuildExecutor 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>> InternalResponseStage 0 0 996
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>>> AntiEntropyStage 1 9694 120067
>>> 0 0
>>>
>>> Then, things start to get worse, the MutationStage also got stuck.
>>>
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
>>> Pool Name Active Pending Completed Blocked
>>> All Time Blocked
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> ReadStage 0 0 40186426
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> CompactionExecutor 0 0 1171479
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> MutationStage 128 201414 16988212
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#76 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#74 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#72 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> GossipStage 0 0 1032988
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair-Task 0 0 85
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> RequestResponseStage 0 0 51307497
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> ValidationExecutor 0 0 36651
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> MemtablePostFlush 1 54 47856
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> MemtableFlushWriter 2 53 21781
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> CacheCleanupExecutor 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#67 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> MemtableReclaimMemory 0 0 21877
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#84 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> PendingRangeCalculator 0 0 177
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#82 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Repair#61 0 0 1344
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> SecondaryIndexManagement 0 0 1
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> HintsDispatcher 0 0 20
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Native-Transport-Requests 0 0 10194549
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> MigrationStage 0 0 5
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> Sampler 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> PerDiskMemtableFlushWriter_0 0 0 21769
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> ViewBuildExecutor 0 0 0
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> InternalResponseStage 0 0 1087
>>> 0 0
>>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>>> AntiEntropyStage 1 10364 120067
>>> 0 0
>>>
>>> Writes on this node starts to timeout and fail. But if left untouched,
>>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>>
>>> By inspecting the heap dump created at OOM, we can see that both of the
>>> MemtableFlushWriter threads are stuck on line 1190
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>>> in the ColumnFamilyStore.java:
>>>
>>> // mark writes older than the barrier as blocking progress,
>>> permitting them to exceed our memory limit
>>> // if they are stuck waiting on it, then wait for them all
>>> to complete
>>> writeBarrier.markBlocking();
>>> writeBarrier.await(); // <----------- stuck here
>>>
>>> And the MemtablePostFlush thread is stuck on line 1094
>>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>>> in the same file.
>>>
>>> try
>>> {
>>> // we wait on the latch for the commitLogUpperBound to
>>> be set, and so that waiters
>>> // on this task can rely on all prior flushes being
>>> complete
>>> latch.await(); // <----------- stuck here
>>> }
>>> Our top suspect is CDC interacting with repair, since this started to
>>> happen shortly after we enabled CDC on the nodes, and each time repair was
>>> running. But we have not been able to reproduce this in a testing cluster,
>>> and don't know what's the next step to troubleshoot this issue. So I'm
>>> posting it in the mailing lists and hoping someone may know something about
>>> it or point me to the right direction.
>>>
>>> p.s.: sorry about posting this to both the user & dev mailing lists.
>>> It's an end-user related issue but involves Cassandra internals, so I can't
>>> decide which one is best suited.
>>>
>>> Cheers,
>>> Bowen
>>>
>>>
>>>
>>
>> --
>> Dmitry Konstantinov
>>
>>
>
> --
> Dmitry Konstantinov
>
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
I am speaking about a thread dump (stack traces for all threads), not a
heap dump. The heap dump should contain thread stacks info.
Thread dump (stack traces) is small and does not have sensitive info.
Regards,
Dmitry
On Tue, 5 Nov 2024 at 13:53, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> It's about 18GB in size and may contain a huge amount of sensitive data
> (e.g. all the pending writes), so I can't share it. However, if there's any
> particular piece of information you would like to have, I'm more than happy
> to extract the info from the dump and and share it here.
> On 05/11/2024 13:01, Dmitry Konstantinov wrote:
>
> Hi Bowen, would it be possible to share a full thread dump?
>
> Regards,
> Dmitry
>
> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Hi all,
>>
>> We have a cluster running Cassandra 4.1.1. We are seeing the memtable
>> flush randomly getting stuck. This has happened twice in the last 10 days,
>> to two different nodes in the same cluster. This started to happen after we
>> enabled CDC, and each time it got stuck, there was at least one repair
>> running involving the affected node.
>>
>> The signs of the stuck memtable flush is most obvious from the
>> "StatusLogger" logs.
>>
>> At the beginning, the MemtablePostFlush and MemtableFlushWriter got
>> stuck, they have 1 and 2 active tasks each, and a small number of pending
>> tasks.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
>> Pool Name Active Pending Completed Blocked
>> All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> ReadStage 0 0 34052333
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> CompactionExecutor 0 0 1019777
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> MutationStage 0 0 14930764
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> MemtableReclaimMemory 0 0 21877
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> PendingRangeCalculator 0 0 177
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> Repair#61 0 0 1344
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> GossipStage 0 0 889452
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> SecondaryIndexManagement 0 0 1
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> HintsDispatcher 0 0 19
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> Repair-Task 0 0 65
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> RequestResponseStage 0 0 44447834
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> Native-Transport-Requests 0 0 8967921
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> MigrationStage 0 0 5
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> MemtableFlushWriter 2 10 21781
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> MemtablePostFlush 1 11 47856
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> PerDiskMemtableFlushWriter_0 0 0 21769
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> ValidationExecutor 0 0 36651
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> InternalResponseStage 0 0 240
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> AntiEntropyStage 1 1773 120067
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
>> CacheCleanupExecutor 0 0 0
>> 0 0
>>
>> The number of pending tasks slowly grows larger over time, and the number
>> of completed tasks do not increase at all.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
>> Pool Name Active Pending Completed Blocked
>> All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> ReadStage 0 0 39905462
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> CompactionExecutor 0 0 1170100
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> MutationStage 0 0 16976992
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#76 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#74 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#72 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> GossipStage 0 0 1026877
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair-Task 0 0 83
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> RequestResponseStage 0 0 51122191
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> ValidationExecutor 0 0 36651
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> MemtablePostFlush 1 54 47856
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> MemtableFlushWriter 2 53 21781
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> CacheCleanupExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#67 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> MemtableReclaimMemory 0 0 21877
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> PendingRangeCalculator 0 0 177
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#82 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Repair#61 0 0 1344
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> SecondaryIndexManagement 0 0 1
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> HintsDispatcher 0 0 20
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Native-Transport-Requests 0 0 10145020
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> MigrationStage 0 0 5
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
>> PerDiskMemtableFlushWriter_0 0 0 21769
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>> ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>> InternalResponseStage 0 0 996
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
>> AntiEntropyStage 1 9694 120067
>> 0 0
>>
>> Then, things start to get worse, the MutationStage also got stuck.
>>
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
>> Pool Name Active Pending Completed Blocked
>> All Time Blocked
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> ReadStage 0 0 40186426
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> CompactionExecutor 0 0 1171479
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> MutationStage 128 201414 16988212
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#76 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#74 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#72 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> GossipStage 0 0 1032988
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair-Task 0 0 85
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> RequestResponseStage 0 0 51307497
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> ValidationExecutor 0 0 36651
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> MemtablePostFlush 1 54 47856
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> MemtableFlushWriter 2 53 21781
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> CacheCleanupExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#67 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> MemtableReclaimMemory 0 0 21877
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#84 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> PendingRangeCalculator 0 0 177
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#82 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Repair#61 0 0 1344
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> SecondaryIndexManagement 0 0 1
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> HintsDispatcher 0 0 20
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Native-Transport-Requests 0 0 10194549
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> MigrationStage 0 0 5
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> Sampler 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> PerDiskMemtableFlushWriter_0 0 0 21769
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> InternalResponseStage 0 0 1087
>> 0 0
>> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
>> AntiEntropyStage 1 10364 120067
>> 0 0
>>
>> Writes on this node starts to timeout and fail. But if left untouched,
>> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>>
>> By inspecting the heap dump created at OOM, we can see that both of the
>> MemtableFlushWriter threads are stuck on line 1190
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
>> in the ColumnFamilyStore.java:
>>
>> // mark writes older than the barrier as blocking progress,
>> permitting them to exceed our memory limit
>> // if they are stuck waiting on it, then wait for them all to
>> complete
>> writeBarrier.markBlocking();
>> writeBarrier.await(); // <----------- stuck here
>>
>> And the MemtablePostFlush thread is stuck on line 1094
>> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
>> in the same file.
>>
>> try
>> {
>> // we wait on the latch for the commitLogUpperBound to be
>> set, and so that waiters
>> // on this task can rely on all prior flushes being
>> complete
>> latch.await(); // <----------- stuck here
>> }
>> Our top suspect is CDC interacting with repair, since this started to
>> happen shortly after we enabled CDC on the nodes, and each time repair was
>> running. But we have not been able to reproduce this in a testing cluster,
>> and don't know what's the next step to troubleshoot this issue. So I'm
>> posting it in the mailing lists and hoping someone may know something about
>> it or point me to the right direction.
>>
>> p.s.: sorry about posting this to both the user & dev mailing lists. It's
>> an end-user related issue but involves Cassandra internals, so I can't
>> decide which one is best suited.
>>
>> Cheers,
>> Bowen
>>
>>
>>
>
> --
> Dmitry Konstantinov
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Bowen Song via user <us...@cassandra.apache.org>.
It's about 18GB in size and may contain a huge amount of sensitive data
(e.g. all the pending writes), so I can't share it. However, if there's
any particular piece of information you would like to have, I'm more
than happy to extract the info from the dump and and share it here.
On 05/11/2024 13:01, Dmitry Konstantinov wrote:
> Hi Bowen, would it be possible to share a full thread dump?
>
> Regards,
> Dmitry
>
> On Tue, 5 Nov 2024 at 12:12, Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> Hi all,
>
> We have a cluster running Cassandra 4.1.1. We are seeing
> the memtable flush randomly getting stuck. This has happened twice
> in the last 10 days, to two different nodes in the same cluster.
> This started to happen after we enabled CDC, and each time it got
> stuck, there was at least one repair running involving the
> affected node.
>
> The signs of the stuck memtable flush is most obvious from the
> "StatusLogger" logs.
>
> At the beginning, the MemtablePostFlush and MemtableFlushWriter
> got stuck, they have 1 and 2 active tasks each, and a small number
> of pending tasks.
>
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:65 - Pool Name Active
> Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ReadStage
> 0 0 34052333 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - CompactionExecutor
> 0 0 1019777 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MutationStage
> 0 0 14930764 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - GossipStage
> 0 0 889452 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - HintsDispatcher 0
> 0 19 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Repair-Task 0
> 0 65 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - RequestResponseStage
> 0 0 44447834 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Native-Transport-Requests
> 0 0 8967921 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtableFlushWriter 2
> 10 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - MemtablePostFlush 1
> 11 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - InternalResponseStage 0
> 0 240 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - AntiEntropyStage 1
> 1773 120067 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
>
> The number of pending tasks slowly grows larger over time, and the
> number of completed tasks do not increase at all.
>
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:65 - Pool Name Active
> Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - ReadStage
> 0 0 39905462 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - CompactionExecutor
> 0 0 1170100 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MutationStage
> 0 0 16976992 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#76 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#74 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#72 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - GossipStage
> 0 0 1026877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair-Task 0
> 0 83 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - RequestResponseStage
> 0 0 51122191 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtablePostFlush 1
> 54 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtableFlushWriter 2
> 53 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#67 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#82 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - HintsDispatcher 0
> 0 20 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Native-Transport-Requests
> 0 0 10145020 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - InternalResponseStage 0
> 0 996 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942
> StatusLogger.java:69 - AntiEntropyStage 1
> 9694 120067 0 0
>
> Then, things start to get worse, the MutationStage also got stuck.
>
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:65 - Pool Name Active
> Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ReadStage
> 0 0 40186426 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - CompactionExecutor
> 0 0 1171479 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MutationStage 128
> 201414 16988212 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#76 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#74 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#72 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - GossipStage
> 0 0 1032988 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair-Task 0
> 0 85 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - RequestResponseStage
> 0 0 51307497 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ValidationExecutor 0
> 0 36651 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtablePostFlush 1
> 54 47856 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtableFlushWriter 2
> 53 21781 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - CacheCleanupExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#67 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MemtableReclaimMemory 0
> 0 21877 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#84 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - PendingRangeCalculator 0
> 0 177 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#82 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Repair#61 0
> 0 1344 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - SecondaryIndexManagement 0
> 0 1 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - HintsDispatcher 0
> 0 20 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Native-Transport-Requests
> 0 0 10194549 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - MigrationStage 0
> 0 5 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - Sampler 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0
> 0 21769 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - ViewBuildExecutor 0
> 0 0 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - InternalResponseStage 0
> 0 1087 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974
> StatusLogger.java:69 - AntiEntropyStage 1
> 10364 120067 0 0
>
> Writes on this node starts to timeout and fail. But if left
> untouched, it's only gonna get worse, and eventually lead to JVM
> OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both
> of the MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking
> progress, permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them
> all to complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the
> commitLogUpperBound to be set, and so that waiters
> // on this task can rely on all prior flushes
> being complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started
> to happen shortly after we enabled CDC on the nodes, and each time
> repair was running. But we have not been able to reproduce this in
> a testing cluster, and don't know what's the next step to
> troubleshoot this issue. So I'm posting it in the mailing lists
> and hoping someone may know something about it or point me to the
> right direction.
>
> p.s.: sorry about posting this to both the user & dev mailing
> lists. It's an end-user related issue but involves Cassandra
> internals, so I can't decide which one is best suited.
>
> Cheers,
> Bowen
>
>
>
>
> --
> Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Dmitry Konstantinov <ne...@gmail.com>.
Hi Bowen, would it be possible to share a full thread dump?
Regards,
Dmitry
On Tue, 5 Nov 2024 at 12:12, Bowen Song via user <us...@cassandra.apache.org>
wrote:
> Hi all,
>
> We have a cluster running Cassandra 4.1.1. We are seeing the memtable
> flush randomly getting stuck. This has happened twice in the last 10 days,
> to two different nodes in the same cluster. This started to happen after we
> enabled CDC, and each time it got stuck, there was at least one repair
> running involving the affected node.
>
> The signs of the stuck memtable flush is most obvious from the
> "StatusLogger" logs.
>
> At the beginning, the MemtablePostFlush and MemtableFlushWriter got stuck,
> they have 1 and 2 active tasks each, and a small number of pending tasks.
>
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> ReadStage 0 0 34052333
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> CompactionExecutor 0 0 1019777
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> MutationStage 0 0 14930764
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> GossipStage 0 0 889452
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> HintsDispatcher 0 0 19
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> Repair-Task 0 0 65
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> RequestResponseStage 0 0 44447834
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 8967921
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> MemtableFlushWriter 2 10 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> MemtablePostFlush 1 11 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> InternalResponseStage 0 0 240
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> AntiEntropyStage 1 1773 120067
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 03:50:15,224 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
>
> The number of pending tasks slowly grows larger over time, and the number
> of completed tasks do not increase at all.
>
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> ReadStage 0 0 39905462
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> CompactionExecutor 0 0 1170100
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> MutationStage 0 0 16976992
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#76 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#74 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#72 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> GossipStage 0 0 1026877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair-Task 0 0 83
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> RequestResponseStage 0 0 51122191
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> MemtablePostFlush 1 54 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> MemtableFlushWriter 2 53 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#67 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#82 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> HintsDispatcher 0 0 20
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 10145020
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,941 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
> InternalResponseStage 0 0 996
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 16:33:05,942 StatusLogger.java:69 -
> AntiEntropyStage 1 9694 120067
> 0 0
>
> Then, things start to get worse, the MutationStage also got stuck.
>
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> ReadStage 0 0 40186426
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> CompactionExecutor 0 0 1171479
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> MutationStage 128 201414 16988212
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#76 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#74 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#72 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> GossipStage 0 0 1032988
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair-Task 0 0 85
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> RequestResponseStage 0 0 51307497
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> ValidationExecutor 0 0 36651
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> MemtablePostFlush 1 54 47856
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> MemtableFlushWriter 2 53 21781
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#67 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 21877
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#84 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 177
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#82 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Repair#61 0 0 1344
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 1
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> HintsDispatcher 0 0 20
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 10194549
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> MigrationStage 0 0 5
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 21769
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> InternalResponseStage 0 0 1087
> 0 0
> INFO [ScheduledTasks:1] 2024-11-03 17:05:40,974 StatusLogger.java:69 -
> AntiEntropyStage 1 10364 120067
> 0 0
>
> Writes on this node starts to timeout and fail. But if left untouched,
> it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of the
> MemtableFlushWriter threads are stuck on line 1190
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190>
> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress,
> permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all to
> complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094
> <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094>
> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to be
> set, and so that waiters
> // on this task can rely on all prior flushes being
> complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to
> happen shortly after we enabled CDC on the nodes, and each time repair was
> running. But we have not been able to reproduce this in a testing cluster,
> and don't know what's the next step to troubleshoot this issue. So I'm
> posting it in the mailing lists and hoping someone may know something about
> it or point me to the right direction.
>
> p.s.: sorry about posting this to both the user & dev mailing lists. It's
> an end-user related issue but involves Cassandra internals, so I can't
> decide which one is best suited.
>
> Cheers,
> Bowen
>
>
>
--
Dmitry Konstantinov
Re: Unexplained stuck memtable flush
Posted by Jeff Jirsa <jj...@gmail.com>.
> On Nov 5, 2024, at 4:12 AM, Bowen Song via user <us...@cassandra.apache.org> wrote:
>
> Writes on this node starts to timeout and fail. But if left untouched, it's only gonna get worse, and eventually lead to JVM OOM and crash.
>
> By inspecting the heap dump created at OOM, we can see that both of the MemtableFlushWriter threads are stuck on line 1190 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1190> in the ColumnFamilyStore.java:
>
> // mark writes older than the barrier as blocking progress, permitting them to exceed our memory limit
> // if they are stuck waiting on it, then wait for them all to complete
> writeBarrier.markBlocking();
> writeBarrier.await(); // <----------- stuck here
>
> And the MemtablePostFlush thread is stuck on line 1094 <https://github.com/apache/cassandra/blob/8d91b469afd3fcafef7ef85c10c8acc11703ba2d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L1094> in the same file.
>
> try
> {
> // we wait on the latch for the commitLogUpperBound to be set, and so that waiters
> // on this task can rely on all prior flushes being complete
> latch.await(); // <----------- stuck here
> }
> Our top suspect is CDC interacting with repair, since this started to happen shortly after we enabled CDC on the nodes, and each time repair was running. But we have not been able to reproduce this in a testing cluster, and don't know what's the next step to troubleshoot this issue. So I'm posting it in the mailing lists and hoping someone may know something about it or point me to the right direction.
>
Wouldn’t be completely surprised if CDC or repair somehow has a barrier, I’ve also seen similar behavior pre-3.0 with “very long running read commands” that have a barrier on the memtable that prevent release.
You’ve got the heap (great, way better than most people debugging), are you able to navigate through it and look for references to that memtable or other things holding a barrier?