You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dmitry Konstantinov (Jira)" <ji...@apache.org> on 2026/02/24 20:51:00 UTC
[jira] [Comment Edited] (CASSANDRA-21186) Test failure: org.apache.cassandra.index.accord.RouteIndexTest
[ https://issues.apache.org/jira/browse/CASSANDRA-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18060861#comment-18060861 ]
Dmitry Konstantinov edited comment on CASSANDRA-21186 at 2/24/26 8:50 PM:
--------------------------------------------------------------------------
I have been able to get a heap dump for the failed test.
!dominators_top_level.png|width=500!
!histogram_retained_heap.png|width=500!
!metrics_cleaner_thread.png|width=500!
!total_number_of_threads.png|width=500!
!accoud_journal_threads.png|width=500!
!accord_executor_threads.png|width=500!
a preliminary cause: too many Accord-related threads are created and destroyed in a short period of time during the test, so sometimes PhantomReference cleaner thread for thread local metrics cannot keep the pace. To be continued ...
was (Author: dnk):
I have been able to get a heap dump for the failed test.
!dominators_top_level.png|width=500!
!histogram_retained_heap.png|width=500!
!metrics_cleaner_thread.png|width=500!
!total_number_of_threads.png|width=500!
!accoud_journal_threads.png|width=500!
!accord_executor_threads.png|width=500!
a preliminary cause: too many threads are created and destroyed in a short period of time, so PhantomReference cleaner thread for thread local metrics cannot keep the pace. To be continued ...
> Test failure: org.apache.cassandra.index.accord.RouteIndexTest
> --------------------------------------------------------------
>
> Key: CASSANDRA-21186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21186
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.x
>
> Attachments: RouteIndexTest_thread_dump.txt, TEST-org.apache.cassandra.index.accord.RouteIndexTest.log.xz, accord_executor_threads.png, accoud_journal_threads.png, dominators_top_level.png, histogram_retained_heap.png, metrics_cleaner_thread.png, total_number_of_threads.png
>
>
> The test is almost always fail now on the main CI, last example: https://ci-cassandra.apache.org/job/Cassandra-trunk/2410/#showFailuresLink
> It happens for different JDKs and configurations (compression, cdc, oa)
> Usually it fails with a JUnit test timeout:
> {code}
> Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
> {code}
> Thanks to CASSANDRA-21172 we can see now a thread dump ( [^RouteIndexTest_thread_dump.txt] ) to clarify where did we stuck.
> https://ci-cassandra.apache.org/job/Cassandra-trunk/2410/cloudbees-pipeline-explorer/?line=450882
> test-cdc jdk11 3/20
> we can observe Java OOM as well as the stuck main thread:
> {code}
> [junit-timeout] "main" #1 prio=5 os_prio=0 cpu=9993.84ms elapsed=480.27s tid=0x00007fc3e401f000 nid=0xf44 waiting on condition [0x00007fc3eabd0000]
> [junit-timeout] java.lang.Thread.State: WAITING (parking)
> [junit-timeout] at jdk.internal.misc.Unsafe.park(java.base@11.0.27/Native Method)
> [junit-timeout] at java.util.concurrent.locks.LockSupport.park(java.base@11.0.27/LockSupport.java:323)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:322)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:300)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:306)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.Awaitable$AsyncAwaitable.await(Awaitable.java:338)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.Awaitable$Defaults.awaitThrowUncheckedOnInterrupt(Awaitable.java:131)
> [junit-timeout] at org.apache.cassandra.utils.concurrent.Awaitable$AbstractAwaitable.awaitThrowUncheckedOnInterrupt(Awaitable.java:235)
> [junit-timeout] at org.apache.cassandra.tcm.log.LocalLog$Async.runOnce(LocalLog.java:760)
> [junit-timeout] at org.apache.cassandra.tcm.log.LocalLog$Async$AwaitCommit.get(LocalLog.java:868)
> [junit-timeout] at org.apache.cassandra.tcm.log.LocalLog$Async$AwaitCommit.get(LocalLog.java:860)
> [junit-timeout] at org.apache.cassandra.tcm.log.LocalLog$Async.awaitAtLeast(LocalLog.java:709)
> [junit-timeout] at org.apache.cassandra.tcm.AbstractLocalProcessor.commit(AbstractLocalProcessor.java:106)
> [junit-timeout] at org.apache.cassandra.tcm.ClusterMetadataService$SwitchableProcessor.commit(ClusterMetadataService.java:973)
> [junit-timeout] at org.apache.cassandra.tcm.Processor.commit(Processor.java:50)
> [junit-timeout] at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:613)
> [junit-timeout] at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:578)
> [junit-timeout] at org.apache.cassandra.ServerTestUtils$ResettableClusterMetadataService.reset(ServerTestUtils.java:385)
> [junit-timeout] at org.apache.cassandra.ServerTestUtils.resetCMS(ServerTestUtils.java:357)
> [junit-timeout] at org.apache.cassandra.cql3.CQLTester.afterTest(CQLTester.java:534)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org