You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2025/12/15 13:45:00 UTC
[jira] [Comment Edited] (CASSANDRA-21074) Flaky test - ExportImportListCompressionDictionaryTest.testExportImportListCompressionDictionary
[ https://issues.apache.org/jira/browse/CASSANDRA-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18045157#comment-18045157 ]
Stefan Miklosovic edited comment on CASSANDRA-21074 at 12/15/25 1:44 PM:
-------------------------------------------------------------------------
Yeah, for some reason, this is happening:
{code}
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:250 - LEAK DETECTED: a reference (class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary) to class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary was not released before the reference was garbage collected
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:290 - Allocate trace class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary:
[junit-timeout] Thread[ScheduledTasks:1,5,ScheduledTasks]
[junit-timeout] at java.base/java.lang.Thread.getStackTrace(Thread.java:1619)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:280)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:201)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:118)
[junit-timeout] at org.apache.cassandra.db.compression.ZstdCompressionDictionary.<init>(ZstdCompressionDictionary.java:61)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionary$Kind$1.createDictionary(CompressionDictionary.java:309)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionary.createFromRow(CompressionDictionary.java:282)
[junit-timeout] at org.apache.cassandra.schema.SystemDistributedKeyspace.retrieveLatestCompressionDictionary(SystemDistributedKeyspace.java:452)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionaryScheduler.refreshDictionaryFromSystemTable(CompressionDictionaryScheduler.java:137)
[junit-timeout] at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:138)
[junit-timeout] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[junit-timeout] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[junit-timeout] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at java.base/java.lang.Thread.run(Thread.java:840)
{code}
I have added a patch here (1) which closes dictionary before we return. That will address the issue Dmitry has filed. I think what happens is that we create "new Ref" in {{ZstdCompressionDictionary}}'s constructor then we populate {{CompositeData}} in {{CompressionDictionaryManager.getCompressionDictionary()}}, we return it, but we just kept that {{ZstdCompressionDictionary}} in a limbo, we never released it anywhere. Then it will be GC'ed but nobody released it. That is what that error message is about, I believe.
However, same is kind of happening when we go to do "refreshDictionaryFromSystemTable", we are not releasing anywhere when that dictionary instance is already in that cache, no? If it is in cache, we just silently throw that instance away but we are not releasing anywhere. If that instance was indeed added to the cache then it would be released when cache is invalidated or similar.
I think that helped to not fail it as this ticket did, (2), there is some other flakiness though I am investigating.
I still see above stacktrace I posted in the logs every now and then though. That is still not resolved.
(1) https://github.com/apache/cassandra/pull/4521
(2) https://app.circleci.com/pipelines/github/instaclustr/cassandra/6167/workflows/d2f6d2e0-db67-4c9f-a1d5-514cbea5e2cb/jobs/465458/tests
EDIT: I have identified one more place we are not closing.
I do not see these logs in CI anymore in multiplexer but that test is flaky as such. Still looking into that.
was (Author: smiklosovic):
Yeah, for some reason, this is happening:
{code}
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:250 - LEAK DETECTED: a reference (class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary) to class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary was not released before the reference was garbage collected
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:290 - Allocate trace class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary:
[junit-timeout] Thread[ScheduledTasks:1,5,ScheduledTasks]
[junit-timeout] at java.base/java.lang.Thread.getStackTrace(Thread.java:1619)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:280)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:201)
[junit-timeout] at org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:118)
[junit-timeout] at org.apache.cassandra.db.compression.ZstdCompressionDictionary.<init>(ZstdCompressionDictionary.java:61)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionary$Kind$1.createDictionary(CompressionDictionary.java:309)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionary.createFromRow(CompressionDictionary.java:282)
[junit-timeout] at org.apache.cassandra.schema.SystemDistributedKeyspace.retrieveLatestCompressionDictionary(SystemDistributedKeyspace.java:452)
[junit-timeout] at org.apache.cassandra.db.compression.CompressionDictionaryScheduler.refreshDictionaryFromSystemTable(CompressionDictionaryScheduler.java:137)
[junit-timeout] at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:138)
[junit-timeout] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[junit-timeout] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[junit-timeout] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at java.base/java.lang.Thread.run(Thread.java:840)
{code}
I have added a patch here (1) which closes dictionary before we return. That will address the issue Dmitry has filed. I think what happens is that we create "new Ref" in {{ZstdCompressionDictionary}}'s constructor then we populate {{CompositeData}} in {{CompressionDictionaryManager.getCompressionDictionary()}}, we return it, but we just kept that {{ZstdCompressionDictionary}} in a limbo, we never released it anywhere. Then it will be GC'ed but nobody released it. That is what that error message is about, I believe.
However, same is kind of happening when we go to do "refreshDictionaryFromSystemTable", we are not releasing anywhere when that dictionary instance is already in that cache, no? If it is in cache, we just silently throw that instance away but we are not releasing anywhere. If that instance was indeed added to the cache then it would be released when cache is invalidated or similar.
I think that helped to not fail it as this ticket did, (2), there is some other flakiness though I am investigating.
I still see above stacktrace I posted in the logs every now and then though. That is still not resolved.
(1) https://github.com/apache/cassandra/pull/4521
(2) https://app.circleci.com/pipelines/github/instaclustr/cassandra/6167/workflows/d2f6d2e0-db67-4c9f-a1d5-514cbea5e2cb/jobs/465458/tests
> Flaky test - ExportImportListCompressionDictionaryTest.testExportImportListCompressionDictionary
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21074
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21074
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Dmitry Konstantinov
> Priority: Normal
> Time Spent: 50m
> Remaining Estimate: 0h
>
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/2353/testReport/junit/org.apache.cas[…]stExportImportListCompressionDictionary_cdc_jdk11_x86_64/|https://ci-cassandra.apache.org/job/Cassandra-trunk/2353/testReport/junit/org.apache.cassandra.tools.nodetool/ExportImportListCompressionDictionaryTest/Tests___test_cdc_jdk11_11_20___testExportImportListCompressionDictionary_cdc_jdk11_x86_64/]
> There is also a suspicious leak report in the test logs:
> {code:java}
> ERROR [Reference-Reaper] 2025-12-12T20:31:35,396 Ref.java:250 - LEAK DETECTED: a reference (class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary) to class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary was not released before the reference was garbage collected ERROR [Reference-Reaper] 2025-12-12T20:31:35,397 Ref.java:290 - Allocate trace class org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary: Thread[RMI TCP Connection(52)-127.0.0.1,5,RMI Runtime] at java.base/java.lang.Thread.getStackTrace(Thread.java:1602) at org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:280) at org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:201) at org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:118) at org.apache.cassandra.db.compression.ZstdCompressionDictionary.<init>(ZstdCompressionDictionary.java:61) at org.apache.cassandra.db.compression.CompressionDictionary$Kind$1.createDictionary(CompressionDictionary.java:309) at org.apache.cassandra.db.compression.CompressionDictionary.createFromRow(CompressionDictionary.java:282) at org.apache.cassandra.schema.SystemDistributedKeyspace.retrieveLatestCompressionDictionary(SystemDistributedKeyspace.java:452) at org.apache.cassandra.db.compression.CompressionDictionaryManager.getCompressionDictionary(CompressionDictionaryManager.java:282) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at java.management/com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at java.management/com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:641) at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.cassandra.audit.AuditLogManager$JmxHandler.invoke(AuditLogManager.java:502) at com.sun.proxy.$Proxy28.getAttribute(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1443) at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307) at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399) at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:637) at java.base/jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359) at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562) at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796) at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org