You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Pankaj Kumar (Jira)" <ji...@apache.org> on 2020/07/02 12:16:00 UTC
[jira] [Updated] (CASSANDRA-15917) LEAK Detected in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pankaj Kumar updated CASSANDRA-15917:
-------------------------------------
Description:
Hi
We are using cassandra docker environment. So 1 out of 3 cassandra, cassandra docker is restarting continuously. This issue is happening in production setup.
Then checked system logs and debug logs and found these traceback occur multiple times
ERROR [Reference-Reaper] 2020-06-29 14:40:20,800 Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@53581121) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@194489638:/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-626-big-Index.db was not released before the reference was garbage collectedDEBUG [SSTableBatchOpen:8] 2020-06-29 14:40:23,698 SSTableReader.java:515 - Opening /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-240-big (0.050KiB)ERROR [SSTableBatchOpen:5] 2020-06-29 14:40:20,801 DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
java.lang.NullPointerException: null
at org.apache.cassandra.io.sstable.format.SSTableReader.getFilename(SSTableReader.java:697) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:543) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.5.jar:3.11.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]
Can somebody help me out here as it customer deployed setup
was:
Hi
We are using cassandra docker environment. So 1 out of 3 cassandra, cassandra docker is restarting continuously. This issue is happening in production setup.
Then checked system logs and debug logs and found these traceback occur multiple times
ERROR [Reference-Reaper] 2020-06-29 14:40:20,800 Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@53581121) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@194489638:/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-626-big-Index.db was not released before the reference was garbage collectedDEBUG [SSTableBatchOpen:8] 2020-06-29 14:40:23,698 SSTableReader.java:515 - Opening /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-240-big (0.050KiB)ERROR [SSTableBatchOpen:5] 2020-06-29 14:40:20,801 DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
java.lang.NullPointerException: null
at org.apache.cassandra.io.sstable.format.SSTableReader.getFilename(SSTableReader.java:697) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:543) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.5.jar:3.11.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]
> LEAK Detected in Cassandra
> --------------------------
>
> Key: CASSANDRA-15917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15917
> Project: Cassandra
> Issue Type: Bug
> Reporter: Pankaj Kumar
> Priority: Normal
>
> Hi
> We are using cassandra docker environment. So 1 out of 3 cassandra, cassandra docker is restarting continuously. This issue is happening in production setup.
> Then checked system logs and debug logs and found these traceback occur multiple times
> ERROR [Reference-Reaper] 2020-06-29 14:40:20,800 Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@53581121) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@194489638:/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-626-big-Index.db was not released before the reference was garbage collectedDEBUG [SSTableBatchOpen:8] 2020-06-29 14:40:23,698 SSTableReader.java:515 - Opening /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-240-big (0.050KiB)ERROR [SSTableBatchOpen:5] 2020-06-29 14:40:20,801 DebuggableThreadPoolExecutor.java:239 - Error in ThreadPoolExecutor
> java.lang.NullPointerException: null
> at org.apache.cassandra.io.sstable.format.SSTableReader.getFilename(SSTableReader.java:697) ~[apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:543) ~[apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:385) ~[apache-cassandra-3.11.5.jar:3.11.5]
> at org.apache.cassandra.io.sstable.format.SSTableReader$3.run(SSTableReader.java:570) ~[apache-cassandra-3.11.5.jar:3.11.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_232]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_232]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]
>
> Can somebody help me out here as it customer deployed setup
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org