You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2013/03/24 00:23:15 UTC
[jira] [Commented] (CASSANDRA-5119) RejectedExecutionException
During Compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611895#comment-13611895 ]
Jonathan Ellis commented on CASSANDRA-5119:
-------------------------------------------
We only shut down the scheduled task executor when Cassandra is exiting.
> RejectedExecutionException During Compactions
> ---------------------------------------------
>
> Key: CASSANDRA-5119
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5119
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.6
> Environment: Sun Java 1.6.0_37 64 Bit
> Ubuntu 12.04
> m1.xlarge EC2 instances
> 12 node cluster with 20Gb data on each node
> Cassandra 1.1.6
> Reporter: Arya Goudarzi
>
> Hi, I frequently see the following ERROR and stack trace after CompactionTask.java (line 109) has triggered a compaction job (These are automatic minor compactions. We never trigger major compactions).
> ERROR [CompactionExecutor:881] 2013-01-05 03:40:07,311 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[CompactionExecutor:881,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67)
> at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:826)
> at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:358)
> at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Looking at the code, it seems the thread has been shutdown when it gets to delayedExecute and isShutdown check would be true resulting a call to reject. It is not clear to me why this happens. I am guessing the implication would be the old SSTables would not get deleted which results in disk space usage to keep going up. Restarting the nodes seem to make this go away for a while. I could not find anything more useful in the logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira