You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yasuharu Goto (JIRA)" <ji...@apache.org> on 2014/07/01 03:38:24 UTC
[jira] [Commented] (CASSANDRA-7469) RejectedExecutionException
causes orphan SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048407#comment-14048407 ]
Yasuharu Goto commented on CASSANDRA-7469:
------------------------------------------
I agree with you. I'm goint to upgrade my cluster.
> RejectedExecutionException causes orphan SSTables
> -------------------------------------------------
>
> Key: CASSANDRA-7469
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7469
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Yasuharu Goto
> Priority: Minor
>
> I noticed that some old SSTables are not deleted and remaining in data dir. They are never compacted.
> {code}
> ./ks2-cf2-he-9690-Data.db
> ./ks2-cf2-he-9691-Data.db
> ./ks2-cf2-he-9679-Data.db <- current version id
> ./ks2-cf2-he-205-Data.db <- very old version id
> ./ks2-cf2-he-201-Data.db
> ./ks2-cf2-he-202-Data.db
> ./ks2-cf2-he-203-Data.db
> {code}
> And I noticed that some RejectedExecutionException causes these orphan SSTables.
> {code}
> ...
> INFO 18:51:45,323 DRAINING: starting drain process
> INFO 18:51:45,324 Stop listening to thrift clients
> ...
> # This compaction is not finished. Terminated by following Exception and nerver retried. So these SSTables are not deleted eternally.
> INFO 18:51:46,512 Compacting [SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-205-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-203-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-202-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-201-Data.db')]
> ...
> # This compaction is finished. They don't get to be orphans.
> INFO 18:51:46,641 Compacting [SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-90-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-89-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-88-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-87-Data.db')]
> INFO 18:51:46,736 Compacted to [/var/cassandra/data/ks1/cf1/ks1-cf1-he-91-Data.db,]. 370,606 to 317,566 (~85% of original) bytes for 193 keys at 3.187943MB/s. Time: 95ms.
> INFO 18:51:46,836 DRAINED
> ERROR 18:51:49,807 Exception in thread Thread[CompactionExecutor:1927,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32b5a2c6 rejected from org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@32d18f2c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3043]
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2013)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
> at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:629)
> at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67)
> at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:806)
> at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:358)
> at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
> at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> INFO 18:52:54,010 Cassandra shutting down...
> {code}
> As a result of log servey, we found some orphan SSTables caused by RejectExcutionException.
> Maybe I can fix each orphan files by nodetool refresh.
> But I'd like to ask if this is a problem that already has been solved in early release.
--
This message was sent by Atlassian JIRA
(v6.2#6252)