[jira] [Created] (CASSANDRA-7469) RejectedExecutionException causes orphan SSTables

["Yasuharu Goto (JIRA)" <ji...@apache.org>]

Yasuharu Goto created CASSANDRA-7469:
----------------------------------------

             Summary: RejectedExecutionException causes orphan SSTables
                 Key: CASSANDRA-7469
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7469
             Project: Cassandra
          Issue Type: Bug
          Components: Core
            Reporter: Yasuharu Goto
            Priority: Minor


I noticed that some old SSTables are not deleted and remaining in data dir. They are never compacted.
{code}
 ./ks2-cf2-he-9690-Data.db
 ./ks2-cf2-he-9691-Data.db
 ./ks2-cf2-he-9679-Data.db    <- current version id
 ./ks2-cf2-he-205-Data.db    <- very old version id
 ./ks2-cf2-he-201-Data.db
 ./ks2-cf2-he-202-Data.db
 ./ks2-cf2-he-203-Data.db
{code}

And I noticed that some RejectedExecutionException causes these orphan SSTables.
{code}
...
 INFO 18:51:45,323 DRAINING: starting drain process
 INFO 18:51:45,324 Stop listening to thrift clients

...

 # This compaction is not finished. Terminated by following Exception and nerver retried. So these SSTables are not deleted eternally.

 INFO 18:51:46,512 Compacting [SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-205-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-203-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-202-Data.db'), SSTableReader(path='/var/cassandra/data/ks2/cf2/ks2-cf2-he-201-Data.db')]

...

# This compaction is finished. They don't get to be orphans.

 INFO 18:51:46,641 Compacting [SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-90-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-89-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-88-Data.db'), SSTableReader(path='/var/cassandra/data/ks1/cf1/ks1-cf1-he-87-Data.db')]
 INFO 18:51:46,736 Compacted to [/var/cassandra/data/ks1/cf1/ks1-cf1-he-91-Data.db,].  370,606 to 317,566 (~85% of original) bytes for 193 keys at 3.187943MB/s.  Time: 95ms.


 INFO 18:51:46,836 DRAINED
ERROR 18:51:49,807 Exception in thread Thread[CompactionExecutor:1927,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32b5a2c6 rejected from org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@32d18f2c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3043]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2013)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
        at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:629)
        at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67)
        at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:806)
        at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:358)
        at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
        at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
 INFO 18:52:54,010 Cassandra shutting down...
{code}

As a result of log servey, we found some orphan SSTables caused by RejectExcutionException.

Maybe I can fix each orphan files by nodetool refresh.
But I'd like to ask if this is a problem that already has been solved in early release.



--
This message was sent by Atlassian JIRA
(v6.2#6252)