You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefania (JIRA)" <ji...@apache.org> on 2018/03/22 02:24:00 UTC
[jira] [Comment Edited] (CASSANDRA-14308) Remove invalid SSTables
from interrupted compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408938#comment-16408938 ]
Stefania edited comment on CASSANDRA-14308 at 3/22/18 2:23 AM:
---------------------------------------------------------------
I'm not sure about deleting them, after all the data is still there. I'd say maybe move them to a separate folder and issue an error so that the operator can then delete them manually. I also wonder if it's possible to recreate a stats component with a valid serialization header but dummy stats, for example by using [SerializationHeader.makeWithoutStats()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SerializationHeader.java#L69]. This would only be safe if the schema hasn't changed.
was (Author: stefania):
I'm not sure about deleting them, after all the data is still there. I'd say maybe move them to a separate folder and issue an error so that the operator can then delete them manually. I also wonder if it's possible to recreate a stats component with a valid serialization header but dummy stats, for example by using [SerializationHeader.makeWithoutStats()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SerializationHeader.java#L69].
> Remove invalid SSTables from interrupted compaction
> ---------------------------------------------------
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Jay Zhuang
> Assignee: Jay Zhuang
> Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable won't be cleaned up, which causes "{{Stats component is missing for sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 - Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable /cassandra/data/keyspace/table-id/mc-12345-big
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458) ~[apache-cassandra-3.0.14.jar:3.0.14]
> at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374) ~[apache-cassandra-3.0.14.jar:3.0.14]
> at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533) ~[apache-cassandra-3.0.14.jar:3.0.14]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}' > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org