[jira] [Comment Edited] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans

["Brandon Williams (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559766#comment-13559766 ] 

Brandon Williams edited comment on CASSANDRA-4021 at 1/22/13 4:49 PM:
----------------------------------------------------------------------

Saw this in the wild again.  It was encountered multiple times on the same component set preventing startup, in the following order: CompressionInfo, CompressionInfo, Index, CompressionInfo, CompressionInfo.  After that, it was finally able to startup so we weren't able to see which components existed or did not, but it looks like it does make incremental progress on the deletions with each restart.  I don't want to paper over something without understanding what's going on here, but it's highly annoying and visible when this error is fatal.  We aren't making progress here too quickly, so maybe in the interim we should add more logging around this and log something at ERROR instead of dying.
                
      was (Author: brandon.williams):
    Saw this in the wild again.  It was encountering multiple times on the same component set preventing startup, in the following order: CompressionInfo, CompressionInfo, Index, CompressionInfo, CompressionInfo.  After that, it was finally able to startup so we weren't able to see which components existed or did not, but it looks like it does make incremental progress on the deletions with each restart.  I don't want to paper over something without understanding what's going on here, but it's highly annoying and visible when this error is fatal.  We aren't making progress here too quickly, so maybe in the interim we should add more logging around this and log something at ERROR instead of dying.
                  
> CFS.scrubDataDirectories tries to delete nonexistent orphans
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-4021
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4021
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7 beta 2
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>            Priority: Minor
>              Labels: datastax_qa
>         Attachments: 4021.txt, node1.log
>
>
> The check only looks for a missing data file, then deletes all other components, however it's possible for the data file and another component to be missing, causing an error:
> {noformat}
>  WARN 17:19:28,765 Removing orphans for /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db]
> ERROR 17:19:28,766 Exception encountered during startup
> java.lang.AssertionError: attempted to delete non-existing file system-HintsColumnFamily-hd-24492-Index.db
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
>         at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
>         at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
>         at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
>         at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
> java.lang.AssertionError: attempted to delete non-existing file system-HintsColumnFamily-hd-24492-Index.db
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
>         at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
>         at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
>         at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
>         at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
>         at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
> Exception encountered during startup: attempted to delete non-existing file system-HintsColumnFamily-hd-24492-Index.db
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira