[jira] [Commented] (CASSANDRA-6245) "nodetool refresh" design is unsafe

["Robert Coli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859230#comment-13859230 ] 

Robert Coli commented on CASSANDRA-6245:
----------------------------------------

{quote}
"don't copy sstables over existing in-use sequence numbers"
{quote}
Can only be a meaningful caveat to the use of "nodetool refresh" if we actually give operators this instruction anywhere. From what I can tell, the only documentation for refresh is :

http://www.datastax.com/documentation/cassandra/2.0/webhelp/cassandra/tools/toolsNodetool_r.html
"
refresh keyspace <table>
Loads newly placed SSTables on to the system without restart.
"

Additionally, an quote from #cassandra showing how some operators clearly do not understand the basics of SSTable naming vis a vis sequence :

{noformat}
17:06 < rcoli> cornedbeefsand: look at the versions of your sstables, via "find"
17:06 < rcoli> it's a component of the name.
17:06 < cornedbeefsand> right, i did that, but which component?
17:07 < cornedbeefsand> e.g., jenkins_preview_data-timeseries_NormalizedTSDouble-hf-23-Data.db
17:07 < cornedbeefsand> v23?
17:07 < rcoli> -hf-
17:07 < rcoli> 23 is a sequence number
{noformat}

Would you accept a patch which either :

1) moves "refresh" tasks so that they run from a subdirectory in the CF directory named "refresh"
or
2) checks for file existence before creation, and inflates the sequence by one if the file exists

?

> "nodetool refresh" design is unsafe
> -----------------------------------
>
>                 Key: CASSANDRA-6245
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6245
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Robert Coli
>            Priority: Minor
>
> CASSANDRA-2991 added a "nodetool refresh" feature by which Cassandra is able to discover non-live SSTables in the datadir and make them live.
> It does this by :
> 1) looking for SSTable files in the data dir
> 2) renaming SSTables it finds into the current SSTable id sequence
> This implementation is exposed to a race with a chance of silent data loss.
> 1) Node's SSTable id sequence is on sstable #2, the next table to flush will get "2" as its numeric part
> 2) Copy SSTable with "2" as its numeric part into data dir
> 3) nodetool flush
> 4) notice that your "2" SSTable has been silently overwritten by a just-flushed "2" SSTable
> 5) nodetool refresh would still succeed, but would now be a no-op
> A simple solution would be to create a subdirectory of the datadir called "refresh/" to serve as the location to refresh from.
> Alternately/additionally, there is probably not really a compelling reason for Cassandra to completely ignore existing files at write time.. a check for existing files at a given index and inflating the index to avoid overwriting them them seems trivial and inexpensive. I will gladly file a JIRA for this change in isolation if there is interest.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)