[jira] [Commented] (CASSANDRA-10047) nodetool aborts when attempting to cleanup a keyspace with no ranges

["Laurent Demontreuille (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-10047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033391#comment-15033391 ] 

Laurent Demontreuille commented on CASSANDRA-10047:
---------------------------------------------------

Hi, 

Same issue for Cassandra 2.1.9. 
Nodetool cleanup crashed with message : "Aborted cleaning up atleast one column family in keyspace KEYSPACE, check server logs for more information."
Log shows : "INFO  [RMI TCP Connection(527941)-127.0.0.1] 2015-12-01 08:14:13,124 CompactionManager.java:362 - Cleanup cannot run before a node has joined the ring"

Nodetool status from faulty node shows every neighbors up and normal. 

Nodetool compactionstats shows cleanup on and active : 

Don't know if it matters but we recently exclude a node. But, as far as we can tell, there is no trace of it anymore (doesn't show in nodetool status, in system.peers or in netstat). 

Stop/start node + "nodetool cleanup" = same result. 

We have 12 keyspaces in our Cassandra cluster and confirm that error is happening only on KS without range in local DC. 

Example (error happens for DC3) : 
CREATE KEYSPACE KS1 WITH replication = {'class': 'NetworkTopologyStrategy', 'DC3': '1'}  AND durable_writes = true;  -- OK
CREATE KEYSPACE KS2 WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '1', 'DC2': '1', 'DC3': '1'}  AND durable_writes = true; --OK
CREATE KEYSPACE KS3 WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND durable_writes = true; -- KO
CREATE KEYSPACE KS4 WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '2'}  AND durable_writes = true; -- OK
CREATE KEYSPACE fluks WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'}  AND durable_writes = true; -- KO

Hope this helps. If you need further informations, please ask. 


> nodetool aborts when attempting to cleanup a keyspace with no ranges
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-10047
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10047
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 2.1.8
>            Reporter: Russell Bradberry
>            Priority: Minor
>             Fix For: 2.1.x
>
>
> When running nodetool cleanup in a DC that has no ranges for a keyspace, nodetool will abort with the following message when attempting to cleanup that keyspace:
> {code}
> Aborted cleaning up atleast one column family in keyspace ks, check server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> 	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
> 	at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
> {code}
> The error messages in the logs are :
> {code}
> CompactionManager.java:370 - Cleanup cannot run before a node has joined the ring
> {code}
> This behavior prevents subsequent keyspaces from getting cleaned up. The error message is also misleading as it suggests that the only reason  a node may not have ranges for a keyspace is because it has yet to join the ring.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)