You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Nick Bailey (JIRA)" <ji...@apache.org> on 2010/12/23 21:20:46 UTC
[jira] Created: (CASSANDRA-1900) Make removetoken force always work
Make removetoken force always work
----------------------------------
Key: CASSANDRA-1900
URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
Project: Cassandra
Issue Type: Improvement
Components: Tools
Affects Versions: 0.7.0 rc 2, 0.7.0
Reporter: Nick Bailey
Fix For: 0.7.1
The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: (was: 1900.txt)
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: 1900.txt
Patch to log a warning instead of throwing an exception when the node is already leaving.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: 1900-v3.txt
v3 fixes an NPE if you try to removetoken force before calling removetoken.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986192#action_12986192 ]
Jonathan Ellis commented on CASSANDRA-1900:
-------------------------------------------
can we initialize replicatingNodes to an empty list/set so we don't risk NPEs instead?
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986069#action_12986069 ]
Jonathan Ellis commented on CASSANDRA-1900:
-------------------------------------------
I'm going to need some more explanation, possibly in improved comments. :)
What I understand so far: "removetoken force" is the forceRemoveCompletion method.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: 1900-v4.txt
v4 initializes replicatingNodes to avoid NPEs.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900-v4.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1900:
--------------------------------------
Remaining Estimate: 6h
Original Estimate: 6h
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900-v4.txt, 1900.txt
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986083#action_12986083 ]
Brandon Williams commented on CASSANDRA-1900:
---------------------------------------------
Right. The problem here is that decommission and removetoken are completely different operations (decom moves its own data off, removetoken has the replicas restore the count.) So if a decommission fails, you need to invoke the removetoken path, but since decommission puts the node's state in STATE_LEAVING, you can't call removetoken currently. This patch removes that restriction. If, for some reason, the streaming during removetoken fails, then finally you need to removetoken force.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986599#action_12986599 ]
Hudson commented on CASSANDRA-1900:
-----------------------------------
Integrated in Cassandra-0.7 #207 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/207/])
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900-v4.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: 1900.txt
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-1900:
-------------------------------------------
Assignee: Brandon Williams
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986477#action_12986477 ]
Jonathan Ellis commented on CASSANDRA-1900:
-------------------------------------------
+1
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900-v4.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1900) Make removetoken force always work
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1900:
----------------------------------------
Attachment: 1900-v2.txt
v2 clarifies the error message about the possible use of 'removetoken force' if someone tries to removetoken a node twice.
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1900) Make removetoken force always
work
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986479#action_12986479 ]
Jonathan Ellis commented on CASSANDRA-1900:
-------------------------------------------
(it would still be good to comment handleStateRemoving + forceRemoveCompletion)
> Make removetoken force always work
> ----------------------------------
>
> Key: CASSANDRA-1900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1900
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Affects Versions: 0.7.0 rc 2, 0.7.0 rc 3
> Reporter: Nick Bailey
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1900-v2.txt, 1900-v3.txt, 1900-v4.txt, 1900.txt
>
>
> The 'removetoken force' command was intended for forcing a removal to complete when the removal stalls for some reason. The most likely reason being a streaming failure causing the node to wait forever for streams to complete.
> The command should be updated so that it can force a removal in any case. For example a node that was decommissioned but killed before a LEFT status was broadcasted. This leaves the node in a permanent 'leaving' state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.