You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/09/09 00:07:57 UTC
[jira] Created: (CASSANDRA-435) unbootstrap
unbootstrap
-----------
Key: CASSANDRA-435
URL: https://issues.apache.org/jira/browse/CASSANDRA-435
Project: Cassandra
Issue Type: New Feature
Reporter: Jonathan Ellis
Fix For: 0.5
add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-435:
-----------------------------------
Attachment: 0004-simplify-getChangedRangesForLeaving.txt
Patch 0004 modifies getChangedRangesForLeaving. IMHO this way is simpler and more effective, but if other people feel otherwise, this patch may as well be left out.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: (was: 0003-r-m-unused-bootstrap-directory.txt)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752785#action_12752785 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
note that this is actually simpler than bootstrap, since we don't have to do anticompaction first -- just move all data, and we're done
the tricky part is not serving invalid data while this happens. see CASSANDRA-397 -- I don't think piling special cases into the token ring is the way to go, but Sandeep may disagree. :)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-435:
----------------------------------------
Assignee: Jonathan Ellis
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774462#action_12774462 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
With 525 done I think the rest looks like this
* unbootstrapping node gossips STATE_LEAVING: token
* other nodes use that to set pending ranges on the nodes that will be responsible for the replica ranges it has, similar to UpdateBoostrapRanges (but you can't just call strategy.getPendingRanges since we're adding ranges to existing nodes rather than a new one -- can we just use strategy.getAddressRanges and figure out which node will get each Range?)
* stream over the data
* gossip STATE_LEFT. other nodes remove it from the gossip network (this automatically makes new node ranges look like what they would have been w/ pendingranges merged). probably worth looking at old MemberShipcleaner code that was r/m'd in r828130.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776441#action_12776441 ]
Jaakko Laine commented on CASSANDRA-435:
----------------------------------------
I think there are two problems with this patchset:
(1) After the leaving node gossips STATE_LEFT, its gossiper continues to broadcast application state, which still includes NORMAL (and LEAVING) from previous states. When other nodes get STATE_LEFT, they remove all information about this node, which will cause them to interpret the following gossip message as new join. Don't know if stopping the gossiper from broadcasting anything more after STATE_LEFT, would cause other nodes to convict it. This would be the simplest solution, but probably need to do something more creative to better handle all possible state transitions.
(2) When all data for a pending range has been streamed, the receiving node should inform about the completion, otherwise pending ranges won't be removed. The simplest way would be to just gossip its token again, as this will cause pending ranges for this endpoint to be removed. Problem is the receiving end does not know when whole data transfer is complete as tables are streamed one by one. Another possibility would be for the leaving node to gossip these endpoints as a part of its STATE_LEFT message.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777020#action_12777020 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
committed 05 and bootstrap flag fix
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752826#action_12752826 ]
Sandeep Tata commented on CASSANDRA-435:
----------------------------------------
:-)
I think it is exactly as complicated as bootstrap.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777483#action_12777483 ]
Hudson commented on CASSANDRA-435:
----------------------------------
Integrated in Cassandra #257 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/257/])
tokenMetadata should be updated before removing obsolete pending ranges, and unset bootstrapped flag when decommission is complete. patch by Jaakko Laine and jbellis for
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775424#action_12775424 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
you're right, the code I posted only handles primary endpoints for the ranges the leaving node handles. (but we still don't need to make it as complicated as comparing sets of ranges/endpoints)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: 0004-add-leaving-mode.txt
0003-r-m-unused-bootstrap-directory.txt
0002-move-more-generic-streaming-code-into-Streaming.java.txt
0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777016#action_12777016 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
> When node is decommisioned, should it delete its saved token? If the same node is brought back online without manual data deletion, it will enter the ring without proper bootstrap.
You're right. So we should delete its "bootstrapped" flag. Leaving the token is harmless and may be useful to someone.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: 0003-add-leaving-mode.txt
0002-move-more-generic-streaming-code-into-Streaming.java.txt
0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776495#action_12776495 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
1) simplest thing is to leave the node in gossip, but remove it from tokenmetadata. this means gossip can tell nodes that were down temporarily about the new state, as designed, w/o nasty hacks. if/when someone actually needs to add/remove so many nodes that clutter of internal Gossiper state becomes a problem, he can patch it. :)
2) added ARS.removeObsoletePendingRanges that idempotently removes pending ranges that are no longer needed because the ring has updated to reflect the pending state change. this approach reduces the chance of accidentally removing something we shouldn't.
3) applied 04 patch
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776972#action_12776972 ]
Hudson commented on CASSANDRA-435:
----------------------------------
Integrated in Cassandra #256 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/256/])
simplify getChangedRangesForLeaving. patch by Jaakko Laine; reviewed by jbellis for
add leaving mode
patch by jbellis; reviewed by Jaakko Laine for
move more generic streaming code into Streaming.java
patch by jbellis; reviewed by Jaakko Laine for
clean up transfer code from BMVH; move to Streaming.java
patch by jbellis; reviewed by Jaakko Laine for
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-435:
-----------------------------------
Attachment: 435-modify-update_leaving_ranges.txt
I think updateLeavingRanges should do:
(1) get all ranges the leaving is currently storing
(2) get current natural endpoints for those ranges
(3) get natural endpoints for these ranges when leaving node is removed
(4) compare these two lists and add pending ranges for all nodes that are new in the lists (that is, taking responsibility for these ranges now that one node is leaving).
I think we need to do this through replication strategy as by simply looking at token list we cannot deduce what other constraints need to be satisfied. The replica list might change by more than one node if rack awareness (or other external considerations) are taken into account.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: (was: 0002-move-more-generic-streaming-code-into-Streaming.java.txt)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776917#action_12776917 ]
Jaakko Laine commented on CASSANDRA-435:
----------------------------------------
When node is decommisioned, should it delete its saved token? If the same node is brought back online without manual data deletion, it will enter the ring without proper bootstrap.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774978#action_12774978 ]
Jaakko Laine commented on CASSANDRA-435:
----------------------------------------
I think that should work. I suppose STATE_LEFT will cause the token to be removed from token ring? Is STATE_LEFT necessary or can the node directly start to bootstrap to the new location after it has gossiped LEAVING and streamed all data?
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: (was: 0004-add-leaving-mode.txt)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-435.
--------------------------------------
Resolution: Fixed
committed w/ above changes
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: (was: 435-0.diff)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774981#action_12774981 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
we want the separate state so we can support removing nodes from the ring entirely, as well as load balancing. for some reason people keep asking for this and it's easy enough to support given that we're doing LB anyway. :)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771128#action_12771128 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
Here is what we need to do.
TokenMetadata currently uses a slightly fragile combination of a Set of nodes known to be bootstrapping, and a Map of their tokens -> inetaddress, to determine the ranges Bootstrap nodes are concerned with (for CASSANDRA-497).
We need to change that to instead have a Map of Range -> inetaddress, representing "these are ranges that the given node doesn't own yet, but will, so send updates in that range there as well as its current destinations."
That allows us to use the same structure for bootstrap (new node X gets these ranges, where before it had none) and unbootstrap (existing node Y gets X's ranges, as well as its existing ones).
Then actually implementing unbootstrap is just wiring up the streaming from one node to another. See comments in the header of BootStrapper.java for the different moving parts involved in bootstrap; the process is basically the same. (Even to requiring anticompaction to split out the different replica ranges.)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775173#action_12775173 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
these patches get us up to but not including the "gossip STATE_LEFT and other nodes remove it" part.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: 435-0.diff
patch to add updateLeavingRanges
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 435-0.diff
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-435:
-----------------------------------
Attachment: 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch
0005
Patch name says it all. tokenMetadata should be updated before removing obsolete pending ranges
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 0005-update-tokenmetadata-before-removing-obsolete-ranges.patch, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-435:
-------------------------------------
Attachment: (was: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt)
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-435) unbootstrap
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775415#action_12775415 ]
Jaakko Laine edited comment on CASSANDRA-435 at 11/10/09 1:53 PM:
------------------------------------------------------------------
I think updateLeavingRanges should do:
(1) get all ranges the leaving node is currently storing
(2) get current natural endpoints for those ranges
(3) get natural endpoints for these ranges when leaving node is removed
(4) compare these two lists and add pending ranges for all nodes that are new in the lists (that is, taking responsibility for these ranges now that one node is leaving).
I think we need to do this through replication strategy as by simply looking at token list we cannot deduce what other constraints need to be satisfied. The replica list might change by more than one node if rack awareness (or other external considerations) are taken into account.
was (Author: jaakko):
I think updateLeavingRanges should do:
(1) get all ranges the leaving is currently storing
(2) get current natural endpoints for those ranges
(3) get natural endpoints for these ranges when leaving node is removed
(4) compare these two lists and add pending ranges for all nodes that are new in the lists (that is, taking responsibility for these ranges now that one node is leaving).
I think we need to do this through replication strategy as by simply looking at token list we cannot deduce what other constraints need to be satisfied. The replica list might change by more than one node if rack awareness (or other external considerations) are taken into account.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776114#action_12776114 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
rewrote, finished up step 4.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775928#action_12775928 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
Started implementing and I see what you mean now -- we don't want to bother sending a range that a node already has because it's already a replica for that range.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-r-m-unused-bootstrap-directory.txt, 0004-add-leaving-mode.txt, 435-0.diff, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-435) unbootstrap
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776452#action_12776452 ]
Jonathan Ellis commented on CASSANDRA-435:
------------------------------------------
1) LEAVING/NORMAL don't get re-broadcast though, so they remain part of the app state but they won't ever be seen again by onChange. So as long as LEFT isn't part of the same gossip as leaving (and the sleep ensures it won't) in practice LEFT is already the last thing gossipped (and we shut down the gossiper right afterwards). This works as planned in my testing. I'm fine with patches improving this but it's not strictly necessary.
2) No, the pending ranges aren't to say "i'm busy streaming" but "when the planned ring changes finish, this will be my range, so I need writes for that range in the meantime so I'm not out of date when the change-over happens." So the right place to remove those is in the STATE_LEFT change. I will add it there.
I will look at the 04 patch shortly.
> unbootstrap
> -----------
>
> Key: CASSANDRA-435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-435
> Project: Cassandra
> Issue Type: New Feature
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-435-clean-up-transfer-code-from-BMVH-move-t.txt, 0002-move-more-generic-streaming-code-into-Streaming.java.txt, 0003-add-leaving-mode.txt, 0004-simplify-getChangedRangesForLeaving.txt, 435-modify-update_leaving_ranges.txt
>
>
> add JMX command to tell a node to decommission itself (moving its data to the next node on the ring)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.