You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jaakko Laine (JIRA)" <ji...@apache.org> on 2009/11/23 10:22:39 UTC
[jira] Created: (CASSANDRA-572) handle old gossip properly
handle old gossip properly
--------------------------
Key: CASSANDRA-572
URL: https://issues.apache.org/jira/browse/CASSANDRA-572
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.5
Reporter: Jaakko Laine
Fix For: 0.5
(1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
(2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-572:
-----------------------------------
Attachment: use-same-APstate-for-all-node-state-gossip.patch
OK, here's a patch that uses same state name (NODE_STATE) to gossip all movement information. Format is (BOOTSTRAPPING|NORMAL|LEAVING|LEFT)|token.
The main things caused by this modification to the state machine were:
(1) When a node is bootstrapping, we should clear pending ranges for this endpoint, as well as remove it from token metadata. These checks are not strictly necessary (I think), but are there to help transition from LEAVING -> BOOTSTRAPPING in case we missed LEFT due to network partition.
(2) For handleStateLeaving and handleStateLeft remove pending ranges for this endpoint before doing anything else. If we missed NORMAL, there might be obsolete pending ranges from BOOTSTRAP. Distant possibility, but possibility nonetheless.
Following additional check is not directly related to gossip format change and could happen even using the current model. This is a very unlikely event, but in a large (say, 200+ nodes) multi-DC cluster with lots of node movement, this could very well happen even with relatively short DC-to-DC network outage:
(1) Added a check to handleStateLeaving and handleStateLeft for the case that a node has made NORMAL -> LEAVING -> LEFT -> BOOTSTRAP -> NORMAL -> LEAVING [->LEFT] movement cycle without us seeing the intermediate stages. In this case we have information for the old token and now the node is leaving _new_ token. We cannot simply assert this, as it is possible this happens.
Now of course this already touches the subject what conditions we must take care of and what should be left to operators to handle. Some of them (like removing all references to the endpoint before continuing to handle bootstrapping) are questionable and might relax safety precautions, but if we do not do that, a modest 30s network outage might cause us not to see STATE_LEFT and we'd end up having strange pending ranges.
I don't expect this patch to be included as it is, but let's see what people think of this gossip change and then discuss what checks should be made :)
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785155#action_12785155 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
Funny thing, I was just thinking about the same thing during breakfast. Have to eat more often :)
The problem with this is that handling state changes will become somewhat more complex as we must be prepared to handle transitions between any two states in any order. Current gossip model leaves a trace of what the node has node, and even in the face of network partitions we can "play back" the transitions when they eventually arrive. That is, if a node moves, we will still see LEAVING, LEFT, BOOTSTRAPPING and NORMAL and construct token metadata according to that. If we only have one value to represent node's current state, we might go from, say NORMAL to NORMAL, or even LEFT to LEAVING without seeing any of the intermediate steps. Of course this can be done, but needs extra care. Don't know how much, though. Might very well be that in the end this would be better than the current way.
But even this would not remove the need to handle old application state correctly. If a node enters the ring when another node is just LEAVING or LEFT, that state will be the first one to be seen, and it must be ignored since there is nothing that can be done if NORMAL has not been seen. I think the real cause is there in any case, so we can't avoid fixing the symptoms that arrive with it.
I'll try this out now that I'm working on the gossiping part anyway so we'll have some more insight on what it would look like.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-572) handle old gossip
properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787093#action_12787093 ]
Jonathan Ellis edited comment on CASSANDRA-572 at 12/7/09 8:37 PM:
-------------------------------------------------------------------
The fundamental question is, is it ever dangerous to overwrite intermediate states, such that a node who has been down or partitioned does something broken when it gets the latest [partial] information?
> if we do not do that, a modest 30s network outage might cause us not to see STATE_LEFT
Trying to figure out what you're referring to here... We can't miss a STATE_LEFT from a node that is leaving permanently since that will remain its last state and will be gossiped forever. And if we miss a STATE_LEFT from a node that is moving, a down node that comes back up later will get the new token location and "snap" it to the right spot immediately.
I assume that as usual I am slow on the uptake here. :)
was (Author: jbellis):
The fundamental question is, is it ever dangerous to overwrite intermediate states, such that a node who has been down or partitioned does something broken when it gets the latest [partial] information?
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-572:
-----------------------------------
Attachment: (was: 572-handle-old-gossip.patch)
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788717#action_12788717 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
Run into one tricky issue with STATE_LEAVING. I'll check that tomorrow with #603 as they are connected.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792714#action_12792714 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
Sorry about the delay. I did this part again yesterday (somewhat different now after #603), but had insufficient time to test it properly. The patch would seem OK, so as soon as I have tested it a bit more, I'll submit it.
The problem earlier was how to handle state jump to 'leaving'. Now that pending ranges is calculated every time and we know what direction a node was going last time, this is not a problem anymore. You can expect a patch soon.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788484#action_12788484 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
I think I would prefer option (1), since from experience I can attest that trying to write a 100% correct "state machine" for (2) is quite difficult.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788469#action_12788469 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
>From the small experiment (patch) it would seem that it is not too dangerous, just needs a bit of extra care. From #617 it would seem that reducing total amount of data in the gossiper would be beneficial, so current move gossip model needs some attention. There are basically two options:
(1) use one state for all moves, and use tuple/triple in application state string to identify what the node is doing (as suggested by jbellis earlier)
(2) return to the "old" gossip model: one state to broadcast the token and separate small states to express what the token means. The reason for moving out of this model was to make all needed information part of one gossip message and not rely on them arriving at the same time. With small addition to StorageService, we might keep arriving token "somewhere" and only add it to token metadata when the actual mode gossip arrives. This would take care of the race condition and would also reduce the amount of data to be gossiped to bare minimum.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-572) handle old gossip
properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784704#action_12784704 ]
Jaakko Laine edited comment on CASSANDRA-572 at 12/2/09 8:39 AM:
-----------------------------------------------------------------
I don't think this is related to move coordination, but a separate issue. This will happen for every new node that enters the cluster (bootstrapping or not) after a node has already moved. Problem is, handling STATE_LEAVING and STATE_LEFT will cause errors for a node that is not in token metadata, and a node will not end up being there before STATE_NORMAL has been seen for it. Since states are handled in the order they were added, STATE_LEAVING and STATE_LEFT will be handled before BOOTSTRAPPING and NORMAL. LEAVING and LEFT in this case have no meaning as they refer to the old already-gone token, but since application state can only grow, these states will remain part of gossip and new nodes must handle them properly.
Edit: This can wait, as there is no point to add now. Let's check the situation after gossiping analysis is done
was (Author: jaakko):
I don't think this is related to move coordination, but a separate issue. This will happen for every new node that enters the cluster (bootstrapping or not) after a node has already moved. Problem is, handling STATE_LEAVING and STATE_LEFT will cause errors for a node that is not in token metadata, and a node will not end up being there before STATE_NORMAL has been seen for it. Since states are handled in the order they were added, STATE_LEAVING and STATE_LEFT will be handled before BOOTSTRAPPING and NORMAL. LEAVING and LEFT in this case have no meaning as they refer to the old already-gone token, but since application state can only grow, these states will remain part of gossip and new nodes must handle them properly.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783825#action_12783825 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
(Waiting on this while I mull over the rack aware bootstrap mailing list thread, since this is irrelevant if we have a problem there and need to add a "coordinator" node to fix it.)
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-572) handle old gossip
properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788469#action_12788469 ]
Jaakko Laine edited comment on CASSANDRA-572 at 12/10/09 3:23 AM:
------------------------------------------------------------------
>From the small experiment (patch) it would seem that it is not too dangerous, just needs a bit of extra care. From #617 it would seem that reducing total amount of data in the gossiper would be beneficial, so current move gossip model needs some attention. There are basically two options:
(1) use one state for all moves, and use tuple/triple in application state string to identify what the node is doing (as suggested by jbellis earlier)
(2) return to the "old" gossip model: one state to broadcast the token and separate small states to express what the token means. The reason for moving out of this model was to make all needed information part of one gossip message and not rely on them arriving at the same time. With small addition to StorageService, we might keep arriving token "somewhere" and only add it to token metadata when the actual mode gossip arrives. This would take care of the race condition and would also reduce the amount of data to be gossiped to bare minimum.
> I assume that as usual I am slow on the uptake here. :)
I think the problem is in my ability to clearly express an idea :)
was (Author: jaakko):
From the small experiment (patch) it would seem that it is not too dangerous, just needs a bit of extra care. From #617 it would seem that reducing total amount of data in the gossiper would be beneficial, so current move gossip model needs some attention. There are basically two options:
(1) use one state for all moves, and use tuple/triple in application state string to identify what the node is doing (as suggested by jbellis earlier)
(2) return to the "old" gossip model: one state to broadcast the token and separate small states to express what the token means. The reason for moving out of this model was to make all needed information part of one gossip message and not rely on them arriving at the same time. With small addition to StorageService, we might keep arriving token "somewhere" and only add it to token metadata when the actual mode gossip arrives. This would take care of the race condition and would also reduce the amount of data to be gossiped to bare minimum.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781719#action_12781719 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
I probably described the problem a bit vaguely, another try:
Suppose all nodes in the cluster are running normally and none of them have moved. Their EP state includes STATE_BOOTSTRAPPING (if they were bootstrapped to the ring) and STATE_NORMAL in this order. Suppose there is nodeA, which gets loadbalanced. It goes through leaving, left and bootstrapping back to normal. After this its EP state includes (in this order!) LEAVING, LEFT, BOOTSTRAPPING, NORMAL. The important thing is that EP state can have only one of each state, and they will be handled by other nodes in the order added originally. This is fine for nodes that already were in the ring, as they have seen the _old_ NORMAL state. However, if we ever want to bootstrap another node to the ring, it will cause errors, as they will start to handle states from LEAVING. They have no knowledge of this node's state before they handle NORMAL, so we must handle LEAVING and LEFT properly. That is, we must do nothing if we do not have knowledge of the node.
So this is not related to the new node serving requests, only to handle state gossip from other nodes properly. My term old gossip was obviously badly chosen, perhaps old state information would be more appropriate.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787093#action_12787093 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
The fundamental question is, is it ever dangerous to overwrite intermediate states, such that a node who has been down or partitioned does something broken when it gets the latest [partial] information?
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792523#action_12792523 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
What was the tricky issue? With CASSANDRA-603 and CASSANDRA-639 committed, this needs rebasing at least.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch, use-same-APstate-for-all-node-state-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784704#action_12784704 ]
Jaakko Laine commented on CASSANDRA-572:
----------------------------------------
I don't think this is related to move coordination, but a separate issue. This will happen for every new node that enters the cluster (bootstrapping or not) after a node has already moved. Problem is, handling STATE_LEAVING and STATE_LEFT will cause errors for a node that is not in token metadata, and a node will not end up being there before STATE_NORMAL has been seen for it. Since states are handled in the order they were added, STATE_LEAVING and STATE_LEFT will be handled before BOOTSTRAPPING and NORMAL. LEAVING and LEFT in this case have no meaning as they refer to the old already-gone token, but since application state can only grow, these states will remain part of gossip and new nodes must handle them properly.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781448#action_12781448 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
See CASSANDRA-559 for more discussion.
Jaakko, can you check on the mailing list to see if anyone actually needs this, before we commit to making our lives (a little bit) harder for 0.5 -> 0.5.1.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-572:
-----------------------------------
Attachment: (was: use-same-APstate-for-all-node-state-gossip.patch)
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781464#action_12781464 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
This feels like we're solving the wrong problem. A bootstrapping node should not be added to the list of nodes clients can connect to (however you want to manage that) until after it is done bootstrapping. I'd rather add
if (StorageService.instance().isBootstrapMode())
throw new UnavailableException()
to the appropriate StorageProxy methods.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-572:
-----------------------------------
Attachment: 572-03-unit-tests.patch
572-02-use-one-move-state.patch
572-01-fix-index-out-of-bounds.patch
01:
Fix ArrayIndex exception in case the first gossip a new node sees is leaving or left.
02:
Use one state for all movement related gossip
Fix also some bugs related to pending ranges on the leaving node itself
03:
A bunch of unit tests for node movement
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-01-fix-index-out-of-bounds.patch, 572-02-use-one-move-state.patch, 572-03-unit-tests.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781449#action_12781449 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
Sorry, I was thrown off by the title, assuming old==0.4. But that is actually not what this is about.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-572) handle old gossip properly
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-572:
-----------------------------------
Attachment: 572-handle-old-gossip.patch
(1) Modified StorageMetadata.onChange to handle STATE_LEAVING and STATE_LEFT only if isMember is true
(2) We might fix this by simply making sure we're calling getNaturalEndpoints for members only, but imho it is better to fix at the source.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-572) handle old gossip properly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784943#action_12784943 ]
Jonathan Ellis commented on CASSANDRA-572:
------------------------------------------
ok, I understand now. for some reason I had a really hard time wrapping my head around this.
trying to make sure we ignore old applicationstate correctly in all cases seems like fixing the symptom instead of the real cause.
wouldn't it be simpler to have a single STATE aplicationstate object whose value would be a json (or whatever) tuple of [NORMAL|LEAVING|LEFT|BOOTSTRAPPING],ARG+? that way only the most recent one would be present in the gossiper.
> handle old gossip properly
> --------------------------
>
> Key: CASSANDRA-572
> URL: https://issues.apache.org/jira/browse/CASSANDRA-572
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Reporter: Jaakko Laine
> Fix For: 0.5
>
> Attachments: 572-handle-old-gossip.patch
>
>
> (1) If a node has been moving in the ring, further bootstraps by other nodes will cause errors as they are handling STATE_LEAVING gossip without having such member in token metadata.
> (2) When a node bootstraps, it handles all ep states in the order they happen to arrive. If the first one to arrive has moved in the past (that is, it has STATE_LEAVING in its ep state), getNaturalEndpoint will throw ArrayIndexOutOfBounds exception as sortedTokens.size() == 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.