You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/12/18 17:19:19 UTC
[jira] Created: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Provide way to remove nodes from gossip entirely
------------------------------------------------
Key: CASSANDRA-644
URL: https://issues.apache.org/jira/browse/CASSANDRA-644
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 0.9
As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine reassigned CASSANDRA-644:
--------------------------------------
Assignee: Jaakko Laine
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793240#action_12793240 ]
Jonathan Ellis commented on CASSANDRA-644:
------------------------------------------
> We could run a check periodically and remove all nodes from gossiper that have been silent for certain period of time, and do not have token associated with it.
that sounds fine.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804461#action_12804461 ]
Jaakko Laine commented on CASSANDRA-644:
----------------------------------------
Committed to trunk. Let's wait a while and commit to 0.5 if nothing breaks.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-644:
-----------------------------------
Attachment: (was: 644-remove-from-gossip-on-removetoken.patch)
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-644:
-----------------------------------
Attachment: 644.patch
yeah, I decided it is not worth the trouble, but you're right, it is better to remove from FD too. attached new version.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-644:
-----------------------------------
Attachment: 644-remove-from-gossip-on-removetoken.patch
yeah, that should probably be fixed. Attached patch should do the trick. I'll fix the other part tomorrow.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 644-remove-from-gossip-on-removetoken.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793178#action_12793178 ]
Jaakko Laine commented on CASSANDRA-644:
----------------------------------------
> connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!
Actually this cannot happen since gossip SYN carries cluster ID and only messages from the same cluster are handled.
I think removing fat clients automatically is OK. We could run a check periodically and remove all nodes from gossiper that have been silent for certain period of time, and do not have token associated with it. I'm not sure in what kind of situations this kind of clients are used, but if a node does not participate in storage and has been dead for an hour, there should be no harm in removing it. It will be back in gossip anyway immediately if it reappears. Cleaning up fat clients automatically would prevent gossip data from growing "rapidly" if there are many clients coming and going from different IP addresses.
As for "proper" storage nodes: These are likely to be more stable than clients, so probably it is best to have removetoken and/or other manual commands to take care of this.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-644) Provide way to remove
nodes from gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801117#action_12801117 ]
Jaakko Laine edited comment on CASSANDRA-644 at 1/16/10 7:23 AM:
-----------------------------------------------------------------
patch attached:
- removeToken now removes node from gossip
- fat clients (nodes without token) are removed from gossip after 1 hour of inactivity
- added justRemovedEndPoints to gossip to prevent removed nodes from reappearing immediately. Removed nodes are kept here for RING_DELAY period, during which time new joins from them are ignored. It takes a while for remove token gossip to propagate to all nodes. During this time some nodes will continue to gossip about the just-now-being-removed-node, while others have already removed it.
- fixed a bug related to removeToken command which allowed node's own token being removed.
was (Author: jaakko):
patch attached:
- removeToken now removes node from gossip
- fat clients (nodes without token) are removed from gossip after 1 hour of inactivity
- added justRemovedEndPoints to gossip to prevent removed nodes from reappearing immediately. Removed nodes are kept here for RING_DELAY period, during which time new joins from them are ignored. It takes a while for remove token gossip to propagate to all nodes. During this time some nodes will continue to gossip about the just-now-being-removed-node, while others have already removed it.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804540#action_12804540 ]
Jonathan Ellis commented on CASSANDRA-644:
------------------------------------------
could you also update CHANGES?
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803729#action_12803729 ]
Jonathan Ellis commented on CASSANDRA-644:
------------------------------------------
+1
are you comfortable committing this to 0.5 branch?
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804522#action_12804522 ]
Hudson commented on CASSANDRA-644:
----------------------------------
Integrated in Cassandra #334 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/334/])
fixed broken test.
modify removetoken to remove node from gossip. remove fat clients automatically after 1h of inactivity. patch by jaakko, reviewed by jbellis.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803954#action_12803954 ]
Jaakko Laine commented on CASSANDRA-644:
----------------------------------------
Before committing to 0.5, it would be good if other people tested this as well.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801844#action_12801844 ]
Jonathan Ellis commented on CASSANDRA-644:
------------------------------------------
shouldn't this also r/m from FD.arrivalSamples?
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-644:
-----------------------------------
Attachment: 644.patch
patch attached:
- removeToken now removes node from gossip
- fat clients (nodes without token) are removed from gossip after 1 hour of inactivity
- added justRemovedEndPoints to gossip to prevent removed nodes from reappearing immediately. Removed nodes are kept here for RING_DELAY period, during which time new joins from them are ignored. It takes a while for remove token gossip to propagate to all nodes. During this time some nodes will continue to gossip about the just-now-being-removed-node, while others have already removed it.
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
> Attachments: 644.patch
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-644:
-------------------------------------
Affects Version/s: 0.5
Issue Type: Bug (was: Improvement)
(I originally marked this as Improvement, but isn't decommission/removeToken not cleaning out from gossip a Bug?)
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.9
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-644) Provide way to remove nodes from
gossip entirely
Posted by "Jaakko Laine (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaakko Laine updated CASSANDRA-644:
-----------------------------------
Attachment: (was: 644.patch)
> Provide way to remove nodes from gossip entirely
> ------------------------------------------------
>
> Key: CASSANDRA-644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-644
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Affects Versions: 0.5
> Reporter: Jonathan Ellis
> Assignee: Jaakko Laine
> Priority: Minor
> Fix For: 0.6
>
>
> As reported in CASSANDRA-634, "Now that we're gossiping about dead nodes as well, gossip digest continues to grow without boundary when nodes come and go. This information will never disappear as it will be propagated to new nodes no matter how old and obsolete it is. To counter this, we need some mechanism to (1) either remove dead node from endpointstateinfo or (2) at some point stop to gossip about it, or both."
> This is also seen when using "fat clients" that participate in the gossip ring; if a client leaves and does not come back it stays in the gossip forever. (This can be confusing if the client does start up again, connecting to a _different_ cluster, but the old one notices it is back and starts gossiping to it again!)
> I would prefer to leave management of these things explicit; 3 days is long enough that the fat client problem in particular needs another solution, and if it needs another solution then that can become the only solution. :)
> So I would be in favor of removeToken clearing out gossip entries, and also adding a command to remove an endpoint from the gossip ring that does not have a token associated with it (like fat clients). A command to ask "what are all the known gossip hosts" would also be useful, since nodeprobe ring only includes nodes w/ tokens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.