You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Nick Bailey (JIRA)" <ji...@apache.org> on 2011/08/16 21:28:27 UTC
[jira] [Created] (CASSANDRA-3041) Move streams data to too many
nodes.
Move streams data to too many nodes.
------------------------------------
Key: CASSANDRA-3041
URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.8.4, 0.7.8
Reporter: Nick Bailey
Fix For: 1.0
When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Nick Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Bailey updated CASSANDRA-3041:
-----------------------------------
Marking this for the next 0.8.x version since the data duplication caused by this bug is data that does actually belong on the nodes, so it can't be removed by cleanup. It either has to be unduplicated with a major compaction or rely on regular compaction.
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.8, 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.5
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103917#comment-13103917 ]
Jonathan Ellis commented on CASSANDRA-3041:
-------------------------------------------
>From IRC:
{quote}
nickmbailey: new patch looks fine, the news.txt update should say move/decomm though
{quote}
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Assignee: Nick Bailey
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch, 3041-v2.txt
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Nick Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Bailey updated CASSANDRA-3041:
-----------------------------------
Fix Version/s: (was: 1.0)
0.8.5
Marking this for the next 0.8.x version since the data duplication caused by this bug is data that does actually belong on the nodes, so it can't be removed by cleanup. It either has to be unduplicated with a major compaction or rely on regular compaction.
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.8, 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.5
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104560#comment-13104560 ]
Hudson commented on CASSANDRA-3041:
-----------------------------------
Integrated in Cassandra-0.8 #327 (See [https://builds.apache.org/job/Cassandra-0.8/327/])
Avoid streaming data to nodes that already have it, on move as well as decommission
patch by Nick Bailey and jbellis for CASSANDRA-3041
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170306
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Assignee: Nick Bailey
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch, 3041-v2.txt
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Nick Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Bailey updated CASSANDRA-3041:
-----------------------------------
Attachment: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch
Simplest change to fix this. I still think we should refactor a lot of that code, but perhaps not in 0.8.
If I have time to figure out a good approach for combining code I'll attach a patch.
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.5
>
> Attachments: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3041:
--------------------------------------
Reviewer: jbellis (was: thepaul)
Priority: Minor (was: Major)
Assignee: Nick Bailey
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Assignee: Nick Bailey
> Priority: Minor
> Fix For: 0.8.6
>
> Attachments: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch, 3041-v2.txt
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3041:
--------------------------------------
Attachment: 3041-v2.txt
v2 uses Sets.difference instead of removeAll
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.6
>
> Attachments: 0001-Don-t-stream-to-replicas-that-already-have-the-data.patch, 3041-v2.txt
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Nick Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Bailey updated CASSANDRA-3041:
-----------------------------------
Affects Version/s: (was: 0.7.8)
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.5
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3041) Move streams data to too many
nodes.
Posted by "Nick Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Bailey updated CASSANDRA-3041:
-----------------------------------
Comment: was deleted
(was: Marking this for the next 0.8.x version since the data duplication caused by this bug is data that does actually belong on the nodes, so it can't be removed by cleanup. It either has to be unduplicated with a major compaction or rely on regular compaction.)
> Move streams data to too many nodes.
> ------------------------------------
>
> Key: CASSANDRA-3041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3041
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.8.4
> Reporter: Nick Bailey
> Fix For: 0.8.5
>
>
> When you decommission a node, it only streams data to the node that is just now gaining responsibility for the node's primary range.
> When you move a node it streams data to every node that is responsible for the node's primary range. This is way more than it needs to, and could be bad in multi-dc setups. We should absolutely use this bug as a chance/reason to better unify that code, since move should be doing the same thing decom does.
> This might be worth backporting to 0.8 as well.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira