You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Fabien Rousseau (Created) (JIRA)" <ji...@apache.org> on 2011/12/15 16:52:30 UTC
[jira] [Created] (CASSANDRA-3639) Move streams too many data
Move streams too many data
--------------------------
Key: CASSANDRA-3639
URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
Project: Cassandra
Issue Type: Improvement
Affects Versions: 0.8.7
Reporter: Fabien Rousseau
Priority: Minor
Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
During a move operation, we observed that the node streamed most of its data and received all its data.
We are running Cassandra 0.8.7 (plus a few patches)
After reading the code related to "move", we found out that :
- in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
We managed to isolate and reproduce this in a unit test.
We also propose a patch which :
- does not rely on any sort
- adds a few unit tests (may not be exhaustive...)
Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3639) Move streams too many data
Posted by "Fabien Rousseau (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabien Rousseau updated CASSANDRA-3639:
---------------------------------------
Attachment: 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
> Move streams too many data
> --------------------------
>
> Key: CASSANDRA-3639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.8.7
> Reporter: Fabien Rousseau
> Priority: Minor
> Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
>
>
> During a move operation, we observed that the node streamed most of its data and received all its data.
> We are running Cassandra 0.8.7 (plus a few patches)
> After reading the code related to "move", we found out that :
> - in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
> We managed to isolate and reproduce this in a unit test.
> We also propose a patch which :
> - does not rely on any sort
> - adds a few unit tests (may not be exhaustive...)
> Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3639) Move streams too many data
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3639:
--------------------------------------
Reviewer: thepaul
Fix Version/s: 1.1
I'm not comfortable changing the move code in a stable release, but this sounds like a good change to make in 1.1. Can you post a version of the patch that applies to trunk?
> Move streams too many data
> --------------------------
>
> Key: CASSANDRA-3639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.8.7
> Reporter: Fabien Rousseau
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
>
>
> During a move operation, we observed that the node streamed most of its data and received all its data.
> We are running Cassandra 0.8.7 (plus a few patches)
> After reading the code related to "move", we found out that :
> - in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
> We managed to isolate and reproduce this in a unit test.
> We also propose a patch which :
> - does not rely on any sort
> - adds a few unit tests (may not be exhaustive...)
> Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3639) Move streams too many data
Posted by "paul cannon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202879#comment-13202879 ]
paul cannon commented on CASSANDRA-3639:
----------------------------------------
https://github.com/thepaul/cassandra/tree/3639 <- a couple minor tweaks to the v2 patch can be found on this branch; renaming "substract" to "subtract", and eliminating all the extra eol whitespace that was added.
> Move streams too many data
> --------------------------
>
> Key: CASSANDRA-3639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.8.7
> Reporter: Fabien Rousseau
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests-v2.patch, 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
>
>
> During a move operation, we observed that the node streamed most of its data and received all its data.
> We are running Cassandra 0.8.7 (plus a few patches)
> After reading the code related to "move", we found out that :
> - in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
> We managed to isolate and reproduce this in a unit test.
> We also propose a patch which :
> - does not rely on any sort
> - adds a few unit tests (may not be exhaustive...)
> Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3639) Move streams too many data
Posted by "Fabien Rousseau (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabien Rousseau updated CASSANDRA-3639:
---------------------------------------
Attachment: 0001-try-to-fix-move-streaming-too-many-data-unit-tests-v2.patch
Sure.
This patch (v2) applies to trunk.
I also made a little modification in the tests (for the first patch, at the last moment, I changed the first token from 0 to 10, and this made most of the test OK with the current code)
> Move streams too many data
> --------------------------
>
> Key: CASSANDRA-3639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.8.7
> Reporter: Fabien Rousseau
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests-v2.patch, 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
>
>
> During a move operation, we observed that the node streamed most of its data and received all its data.
> We are running Cassandra 0.8.7 (plus a few patches)
> After reading the code related to "move", we found out that :
> - in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
> We managed to isolate and reproduce this in a unit test.
> We also propose a patch which :
> - does not rely on any sort
> - adds a few unit tests (may not be exhaustive...)
> Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3639) Move streams too many data
Posted by "paul cannon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202875#comment-13202875 ]
paul cannon commented on CASSANDRA-3639:
----------------------------------------
+1, although I agree the tests should probably move somewhere else. I worked a bit on putting similar tests in the dtest2 framework, and checking which ranges *actually* got streamed, but yeah, that's overkill for this.
I was a bit tripped up by the fact that {{calculateStreamAndFetchRanges}} is only valid for ranges for a moving node itself, and not for calculating changes for a remote node's movement (i.e., {{calculatePendingRanges}}), but that's ok, because it's only used in that way. A comment to that effect would not go amiss, though.
> Move streams too many data
> --------------------------
>
> Key: CASSANDRA-3639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3639
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.8.7
> Reporter: Fabien Rousseau
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-try-to-fix-move-streaming-too-many-data-unit-tests-v2.patch, 0001-try-to-fix-move-streaming-too-many-data-unit-tests.patch
>
>
> During a move operation, we observed that the node streamed most of its data and received all its data.
> We are running Cassandra 0.8.7 (plus a few patches)
> After reading the code related to "move", we found out that :
> - in StorageService.java, line 2002 and line 2004 => ranges are returned in a non ordered collection, but calculateStreamAndFetchRanges() method (line 2011) assume ranges are sorted, thus, resulting in wider ranges to be fetched/streamed
> We managed to isolate and reproduce this in a unit test.
> We also propose a patch which :
> - does not rely on any sort
> - adds a few unit tests (may not be exhaustive...)
> Unit tests are done only for RF=2 and for the "OldNetworkStrategyTopology". For the sake of simplicity, we've put them in OldNetworkStrategyTopologyTest, but they probably should be moved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira