You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Daniel Doubleday (JIRA)" <ji...@apache.org> on 2011/05/18 12:55:47 UTC
[jira] [Created] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Nodes get ignored by dynamic snitch when read repair chance is zero
-------------------------------------------------------------------
Key: CASSANDRA-2662
URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.6
Reporter: Daniel Doubleday
Priority: Trivial
DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
This leads to a stable order of hosts until reads will lead to recorded scores.
If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035618#comment-13035618 ]
Brandon Williams commented on CASSANDRA-2662:
---------------------------------------------
There is one problem with this patch, rather that initializing the score to zero, it needs to create an empty AdaptiveLatencyTracker for them before checking the scores, otherwise DES.reset() will never have an affect on those hosts.
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2662) Nodes get ignored by
dynamic snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035618#comment-13035618 ]
Brandon Williams edited comment on CASSANDRA-2662 at 5/18/11 7:48 PM:
----------------------------------------------------------------------
There is one problem with this patch, rather that initializing the score to zero, it needs to create an empty AdaptiveLatencyTracker for them before checking the scores, otherwise DES.reset() will never have an effect on those hosts.
was (Author: brandon.williams):
There is one problem with this patch, rather that initializing the score to zero, it needs to create an empty AdaptiveLatencyTracker for them before checking the scores, otherwise DES.reset() will never have an affect on those hosts.
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-2662:
-------------------------------------------
Assignee: Brandon Williams
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072619#comment-13072619 ]
Hudson commented on CASSANDRA-2662:
-----------------------------------
Integrated in Cassandra-0.7 #537 (See [https://builds.apache.org/job/Cassandra-0.7/537/])
DES shouldn't ignore nodes when the read repair chance is zero.
Patch by brandonwilliams reviewed by Paul Cannon for CASSANDRA-2662
brandonwilliams : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152038
Files :
* /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/locator/DynamicEndpointSnitch.java
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.5
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Fix For: 0.7.9, 0.8.3
>
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2662:
--------------------------------------
Reviewer: thepaul
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Daniel Doubleday (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Doubleday updated CASSANDRA-2662:
----------------------------------------
Attachment: dynsnitch.patch
One very simple fix is to initialize scores with 0 which forces at least one read.
Dunno if thats a good idea when using multi dc snitches ...
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035597#comment-13035597 ]
Brandon Williams commented on CASSANDRA-2662:
---------------------------------------------
Thinking this through a bit more:
Given coordinator A, and replicas X, Y, and Z (in subsnitch order), on the first round X will be chosen, and let's say it receives a score of 1. With the patch, at this point Y and Z will be initialized with zero. On the next round, Y will be chosen, and let's say it receives a score of or near 1, depending on network latency. On the third round, Z will be chosen, and let's say it also receives a score similar to Y. Now the cache is hot on all nodes, and subsequent reads have the possibility to oscillate between all three based on network latency variance. This can be mitigated though with the badness threshold. With the badness threshold on, the first round will occur as before, but subsequent rounds will continue to use X until it degrades past the threshold, at which point they will use Y, until the dynamic snitch reset()s, at which point everything will repeat. I don't think this is harmful to CASSANDRA-1314 after all.
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2662:
----------------------------------------
Attachment: 2662.txt
Updated patch ensures ALTs are created for the hosts by using receiveTiming
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "paul cannon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072583#comment-13072583 ]
paul cannon commented on CASSANDRA-2662:
----------------------------------------
+1
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2662:
--------------------------------------
Fix Version/s: (was: 0.8.3)
0.8.5
(This didn't get merged to 0.8 until just now.)
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.5
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Fix For: 0.7.9, 0.8.5
>
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035431#comment-13035431 ]
Brandon Williams commented on CASSANDRA-2662:
---------------------------------------------
To clarify what this patch does: on the first round, behavior is unchanged, it will fall back to the subsnitch as usual. On the second round however, the nodes will have scores so the dynamic snitch will determine the best host. This seems like a good idea, but I think we should proceed cautiously with regard to CASSANDRA-1314. This is certainly going to blow the cache on the second round, since the unpinned hosts at zero are guaranteed to be ranked higher than the pinned one. However, in the current implementation, the badness threshold in CASSANDRA-1519 can never take effect at RR = 0 since the scores of the other replicas will always be null, and thus diversion to the subsnitch will always occur, returning the pinned replica. With this patch, the second replica is likely to end up being the pinned one, which I think will be ok, as long as all coordinators agree, though this might not always be the case.
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Reporter: Daniel Doubleday
> Priority: Trivial
> Attachments: dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2662) Nodes get ignored by dynamic
snitch when read repair chance is zero
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2662:
----------------------------------------
Affects Version/s: (was: 0.7.6)
0.6.5
Fix Version/s: 0.8.3
0.7.9
> Nodes get ignored by dynamic snitch when read repair chance is zero
> -------------------------------------------------------------------
>
> Key: CASSANDRA-2662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2662
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.5
> Reporter: Daniel Doubleday
> Assignee: Brandon Williams
> Priority: Trivial
> Fix For: 0.7.9, 0.8.3
>
> Attachments: 2662.txt, dynsnitch.patch
>
>
> DynamicEndpointSnitch falls back to subsnitch when one of the scores of the endpoints being compared is missing.
> This leads to a stable order of hosts until reads will lead to recorded scores.
> If setting read repair chance to 0 and reads are performed with quorum then (rf - # quorum nodes) will never get reads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira