You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/04/15 16:38:50 UTC
[jira] Created: (CASSANDRA-981) Dynamic endpoint snitch
Dynamic endpoint snitch
-----------------------
Key: CASSANDRA-981
URL: https://issues.apache.org/jira/browse/CASSANDRA-981
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Jonathan Ellis
Fix For: 0.7
An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884350#action_12884350 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
Updates:
LBD is threadsafe. Not sure what I was smoking there.
Big Hammer latency reset is fine for now but I would like to have more graceful aging out of latencies so if you have latencies like .|.|... and we age out the || then it sorts to its "normal" place again without having to rediscover things from scratch.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898717#action_12898717 ]
Brandon Williams commented on CASSANDRA-981:
--------------------------------------------
Committed to 0.6 and trunk. Also worth noting that DES has been broken in trunk due to incorrectly calculating the probability (needs to be 1 - Math.pow) which I fixed also. We missed this because the tests were wrong.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898927#action_12898927 ]
Hudson commented on CASSANDRA-981:
----------------------------------
Integrated in Cassandra #515 (See [https://hudson.apache.org/hudson/job/Cassandra/515/])
fall back to subsnitch on score equality as well as no score at all, fix probability calculation. Patch by jbellis and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-981
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869181#action_12869181 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack.
A crappy alternative would be run-time loading of a new snitch (which could be hard-coded to sort whichever way).
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-981:
-------------------------------------
Fix Version/s: 0.6.5
(was: 0.7 beta 1)
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-981:
------------------------------------------
Assignee: Brandon Williams
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877855#action_12877855 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
(1)
combine
+ windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE));
+ AdaptiveLatencyTracker tracker = windows.get(host);
to
+ AdaptiveLatencyTracker tracker = windows.putIfAbsent(host, new AdaptiveLatencyTracker(WINDOW_SIZE));
Even better: get, then putIfAbsent only if null, would avoid creating new ALT objects every time
(2)
receiveTiming isn't threadsafe. use AtomicInteger?
(3)
AdaptiveLatencyTracker doesn't look threadsafe either. definitely LBD isn't. I think just using a threadsafe queue like CLQ would work? (My fault for naming ASD a Deque, when it only really needs a Queue)
(4)
do we need unregister()? if not, let's drop that api
(5)
sortByProximity needs to return zero if both scores are null. even better, take a non-dynamic snitch and use the static topology when there is no score info yet (this would save us from sending data requests to another data center after every clear of the stats). So, rather than using DES directly in the config, maybe having a boolean for whether to wrap your regular snitch, with the dynamic one, is the way to go.
what if reset cleared scores, instead of latencies? this would result in a more gradual aging out of both slow and fast latencies as new ones were pushed in, which would make it more tolerant of brief hiccups where a mostly fast node had a couple slow responses. Feels more how phi was meant to work, to me.
(6)
+ if (address != FBUtilities.getLocalAddress()) // we only know about ourself
+ return addresses;
let's change this to an assert
(7)
deque.offer is more idiomatic than try/except in java
(8)
let's use a single timer for both update and reset
(9)
it's not completely clear to me that the phi code designed for telling "how long has been too long to wait for updates that are supposed to arrive at the same interval," applies well to latency information that arrives in bursts up to our max per interval. can you add some tests showing that it does the right thing, given several mixes of latencies?
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-981:
--------------------------------------
re-opening for backport to 0.6.5
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897747#action_12897747 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
+1
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869713#action_12869713 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
Right, you want to adapt to "effective reply time" not just the network part. That is why I don't think the vivaldi stuff helps much since it's trying to solve a static network question instead.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872380#action_12872380 ]
Brandon Williams commented on CASSANDRA-981:
--------------------------------------------
It looks like the phi algorithm can work for this. It's simple to replace timestamps with latency values, and then use an arbitrary value when calculating the phi score for a host. The main problem is calculating phi is less than free, and doing so for each request to determine the least latent endpoint will be too expensive. We can get around that by doing it up front when we add a sample and keeping a cache of the scores, but that means we'll have to add samples fairly infrequently, or at least batch the updates so the mean only has to be calculated once per batch.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-981:
-------------------------------------
Attachment: 981-fix-v2.txt
fix-v2 switches to EndPointSnitch (SimpleSnitch does not exist in 0.6), fixes order for "make host3 the worst" and adds missing assert.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870255#action_12870255 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
Right, we've seen lots of instances of nodes that have enough cpu to spit out gossip heartbeats, but not enough that they can handle a normal complement of requests.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Wouter de Bie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914374#action_12914374 ]
Wouter de Bie commented on CASSANDRA-981:
-----------------------------------------
There seems to be a bug in the 0.6.5 DatabaseDescriptor.java. On line 629:
{noformat}
String dynamic = System.getProperty("cassandra.dynamic_snitch");
if (dynamic == null || Boolean.getBoolean(dynamic) == false)
epSnitch = snitch;
else
epSnitch = new DynamicEndpointSnitch(snitch);
{noformat}
Boolean.getBoolean(String s) checks for System.getProperty(). I think the code should read:
{noformat}
if (Boolean.getBoolean('cassandra.dynamic_snitch') == false)
epSnitch = snitch;
else
epSnitch = new DynamicEndpointSnitch(snitch);
{noformat}
A workaround is to supply JVM properties like "-Dcassandra.dynamic_snitch_enabled=true -Dcassandra.dynamic_snitch=cassandra.dynamic_snitch_enabled"
Has anyone succesfully run with 0.6.5 and the dynamic snitch?
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914472#action_12914472 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
oh, that's funny, in an unfortunate way. you're right about the getBoolean. fixed in r1000877 on 0.6 branch.
you're right about the second part too, but that was already fixed in CASSANDRA-1429 (also on 0.6 branch).
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888341#action_12888341 ]
Hudson commented on CASSANDRA-981:
----------------------------------
Integrated in Cassandra #491 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/491/])
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: (was: 981-0.6.txt)
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870209#action_12870209 ]
Brandon Williams commented on CASSANDRA-981:
--------------------------------------------
Wouldn't sorting the endpoints by their phi value in the FD work well since FD is already adaptive?
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: (was: 981.txt)
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857338#action_12857338 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
Stu suggested Vivaldi positioning (http://swtch.com/~rsc/talks/vivaldi-ccs.pdf), but that is actually solving a different problem. (How do you estimate network distance, without actually talking to the node in question?) Since we are in near-constant communication with all the nodes in the system, we should be able to leverage that for a solution that is both simpler and quicker to adapt to changing conditions.
This problem seems similar to the sliding window we maintain of heartbeat times for failure detection, but this seems too heavyweight to apply when processing thousands of responses per second. Perhaps restricting the sampling to "at most N per second" would work, although I'm not sure if the phi algorithm can work on samples that don't occur at evenly-spaced intervals.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872655#action_12872655 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
right, even for something simpler than phi we'd want to sample, say, 10x per second at most.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869685#action_12869685 ]
Ryan King commented on CASSANDRA-981:
-------------------------------------
>I think what we want to do is have the "dynamic" part just apply to endpoint-sorting, and leave replica determination up to the strategy in terms of DC + Rack.
I think this could be good for more than network distance, it would also help a cluster route around a node that is slow (either permanently or temporarily). In addition to being in compaction, a node that has just been restarted and therefore has empty caches should be routed around (but not completely, since it needs traffic in order to warm its caches).
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869473#action_12869473 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
Once the other nodes come back up they'll get _some_ traffic still b/c of RR (and of course writes), so it will quickly figure out that the cheap route is back.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-981:
--------------------------------------
Getting this test failure in 0.6:
[junit] Testcase: testSnitch(org.apache.cassandra.locator.DynamicEndpointSnitchTest): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.cassandra.locator.DynamicEndpointSnitchTest.testSnitch(DynamicEndpointSnitchTest.java:70)
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Wouter de Bie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914377#action_12914377 ]
Wouter de Bie commented on CASSANDRA-981:
-----------------------------------------
This stuff is actually more broken than I thought..
It only works with one keyspace, since the DynamicEndpointSnitchMBean is registered per keyspace, which is not allowed.
Next, the DynamicEndpointSnitch doesn't work with the RackAwareStrategy, since that one checks the following:
{noformat}
if (!(snitch instanceof EndPointSnitch))
throw new IllegalArgumentException(("RackAwareStrategy requires EndPointSnitch."));
{noformat}
And breaks, since the DynamicEndPointSnitch extends AbstractEndpointSnitch and not EndPointSnitch. RackAwareStrategy should check for the interface IEndPointSnitch.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix-v2.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-981:
-------------------------------------
Attachment: 981-fix.txt
fix patch attached to
1) r/m unused imports
2) fall back to subsnitch on score equality as well as no score at all
3) order lower scores first in proximity list
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981-fix.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: 981.txt
Updated patch incorporating above suggestions.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: 981-0.6.txt
Backport for 0.6.5. Requires -Dcassandra.dynamic_snitch=true to be added to the JVM options to enable.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870319#action_12870319 ]
Brandon Williams commented on CASSANDRA-981:
--------------------------------------------
I suspect the phi algorithm can be modified to satisfy the "at N most per second" criterion wrt RPC calls, but I'll have to experiment a bit to be certain.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: (was: 981.txt)
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869458#action_12869458 ]
Vijay commented on CASSANDRA-981:
---------------------------------
Things can get complicated when there is a network outage and the network being smart trys to re-roue the traffic in the more expensive route, in which case the whole equation changes.... I am not sure if this sorting has to be done only once when the node comes up? even then we dont know if during bootstrapping there was a network outage.....
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: 981.txt
There's a problem with this snitch: once a machine sees a host and sorts it to the end, it will never read from that host again, meaning that it will never update that host's score, until such time that another host performs worse.
Updated patch resolves this by resetting all the timing windows once per minute, giving the 'bad' hosts a chance to recover.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: 981.txt
Here is a dynamic snitch that listens to read latencies from other machines, accepts at most 10 timings per second, and once per second calculates the phi value for all hosts it knows about. Comments and testing would be most welcome.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886539#action_12886539 ]
Jonathan Ellis commented on CASSANDRA-981:
------------------------------------------
+1 after minor clean up:
+ if (conf.dynamic_snitch == false)
+ return snitch;
+ else
+ return (IEndpointSnitch) new DynamicEndpointSnitch(snitch);
prefer !conf.dynamic_snitch (and prefer positive conditions first, to avoid "!..else")
the cast should be unnecessary
personally I would go for a ?: here but if/else is ok
+ public void add(double i)
+ {
+ latencies.offer(i);
+ }
indent is wrong there
+ double exponent = (-1)*(t)/mean;
space b/t arithmetic operators here and everywhere else please
DynamicEndpointSnitchTest: several { need to be on new lines
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.7
>
> Attachments: 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-981:
---------------------------------------
Attachment: 981-0.6.txt
Also includes improvements from CASSANDRA-1296.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 981-0.6.txt, 981.txt
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870213#action_12870213 ]
Ryan King commented on CASSANDRA-981:
-------------------------------------
The phi value is only effected by gossip, right? I think this would only work (or work much better) if it took into account rpc response times.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864550#action_12864550 ]
Stu Hood commented on CASSANDRA-981:
------------------------------------
> but that is actually solving a different problem.
I think the key takeaway from Vivaldi is using a coordinate system, so that you don't have to store latency information for every endpoint you've ever communicated with.
> I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results
Would it be possible to have the snitch store its coordinates in the system table, so that during bootstrapping, it looks at gossip latency, tunes its coordinates, and then persists them forever? It rules out using endpointsnitches to dynamically adjust position based on load on a machine, but I think that might be better solved by load balancing anyway.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-981) Dynamic endpoint snitch
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857969#action_12857969 ]
Ryan King commented on CASSANDRA-981:
-------------------------------------
I'm sure you thought about this, but in cases where you're using the RackAwareStrategy, the dynamic snitch will need to yield consistent results, otherwise, you could create situations where the natural endpoints for a token keep changing depending on network performance.
> Dynamic endpoint snitch
> -----------------------
>
> Key: CASSANDRA-981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-981
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7
>
>
> An endpoint snitch that automatically and dynamically infers "distance" to other machines without having to explicitly configure rack and datacenter positions solves two problems:
> The killer feature here is adapting to things like compaction or a failing-but-not-yet-dead disk. This is important, since when we are doing reads we pick the "closest" replica for actually reading data from (and only read md5s from other replicas). This means that if the closest replica by network topology is temporarily slow due to compaction (for instance), we'll have to block for its reply even if we get the other replies much much faster.
> Not having to manually re-sync your configuration with your network topology when changes (adding machines) are made is a nice bonus.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira