You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/08/28 18:18:59 UTC
[jira] Created: (CASSANDRA-397) bootstrap and ringcache
bootstrap and ringcache
-----------------------
Key: CASSANDRA-397
URL: https://issues.apache.org/jira/browse/CASSANDRA-397
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jonathan Ellis
Fix For: 0.5
Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748981#action_12748981 ]
Jun Rao commented on CASSANDRA-397:
-----------------------------------
I don't quite get Jonathan's point either. With RingCache, a client will still read/write through thrift. All RingCache does is for the client to pick a more optimal node. So, as long as each cassandra node can handle read/write properly with respect to Boostrap node, things should be fine, right?
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748905#action_12748905 ]
Sandeep Tata commented on CASSANDRA-397:
----------------------------------------
Doesn't RingCache just get data from cloneTokenEndPointMap() ? So fat clients won't see the Bootstrapping node until it finishes. What am I missing?
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748978#action_12748978 ]
Jonathan Ellis commented on CASSANDRA-397:
------------------------------------------
> The fat client's don't have their own write path, do they?
Why would you write one that does reads over MessagingService but writes over Thrift? Doesn't make sense.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748923#action_12748923 ]
Jonathan Ellis commented on CASSANDRA-397:
------------------------------------------
The point is that RingCache is not aware of TokenMetadata.bootstrapNodes, so fat clients using it cannot ever "discover" bootstrapping nodes. Rather than making RingCache even more complex now (and again, when we do load balancing / node decommissioning) it seems like making the nodes doing the work is a better encapsulation.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748913#action_12748913 ]
Sandeep Tata commented on CASSANDRA-397:
----------------------------------------
The echo-ing of writes happens whether you're the bootstrap coordinator or just another node in the ring (getStorageEndPointsForWrite fix in 375). The only requirement is that the node doing the write has discovered the bootstrapping node. Since this fact is advertised on gossip, all nodes know who is bootstrapping.
(Writes that sneak in before the node has discovered the bootstrapping node will not be sent, but such is life when we have eventual consistency.)
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-397.
--------------------------------------
Resolution: Won't Fix
Thinking about this further, I think the proposed solution (making bootstrap sources responsible for forwarding writes to the targets during bootstrap) is worse than the problem.
The reason is, we want to preserve ConsistencyLevel semantics during bootstrap, and if a CL.quorum/all write is sent to a bootstrap target, it would have to wait for the forwarded write to complete to report in turn that the write is good.
Leaving write propagation to be done by the write originator means we don't have this extra layer of latency.
I will create another ticket for making the write originator include bootstrap targets in the consistency computation.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748973#action_12748973 ]
Sandeep Tata commented on CASSANDRA-397:
----------------------------------------
>> it seems like making the nodes doing the work is a better encapsulation.
Yes, I'm trying to say that the nodes are already doing the work.
The fat client's don't have their own write path, do they? AFAIK, they still talk thrift and submit writes to a Cassandra node.
So long as the fat client is not in charge of sending the write to all the right replicas and doing HHO if needed, it doesn't need to discover the bootstrapping nodes. It doesn't need to know *anything* about the bootstrap process.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748990#action_12748990 ]
Jonathan Ellis commented on CASSANDRA-397:
------------------------------------------
right -- if you need to skip the extra hop for performance, you will want to skip the extra serialize/deserialize from Thrift, too, and you're already importing a whole bunch of Cassandra classes so you might as well use MessagingService + StorageProxy.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748907#action_12748907 ]
Jonathan Ellis commented on CASSANDRA-397:
------------------------------------------
that's the problem -- fat clients won't send data to the new node when doing writes, so it will be out of date when it starts serving reads, which we want to avoid.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748987#action_12748987 ]
Sandeep Tata commented on CASSANDRA-397:
----------------------------------------
Ah ... I think I see now.
Jonathan's assuming that a "fat client" does reads and writes using the MessagingService but uses thrift to get at the ringcache. (We don't have one of these in trunk yet.)
Like Jun says, I was assuming that a "fat client" is still talking over thrift for reads and writes, but uses RingCache to just pick the right node.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-397) bootstrap and ringcache
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748994#action_12748994 ]
Sandeep Tata commented on CASSANDRA-397:
----------------------------------------
>> "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
Ugh .. right in the description :-| As I said, makes sense now.
Perhaps we can worry about this once we actually have fat clients that use messaging service As jbellis said in CASSANDRA-197, I'd assume this kind of fat client should probably participate in gossip without grabbing a token so it doesn't have to deal with RingCache over thrift.
> bootstrap and ringcache
> -----------------------
>
> Key: CASSANDRA-397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-397
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.5
>
>
> Bootstrap mode makes it easy to shoot yourself in the foot with RingCache (see CASSANDRA-197).
> One solution would be to have the node that is providing data to the new, bootstrapping node, echo writes to the new node. Currently, the coordinator node is in charge of including the new node as an "extra" replica, but if a fat client is using messagingservice directly with RingCache that doesn't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.