You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Rick Branson (JIRA)" <ji...@apache.org> on 2012/10/19 22:40:12 UTC
[jira] [Created] (CASSANDRA-4839) Online toggle for node write-only
status
Rick Branson created CASSANDRA-4839:
---------------------------------------
Summary: Online toggle for node write-only status
Key: CASSANDRA-4839
URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Rick Branson
Priority: Minor
It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Rick Branson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480548#comment-13480548 ]
Rick Branson commented on CASSANDRA-4839:
-----------------------------------------
You can't receive hints or streams if thrift is disabled though, right?
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480471#comment-13480471 ]
Brandon Williams commented on CASSANDRA-4839:
---------------------------------------------
You could disable thrift so that can't happen.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Rick Branson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484228#comment-13484228 ]
Rick Branson commented on CASSANDRA-4839:
-----------------------------------------
I want a JMX method so I can run it for a long (out of the ordinary) repair or upgrading nodes too.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480421#comment-13480421 ]
Brandon Williams commented on CASSANDRA-4839:
---------------------------------------------
It's not a total solution, but one way we can optimize this is to set the SEVERITY gossip state (in 1.2) very high at startup or when receiving hints and then lower it later so that the dynamic snitch won't ever prefer the node.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481457#comment-13481457 ]
Jonathan Ellis commented on CASSANDRA-4839:
-------------------------------------------
My question here is, how does a node know when it's done receiving hints?
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481456#comment-13481456 ]
Jonathan Ellis commented on CASSANDRA-4839:
-------------------------------------------
Thrift is only used for client connections.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484040#comment-13484040 ]
Brandon Williams commented on CASSANDRA-4839:
---------------------------------------------
bq. I think it would be something like wait N seconds on boot to receive hints, and then wait until any in-progress hint deliveries finished or timed out before taking reads. It doesn't have to be perfect, but would significantly increase the consistency of CL.ONE reads during reboots, which along with the other tickets that cover faster table loading, would promote more productive restarts – think tuning and minor upgrades for both C* and the JVM.
That sounds like a reasonable solution if it's configurable.
bq. Either way, I think that automatically doing that would be great but isn't coupled to the original idea. Just getting the ability to do this from JMX would be great.
I actually like your first idea better than a JMX method that does spooky things :)
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Rick Branson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480454#comment-13480454 ]
Rick Branson commented on CASSANDRA-4839:
-----------------------------------------
That could be an interim solution. What about reads where the "write-only" coordinator is also a replica for the requested key?
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Rick Branson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483472#comment-13483472 ]
Rick Branson commented on CASSANDRA-4839:
-----------------------------------------
For some reason I was confused about thrift -- there seems to be a lack of information about exactly what goes over the storage port and what goes over thrift. Originally I was correct -- that it's any internode communication, but then I read CASSANDRA-4162 and for some reason it gave me the idea that thrift was also used for stream & hint delivery. Oh well.
I think it would be something like wait N seconds on boot to receive hints, and then wait until any in-progress hint deliveries finished or timed out before taking reads. It doesn't have to be perfect, but would significantly increase the consistency of CL.ONE reads during reboots, which along with the other tickets that cover faster table loading, would promote more productive restarts -- think tuning and minor upgrades for both C* and the JVM.
Either way, I think that automatically doing that would be great but isn't coupled to the original idea. Just getting the ability to do this from JMX would be great.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4839) Online toggle for node
write-only status
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482519#comment-13482519 ]
Brandon Williams commented on CASSANDRA-4839:
---------------------------------------------
I don't think it ever can with absolute certainty, but we could set the severity based on how long it has been since we last received a hint and mitigate the problem.
> Online toggle for node write-only status
> ----------------------------------------
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Rick Branson
> Priority: Minor
>
> It would be really great if users could disable/enable reads on a given node, while still allowing write operations to take place. This would be similar to how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for maintenance for a few minutes, and while the node is catching up from hints, which can take 10-30 minutes depending on write load, it will serve stale data. Do the math for a rolling restart of a large cluster and you have potential windows of hours or days where a large amount of inconsistency is surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance alleviates concerns about inconsistent data surfaced to users during normal, planned activities. While a read consistency >ONE can indeed be used to prevent any inconsistency from the scenario above, it seems ridiculous to always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) automatically "go dark" for reads while it's receiving hints after joining the cluster or perhaps during repair. These obviously have their own complications and justify separate tickets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira