You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alexey Zotov (JIRA)" <ji...@apache.org> on 2012/10/05 08:52:47 UTC
[jira] [Created] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Alexey Zotov created CASSANDRA-4768:
---------------------------------------
Summary: Add separate max_hint_window_in_ms option for remote data centers
Key: CASSANDRA-4768
URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.1.2
Reporter: Alexey Zotov
Assignee: Alexey Zotov
Priority: Minor
It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473084#comment-13473084 ]
Alexey Zotov commented on CASSANDRA-4768:
-----------------------------------------
Jonathan, if we're talking about latency, that you're right. We discussed that in CASSANDRA-4761.
Now we're talking about size of hints CF. For local failed node we should store much less hints that for whole remote DC. Lack of connectivity to remote DC is not a common problem, but it could happen. The new option should save us from "overfull" of hints CF in that case. Saying "overfull" I mean the following cases:
1. if we use populate_io_cache_on_flush option hints will use part of memory until they wont be supplanted by another data. It's actual until CASSANDRA-4694 is not resolved.
2. if we don't have a lot of free space on disc (it can be reserved by compaction processes)
Let me remind that amount of hints for remote DC could be a huge in case of large count of write/update operations.
Brandon, yes, we have to repair anyway if the remote hint window has been exceeded.
We've thought and decided that it really can help only in the specific use cases, e.g. with master-slave data centers (when write to one, read from all).
Actually we think that the new option is not necessary for us and we leave decision to include or not include it to your notice.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.7
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Sergey Tryuber (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471716#comment-13471716 ]
Sergey Tryuber commented on CASSANDRA-4768:
-------------------------------------------
Brandon, you're right. But we're talking about the period of time we can rely on hinted handoff. If there is only one configuration option, we must set it up considering the worst case: lack of cross datacenter connectivity. Because in that case a lot of hints will be saved. For local DC (one-two node failures in most of the cases) not as many data will be saved as hints. So larger window size can be configured (and admins can sleep well at nights or maintain servers without being in a hurry).
I understand that Cassandra is well known as "easy to set up" (despite of HBase). So may be we can add this option as a "hidden" with only one default value in cassandra.yaml for both windows (the same approach is used by Hadoop, Hive and HBase).
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4768:
------------------------------------
Fix Version/s: 1.2.0
1.1.6
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471607#comment-13471607 ]
Brandon Williams commented on CASSANDRA-4768:
---------------------------------------------
If you lose a DC and exceed the hint window you'll have to repair... which would fix any local damage as well, so it doesn't really matter if you have a longer local hint window since you have to repair anyway.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472444#comment-13472444 ]
Jonathan Ellis commented on CASSANDRA-4768:
-------------------------------------------
Sorry, I used the wrong terminology. The right fix is not parallel HH with each thread waiting for cross-dc latency, but asynchronous HH where each thread continuously sends mutations across the link and marks them successful when acks arrive later. Latency then becomes basically irrelevant and your throughput is independent of whether replay is local or over wan.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.7
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Sergey Tryuber (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473415#comment-13473415 ]
Sergey Tryuber commented on CASSANDRA-4768:
-------------------------------------------
Thank you, Brandon
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.7
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470628#comment-13470628 ]
Alexey Zotov commented on CASSANDRA-4768:
-----------------------------------------
I'll try to explain my motivation. There is the following use case:
We have a few DCs. One of DCs has been lost connection to others. So we should store hints for all ranges of data (for all data set) for that DC. Write's throughput is very high. Amount of stored hints for the lost remote DC grows very quickly.
We can reduce max_hint_window_size, but we want to keep repair mechanism for local nodes.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4768:
------------------------------------
Attachment: cassandra-1.2-4768-remote_hint_window.txt
cassandra-1.1-4768-remote_hint_window.txt
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470367#comment-13470367 ]
Jonathan Ellis commented on CASSANDRA-4768:
-------------------------------------------
I'm not sure if this is the right approach. Cross-DC repair is painful too, after all, and that's basically your alternative. I think parallelizing HH as discussed in your other ticket is a better solution.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4768) Add separate
max_hint_window_in_ms option for remote data centers
Posted by "Sergey Tryuber (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471520#comment-13471520 ]
Sergey Tryuber commented on CASSANDRA-4768:
-------------------------------------------
Basically I think we should keep hinted handoff configuration as independent as possible for local and remote datacenter use cases. Basically those are two different kind of fail overs: node failure inside a local datacenter and lack of connection between two data centers. In the first case we expect quite small amount of hints (to only one node) and hints can be replayed very fast because we have good high throughput network connection between nodes within a single DC. That's why we could set quite large window size (to, say, 12 hours) and quite small amount of parallel workers for hints replaying (because network roundtrip is fast and we're able to replay hints in sequential mode with quite fast speed). In the second case the situation is opposite: we need a lot of parallel workers and quite small window size.
Once again, I think hinted handoff is great, but there should be two different sets of configuration options for replaying inside a local DC and to a remote DC. Going further I'd even add possibility to set up different options for different data centers, but it is a theme for another ticket.
> Add separate max_hint_window_in_ms option for remote data centers
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4768
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4768
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Priority: Minor
> Labels: configuration, hintedhandoff
> Fix For: 1.1.6, 1.2.0
>
> Attachments: cassandra-1.1-4768-remote_hint_window.txt, cassandra-1.2-4768-remote_hint_window.txt
>
>
> It would be nice to have possibility to configure hint window size for remote dc separately. It will allow to prevent accumulating of big amount of data for remote dc and long hints delivery as the result of it.
> I suggest to add max_hint_window_for_remote_dc_in_ms option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira