You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (JIRA)" <ji...@apache.org> on 2012/04/25 20:22:20 UTC
[jira] [Created] (CASSANDRA-4189) Improve hints replay
Vijay created CASSANDRA-4189:
--------------------------------
Summary: Improve hints replay
Key: CASSANDRA-4189
URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.2
Reporter: Vijay
Assignee: Vijay
Priority: Minor
Fix For: 1.2
Problem: Hints are stored in one row.
when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
Problem: Hints replay is too slow and single threaded.
There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292375#comment-13292375 ]
Vijay commented on CASSANDRA-4189:
----------------------------------
{quote}
I don't see a ton of benefit from multiple threads trying to run deliverHintsToEndpointInternal to the same target
{quote}
Hi Jonathan, Thats exactly what the attached patch does, it has one thread per host. Do you see a bug?
{code}
if (queuedDeliveries.contains(to) || !queuedDeliveries.add(to))
{code}
Also ensures there isn't 2 threads running parallel.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4189.patch
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4189) Improve hints replay
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-4189:
-----------------------------
Attachment: 0001-CASSANDRA-4189.patch
Attached patch make Hints delivery multi threaded, this patch replaces "hinted_handoff_throttle_delay_in_ms" to "hinted_handoff_throttle_in_kb" and adds "max_hints_delivery_threads" configurations.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4189.patch
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261991#comment-13261991 ]
Vijay commented on CASSANDRA-4189:
----------------------------------
{quote}
Also, on (mis)completion we flush and force a compaction that should clear out the tombstones (see CASSANDRA-3733) so I'm skeptical this is a real problem.
{quote}
May be the above will fix it, the hints CF (about 10GB) is too large for the node in question... so i have to do more tests.
{quote}
Sure, in a two node cluster maybe the single threaded nature is a problem, but in any cluster of appreciable size it's always overload that's an issue, so I don't see much to be gained by multithreading it.
{quote}
No the problem is when you have 10's of nodes and they are all in different DC's, it is naturally throttled by the latency of 100's of milliseconds. Now while replaying hints, the thread gets stuck replaying the hints to the remote node, no other node gets the hints. What i am suggesting is to throttle but in a multi threaded way.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261975#comment-13261975 ]
Brandon Williams commented on CASSANDRA-4189:
---------------------------------------------
bq. It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
It's not clear to me that the extra code complexity (and IO) is worth this kind of tradeoff. Also, on (mis)completion we flush and force a compaction that should clear out the tombstones (see CASSANDRA-3733) so I'm skeptical this is a real problem.
bq. Problem: Hints replay is too slow and single threaded.
I disagree, historically our problem with hints has always been *overload*, which I think we finally got right in CASSANDRA-3554. Sure, in a two node cluster maybe the single threaded nature is a problem, but in any cluster of appreciable size it's always overload that's an issue, so I don't see much to be gained by multithreading it.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292370#comment-13292370 ]
Jonathan Ellis commented on CASSANDRA-4189:
-------------------------------------------
I don't see a ton of benefit from multiple threads trying to run deliverHintsToEndpointInternal to the same target, and a lot of contention/duplicate handoff attempts (hence, CASSANDRA-4320).
Restricting the handoff workers to different targets would be a straightforward improvement (sounds like this would still address your use case above).
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4189.patch
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295322#comment-13295322 ]
Jonathan Ellis commented on CASSANDRA-4189:
-------------------------------------------
You're right, my mistake. Committed, with a tweak of the default thread count to 2.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-CASSANDRA-4189.patch
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262066#comment-13262066 ]
Jonathan Ellis commented on CASSANDRA-4189:
-------------------------------------------
bq. It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
Not sure this is a problem now that we drop the tombstones after each handoff. Since there is only one handoff per target at a time, we don't need to worry about having to skip tombstones in a meaningful volume.
bq. What i am suggesting is to throttle but in a multi threaded way
+1
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4189) Improve hints replay
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262046#comment-13262046 ]
Brandon Williams commented on CASSANDRA-4189:
---------------------------------------------
bq. Now while replaying hints, the thread gets stuck replaying the hints to the remote node, no other node gets the hints.
Ok, that makes sense.
> Improve hints replay
> --------------------
>
> Key: CASSANDRA-4189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4189
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.2
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.2
>
>
> Problem: Hints are stored in one row.
> when there are a lot of hints stored and we store Tombstones for the ones which has been replayed.
> It might be worth shading the hints based on Hour at which the hints are stored. This can reduce the complexity of the scanning for hints.
> Problem: Hints replay is too slow and single threaded.
> There are use-case where the hints needs to be replayed ASAP to make the cluster more consistent.
> In Multi region cluster, the throttle is already done due to the latency which is in the order of 100's of millisecond.
> It might be worth trying to replay the hints in parallel and throttle on the number of bytes read from the disk or use the existing setting of throttle based on sleep interval on all the threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira