You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alexey Zotov (JIRA)" <ji...@apache.org> on 2012/10/04 15:19:07 UTC
[jira] [Created] (CASSANDRA-4761) Batch hints delivery
Alexey Zotov created CASSANDRA-4761:
---------------------------------------
Summary: Batch hints delivery
Key: CASSANDRA-4761
URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.1.2
Reporter: Alexey Zotov
Assignee: Alexey Zotov
Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hint's delivery using batch operations.
What do you think about it? Is there some facts that won't allow to implement that mechanism?
I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474282#comment-13474282 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
Wnen sendMutation throws TimeouException it gives control to begining of the while loop ("break delivery;" line). So process starts again - it executes slice query from the last column (that was timed out on previous step).
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4761:
------------------------------------
Attachment: cassandra-1.2-4761-async_hints-v2.txt
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Batch hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4761:
------------------------------------
Description:
Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
What do you think about it? Is there some facts that won't allow to implement that mechanism?
I'll try to implement it if you approve and clarify right way.
was:
Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hint's delivery using batch operations.
What do you think about it? Is there some facts that won't allow to implement that mechanism?
I'll try to implement it if you approve and clarify right way.
> Batch hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474482#comment-13474482 ]
Jonathan Ellis commented on CASSANDRA-4761:
-------------------------------------------
Why does making it async require this code?
{code}
. // if the last column has been reached
if (!ByteBufferUtil.EMPTY_BYTE_BUFFER.equals(startColumn))
{
// start query from the first column
// it's needed to check if some hints were not delivered because of timeouts
startColumn = ByteBufferUtil.EMPTY_BYTE_BUFFER;
continue;
}
// we've started from the beginning and could not find anything (only maybe tombstones)
else
{
break;
}
{code}
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Async hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4761:
--------------------------------------
Reviewer: jbellis
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Batch hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473229#comment-13473229 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
I've attached two files: first version of the patch and file with comments.
> Batch hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Attachments: cassandra-1.1-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474311#comment-13474311 ]
Brandon Williams commented on CASSANDRA-4761:
---------------------------------------------
I think you're right, we'll loop forever trying the host without another FD.isAlive check.
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4761:
------------------------------------
Attachment: cassandra-1.2-4761-async_hints.txt
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474527#comment-13474527 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
Thanks!
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Batch hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4761:
------------------------------------
Attachment: notices.txt
> Batch hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Attachments: cassandra-1.1-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474228#comment-13474228 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
I've attached file with discussed fixes. Also I've added some test for slice queries on expired columns.
One more question, in 1.1 version there is the following bug:
When destination node has been down during hints delivery process, source node doesn't stop the process. It tries to send hints until destination node won't be up again. Should I create separate ticket or can simply attach patch to it?
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474507#comment-13474507 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
It's needed to handle situation when some hints were not delivered because of timeouts. I see only one way to do that - it's to make a new slice query and if it won't find anything - all hints have been successfully delivered.
So when the first time we get "paging finished" condition (we've simply sent all hints without any checking) the startColumn is not equal to EMPTY. And then we'are trying to do the same query from the first column. If we find anything we'll continue hints delivery because found hints have been timed out and not deleted. If we don't find anything we'll exit because startColumn is equal to EMPTY.
Do you have another thoughts about how to handle timeouts?
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474399#comment-13474399 ]
Alexey Zotov commented on CASSANDRA-4761:
-----------------------------------------
I agree with Jonathan too, sorry for the frictions :)
New file has been attached.
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Batch hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469460#comment-13469460 ]
Jonathan Ellis commented on CASSANDRA-4761:
-------------------------------------------
I don't think they are very similar. 4047 is about "how can I hint an entire sstable," and this is about "how can I optimize hint delivery over the WAN."
Personally I don't think you'd be well-served by writing out sstables locally for hints and streaming them over. Easier and more effective to make hint delivery asynchronous -- instead of waiting for each hint to be acked before sending another, send hints continuously (until throttled) and create callbacks to delete successfully delivered ones.
> Batch hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474329#comment-13474329 ]
Brandon Williams commented on CASSANDRA-4761:
---------------------------------------------
Changed my mind, Jonathan is right. "break delivery" breaks the while loop, it doesn't 'jump' to the label.
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4761) Async hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4761.
---------------------------------------
Resolution: Fixed
LGTM, committed (and inlined the hint cleanup callback)
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, cassandra-1.2-4761-async_hints-v2.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4761) Async hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474248#comment-13474248 ]
Jonathan Ellis commented on CASSANDRA-4761:
-------------------------------------------
Not sure I follow. If a node goes down during delivery, sendMutation will timeout and deliverHintsToEndpointInternal will stop.
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, cassandra-1.2-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Async hints delivery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4761:
--------------------------------------
Affects Version/s: (was: 1.1.2)
Fix Version/s: 1.2.0 beta 2
Summary: Async hints delivery (was: Batch hints delivery)
Thanks for the patch. At a high level things look reasonable. Some feedback:
- patch needs to be against trunk
- you're not going to gain much from adding multiple threads to RowMutation delivery, since it has to go over the same MessagingService thread anyway. We do want to allow multiple threads to allow parallelism across multiple destinations, but trunk already supports this.
> Async hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Fix For: 1.2.0 beta 2
>
> Attachments: cassandra-1.1-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4761) Batch hints delivery
Posted by "Alexey Zotov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zotov updated CASSANDRA-4761:
------------------------------------
Attachment: cassandra-1.1-4761-async_hints.txt
> Batch hints delivery
> --------------------
>
> Key: CASSANDRA-4761
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4761
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.2
> Reporter: Alexey Zotov
> Assignee: Alexey Zotov
> Labels: hintedhandoff
> Attachments: cassandra-1.1-4761-async_hints.txt, notices.txt
>
>
> Hints delivery to remote DC can take a long time (currently we have 70 ms for each hint). In our estimates 700 MB of data (stored hints) will be transmitting to remote DC more than one day (in our case), it's unacceptable for us. We suggest to enter hints delivery using batch operations.
> What do you think about it? Is there some facts that won't allow to implement that mechanism?
> I'll try to implement it if you approve and clarify right way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira