You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Melvin Wang (JIRA)" <ji...@apache.org> on 2011/08/09 01:42:27 UTC
[jira] [Created] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
-------------------------------------------------------------------------------------
Key: CASSANDRA-3005
URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
Project: Cassandra
Issue Type: Improvement
Reporter: Melvin Wang
OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136557#comment-13136557 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
The v2 version of the patch is ready for review now.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137990#comment-13137990 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
by dropping you mean 'putting an earlier message to the end of the queue', right? deque uses a lock when multiple threads trying to add messages to the end of the queue, while concurrentLinkedQueue is a lock-free implementation. Another point is that we want to drop messages (because they are timed out) at two places:
1) where we send the message out to network
2) where we add messages.
The reason for this has been discussed some time ago, which is simply to prevent a backed-up queue hurting the system by having the system process already timed out messages.
For deque, there will be contention when we sending messages out to network while we are trying to examine and remove timed out messages( when adding messages)
Based on this consideration, I tried to use a lock-free implementation. I'll be boarding now. Will talk more after couple of hours.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
Posted by "Melvin Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang reassigned CASSANDRA-3005:
--------------------------------------
Assignee: Melvin Wang
> OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137823#comment-13137823 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
bq. I don't think it is a race per se. If the sending thread swaps the two queues, we then stop expiring messages and break.
Here's the problem:
{noformat}
+ Entry entry = producer.peek();
+ if (entry == null)
+ break;
+ if (entry.timestamp + DatabaseDescriptor.getRpcTimeout() < System.currentTimeMillis()) {
+ if (producer.poll() == null)
+ break; // consumer swaps the pointers so we end up having an empty queue here.
+ }
{noformat}
poll() may be an unexpired entry, not the one peeked, if the sending thread switched queues and also took the front entry off in between the peek and the poll. In other words: you still have the same problem you had to start with, just more subtle.
bq. The point of this design is that we don't need to lock the queues whereas BlockingQueue/Deque locks it
Producer/consumer is what the concurrent.Blocking classes are designed for. The "Blocking" means you can call take() and it will block until an entry is ready, not that it generates a lot of contention.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141876#comment-13141876 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
The reason I want to put it at the head is that minimizes the chance that this message would be timed out when it wouldn't have otherwise, by having to "wait in line" twice.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3005:
--------------------------------------
Attachment: 3005-v4.txt
bq. all the messages in 'active' comes before the ones in 'backlog'
True enough. v4 attached.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, 3005-v4.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083974#comment-13083974 ]
Stu Hood commented on CASSANDRA-3005:
-------------------------------------
Rather than nesting the Pair objects, you should probably create {code}class Entry { Message message; String id; long timestamp }{code} which will allow you to avoid boxing the long and the extra Pair.
> OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3005.
---------------------------------------
Resolution: Fixed
Fix Version/s: 1.1
Reviewer: jbellis (was: vijay)
Committed, thanks Melvin!
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Fix For: 1.1
>
> Attachments: 3005-v3.txt, 3005-v4.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Stu Hood (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136485#comment-13136485 ]
Stu Hood commented on CASSANDRA-3005:
-------------------------------------
We experienced a bit of cascading on a high-write-throughput cluster that likely would have been alleviated by this, so one of us will be getting back on it soon.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang updated CASSANDRA-3005:
-----------------------------------
Attachment: c3005-v4
Agreed on the arguments in the case of multiple producers ( I believe it is right when there is a single producer). I'd like to give it another try. I will put the polled element back to the consumer queue. This may result an earlier message being put to the end of the queue if the sending thread swaps, however, all the messages in the consumer queue will be processed before we swap again. Since it is OK for us to drop messages, processing an earlier message later may not be that bad.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127008#comment-13127008 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
Melvin, are you working on a patch incorporating Stu's feedback?
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3005:
--------------------------------------
Attachment: 3005-v3.txt
v2 still has a race if the sending thread swaps the queues between the peek and the poll in expire.
v3 attached that attempts to fix this by using deques instead of queues, and simply putting the polled entry "back" at the front of the queue if a swap happened.
v3 also cleans out the condition (unnecessary if using a BlockingQueue/Deque) and the pending count (just uses sum of the two queues' size()).
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141737#comment-13141737 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
ok, i cannot say for sure that ConcurrentLinkedQueue performs better than LinkedBlockingQueue. v4 and v3 differs mainly in the underlying queue implementation which is not hard to change if it is a bottleneck. And I agree that in run() ConcurrentLinkedQueue + condition variable does look like a blocking queue, so we can settle this. :) There is only one thing. Can we just append the element to 'active' rather than push back to 'backlog' in expireMessage? The reason (as I understand) is to save one round of racing to gain the lock to the beginning of 'backlog' because you need to race to get the lock to push at the beginning of queue then multiple threads will race to remove elements from the beginning of the queue as well (to expire messages), whereas you just race once to get the element to the end of 'active' which will remove elements in a single thread.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141934#comment-13141934 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
bq. The reason I want to put it at the head is that minimizes the chance that this message would be timed out when it wouldn't have otherwise, by having to "wait in line" twice.
Well, the way the code is structured is that all the messages in 'active' comes before the ones in 'backlog' so putting it at the end of 'active' queue doesn't mean to have it wait since it has to be after those messages anyway. I agree it is not a major issue here though.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
Posted by "Melvin Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang updated CASSANDRA-3005:
-----------------------------------
Attachment: c3005.patch
The goal here is to remove the expired messages as many as we can to avoid the queue going unbounded.
Two places to do this.
1) when we try to send the message, we drop it if it is already expired. This doesn't work if the connection is somehow slow because the queue may be piled up already. Thus we have
2) when we enqueue a message we try to remove some expired messages as well. This is tricky because we now have two threads trying to remove from one queue. Although the operation on the queue is protected, w/o synchronization, we have enqueue() trying to remove a message which is not he peeked a moment ago (since it may already be removed from the queue by run()).
To solve this w/o using lock, I use two queues. Whenever run() finishes one queue, it (and only it) will swap the pointers of queues to process the other one. ConcurrentLinkedQueue is implemented using a non-block algorithm, so we don't suffer much here.
> OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
Posted by "Melvin Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang updated CASSANDRA-3005:
-----------------------------------
Attachment: (was: c3005.patch)
> OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145994#comment-13145994 ]
Hudson commented on CASSANDRA-3005:
-----------------------------------
Integrated in Cassandra #1194 (See [https://builds.apache.org/job/Cassandra/1194/])
add message expiration logic to OutboundTcpConnection
patch by Melvin Wang and jbellis for CASSANDRA-3005
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199010
Files :
* /cassandra/trunk
* /cassandra/trunk/CHANGES.txt
* /cassandra/trunk/contrib
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
* /cassandra/trunk/src/java/org/apache/cassandra/net/MessagingService.java
* /cassandra/trunk/src/java/org/apache/cassandra/net/MessagingServiceMBean.java
* /cassandra/trunk/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Fix For: 1.1
>
> Attachments: 3005-v3.txt, 3005-v4.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue goes unboundedly without any backpressure logic
Posted by "Melvin Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang updated CASSANDRA-3005:
-----------------------------------
Attachment: c3005.patch
The previous patch was not up-to-date. sorry for the confusion.
> OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137924#comment-13137924 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
That may be safe if there is only ever a single producer, although it feels sketchy to me. It's clearly broken though with multiple producers, which is what we need to support. If a second producer adds a new Entry to the "empty" swapped producer queue, the first producer in the expire loop will happily discard it in poll().
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137994#comment-13137994 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
I suspect you're committing premature optimization. I've never seen LB[Q|D] be a significant source of contention.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Stu Hood (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138981#comment-13138981 ]
Stu Hood commented on CASSANDRA-3005:
-------------------------------------
(sidenote: we should consider counting dropped messages on the sending side the same way we count them on the receiving side: in the dropped message counts in MessagingService. Alternatively, it might be good to count in both locations)
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137711#comment-13137711 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
bq. v2 still has a race if the sending thread swaps the queues between the peek and the poll in expire.
I don't think it is a race per se. If the sending thread swaps the two queues, we then stop expiring messages and break. If it doesn't, then 'producer' is still pointing to the instance from which we peek. If the message we peek is meant be expired, and the sending thread swaps the queues, it will be dropped in run() anyway. The queues will only be swapped once 'consumer' is empty, i.e. nothing to send to the network.
The point of this design is that we don't need to lock the queues whereas BlockingQueue/Deque locks it). The goal is to be lock-free.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-3005:
-----------------------------------------
Summary: OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic (was: OutboundTcpConnection's sending queue goes unboundedly without any backpressure logic)
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143565#comment-13143565 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
+1
// Thanks for the intelligent review comments.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, 3005-v4.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Melvin Wang updated CASSANDRA-3005:
-----------------------------------
Attachment: c3005-v2
Sorry for the slackness. I got my hands really full these days. Incorporated Stu's feedback.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137981#comment-13137981 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
True, but why risk dropping something you don't have to when you can just use a deque?
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Melvin Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137897#comment-13137897 ]
Melvin Wang commented on CASSANDRA-3005:
----------------------------------------
bq. poll() may be an unexpired entry, not the one peeked, if the sending thread switched queues and also took the front entry off in between the peek and the poll. In other words: you still have the same problem you had to start with, just more subtle.
The logic in run() guarantees that if the sending thread swaps, the queue pointed by 'producer' will be an empty one from which producer.poll() will yield 'null' and we will break from there w/o any operation.
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3005) OutboundTcpConnection's sending
queue grows unboundedly without any backpressure logic
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141878#comment-13141878 ]
Jonathan Ellis commented on CASSANDRA-3005:
-------------------------------------------
It's enough of a corner case that I don't think the lock contention matters much, compared to losing messages entirely (from being expired).
> OutboundTcpConnection's sending queue grows unboundedly without any backpressure logic
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3005
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Melvin Wang
> Assignee: Melvin Wang
> Attachments: 3005-v3.txt, c3005-v2, c3005-v4, c3005.patch
>
>
> OutboundTcpConnection's sending queue unconditionally queues up the request and process them in sequence. Thinking about tagging the message coming in with timestamp and drop them before actually sending it if the message stays in the queue for too long, which is defined by the message's own time out value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira