[jira] [Commented] (ZOOKEEPER-1609) Improve ZooKeeper performane under mixed workload

["Alexander Shraer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538912#comment-13538912 ] 

Alexander Shraer commented on ZOOKEEPER-1609:
---------------------------------------------

This mixed workload problem is exactly what I meant in my previous comments on ZK-1505. A write should only block the client's operations (and possibly other clients hashed to the same worker thread) but not other clients. To handle watch order you guys pointed out I suggest to have the handling thread responsible for watch events and all communication with the clients it handles - otherwise indeed its difficult to order watch events with getData completions, etc. 

>It will have to unblock the write requests in the same order that it sent to the leader

I'm not sure the order in which the write requests were sent to the leader is important. What's important is the order they come back from the leader (which has to be applied in the same way on all followers) - as long as the writes can't be reordered after they returned to the follower we should be fine. This probably means that handling such commits has to be sequential.

Alex


                
> Improve ZooKeeper performane under mixed workload
> -------------------------------------------------
>
>                 Key: ZOOKEEPER-1609
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1609
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: server
>    Affects Versions: 3.4.3
>            Reporter: Thawan Kooburat
>
> ZOOKEEPER-1505 allows 1 write or N reads to pass through the CommitProcessor at any given time. I did performance experiment similar to http://wiki.apache.org/hadoop/ZooKeeper/Performance and found that read throughput drop dramatically when there are write requests. After a bit more investigation, I found that
> the biggest bottleneck is at the request queue entering the CommitProcessor.
> When the CommitProcessor see any write request, it will need to block the entire pipeline and wait until matching commit from the leader. This means that all read requests (including ping request) won't be able to go through. The time spent waiting for commit from the leader far exceed the time spent waiting for 1 write to goes through the CommitProcessor. 
> The current plan is to create multiple request queues at the front of the CommitProcessor. Requests are hashed using sessionId and send to one of the queue. Whenever, the CommitProcessor saw a write request on one of the queue it moves on to process read requests. It will have to unblock the write requests in the same order that it sent to the leader, so it may need to maintain a separate list to keep track of that. 
> The correctness is the same as having more learners in the ensemble. Sessions which are hashed onto a different queue is similar to sessions connecting to a different learners in the ensemble. 
> I am hoping that this will improve read throughput and reduce disconnect rate on an ensemble with large number of clients  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira