You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Cao Manh Dat (JIRA)" <ji...@apache.org> on 2017/10/06 14:14:00 UTC
[jira] [Comment Edited] (SOLR-11443) Remove the usage of workqueue
for Overseer
[ https://issues.apache.org/jira/browse/SOLR-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194627#comment-16194627 ]
Cao Manh Dat edited comment on SOLR-11443 at 10/6/17 2:13 PM:
--------------------------------------------------------------
Patch for this ticket. The idea here is simple, we peek for 1000 messages in the queue, processed them, write new clusterstate to ZK, then poll out these messages. So we only poll out processed messages when new clusterstate is written.
In case of Overseer get restarted, all the uncommitted messages still in the queue ( no need for workqueue, only keep them for backward compatible ), we will reprocess them and still achieve the desired state.
Here are some benchmark number ( OverseerTest.testPerformance() )
Before optimize : {{avgRequestsPerSecond: 1551.8934622998179}}
After optmize : {{avgRequestsPerSecond: 3425.594762960455}}
was (Author: caomanhdat):
Patch for this ticket. The idea here is simple, we peek for 1000 messages in the queue, processed them, write new clusterstate to ZK, then poll out these messages. So we only poll out processed messages when new clusterstate is written.
In case of Overseer get restarted, all the uncommitted messages still in the queue, we will reprocess them and still achieve the desired state.
Here are some benchmark number ( OverseerTest.testPerformance() )
Before optimize : {{avgRequestsPerSecond: 1551.8934622998179}}
After optmize : {{avgRequestsPerSecond: 3425.594762960455}}
> Remove the usage of workqueue for Overseer
> ------------------------------------------
>
> Key: SOLR-11443
> URL: https://issues.apache.org/jira/browse/SOLR-11443
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Cao Manh Dat
> Assignee: Cao Manh Dat
> Attachments: SOLR-11443.patch
>
>
> If we can remove the usage of workqueue, We can save a lot of IO blocking in Overseer, hence boost performance a lot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org