You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Mike Drob (Jira)" <ji...@apache.org> on 2021/01/11 20:16:00 UTC
[jira] [Comment Edited] (SOLR-15052) Reducing overseer bottlenecks
using per-replica states
[ https://issues.apache.org/jira/browse/SOLR-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262918#comment-17262918 ]
Mike Drob edited comment on SOLR-15052 at 1/11/21, 8:15 PM:
------------------------------------------------------------
[~noble.paul] I see that you already merged this in to 8x.
Do you have any performance numbers that you can share about this? I did a lot of reviewing of the code for structure and clarity, etc, but I haven't had the opportunity to run this in a cluster.
Edit: I know that Ishan posted numbers at the beginning of this journey, but wanted to check if you had something run on the final version of this.
was (Author: mdrob):
[~noble.paul] I see that you already merged this in to 8x.
Do you have any performance numbers that you can share about this? I did a lot of reviewing of the code for structure and clarity, etc, but I haven't had the opportunity to run this in a cluster.
> Reducing overseer bottlenecks using per-replica states
> ------------------------------------------------------
>
> Key: SOLR-15052
> URL: https://issues.apache.org/jira/browse/SOLR-15052
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Ishan Chattopadhyaya
> Priority: Major
> Attachments: per-replica-states-gcp.pdf
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> This work has the same goal as SOLR-13951, that is to reduce overseer bottlenecks by avoiding replica state updates from going to the state.json via the overseer. However, the approach taken here is different from SOLR-13951 and hence this work supercedes that work.
> The design proposed is here: https://docs.google.com/document/d/1xdxpzUNmTZbk0vTMZqfen9R3ArdHokLITdiISBxCFUg/edit
> Briefly,
> # Every replica's state will be in a separate znode nested under the state.json. It has the name that encodes the replica name, state, leadership status.
> # An additional children watcher to be set on state.json for state changes.
> # Upon a state change, a ZK multi-op to delete the previous znode and add a new znode with new state.
> Differences between this and SOLR-13951,
> # In SOLR-13951, we planned to leverage shard terms for per shard states.
> # As a consequence, the code changes required for SOLR-13951 were massive (we needed a shard state provider abstraction and introduce it everywhere in the codebase).
> # This approach is a drastically simpler change and design.
> Credits for this design and the PR is due to [~noble.paul]. [~markrmiller@gmail.com], [~noble.paul] and I have collaborated on this effort. The reference branch takes a conceptually similar (but not identical) approach.
> I shall attach a PR and performance benchmarks shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org