You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Erick Erickson (JIRA)" <ji...@apache.org> on 2018/10/30 16:46:00 UTC
[jira] [Commented] (SOLR-12940) Optimize and expunge deletes should
execute only on the leader instead of all replicas of the collection
[ https://issues.apache.org/jira/browse/SOLR-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669027#comment-16669027 ]
Erick Erickson commented on SOLR-12940:
---------------------------------------
I want to raise one concern here. This will essentially do a "full sync" of the index on the leader on all replicas in a collection at once, correct? That can move massive amounts of data around the network, and I've seen cascading failures result from that, endless recovery loops and the like.
Is there a way to throttle this?
I happen to be working on SOLR-12259 currently, which will add another operation to the mix so I'll link that in. The thrust of that JIRA is to rewrite all the segments to "do something", one case is to add docValues without re-indexing. Seems to me that the mechanism in this JIRA is applicable to SOLR-12259 too.
> Optimize and expunge deletes should execute only on the leader instead of all replicas of the collection
> --------------------------------------------------------------------------------------------------------
>
> Key: SOLR-12940
> URL: https://issues.apache.org/jira/browse/SOLR-12940
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Reporter: Shalin Shekhar Mangar
> Priority: Major
> Fix For: 7.6, master (8.0)
>
>
> Optimize and expunge deletes are broadcasted to all replicas of the collection (even to replicas of inactive shards!) but they don't need to be. We can optimize by only executing such commands on the leader and have the replicas pull the index over the network when ready.
> Synchronizing replication recovery to happen after optimize completes was trickier in the past when all we had was HTTP requests but now we have the terms mechanism which goes via ZK and can be relied on.
> This gives us a nice way to index/update fully and then call optimize/expunge deletes only on the leader while replicas continue to serve query traffic without disruption. This use-case will also need the ability to route queries only to replicas and not to the leader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org