You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org> on 2017/07/26 04:07:00 UTC
[jira] [Comment Edited] (SOLR-6086) Replica active during Warming
[ https://issues.apache.org/jira/browse/SOLR-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101141#comment-16101141 ]
Shalin Shekhar Mangar edited comment on SOLR-6086 at 7/26/17 4:06 AM:
----------------------------------------------------------------------
Here is patch with the fix and tests.
All in all, the following are the ways which cause this issue:
# Replica is the only one in the shard -- hence on startup becomes leader, skips recovery, becomes active without waiting for warming to complete
# Replica goes into recovery, discovers that it is the leader, becomes active without waiting for warming to complete
# When peersync fails but replication reports that there is nothing to replicate then replica becomes active without waiting for warming to complete. This can happen in the following ways:
## peersync could be skipped if firstTime=false which can happen if recovering on startup we discover that last operation in ulog had flag gap set or if the previous recovery attempt failed and we're retrying recovery
## peersync could fail if the peersync request failed due to exceptions
The fix in this patch (based on Tim's patch) ensure that there is a registered searcher before we publish any replica as active. Doing it inside RecoveryStrategy was not sufficient because that does not cover scenario 1. The tests in this patch inject failure into peer sync and simulate wrong index fingerprint computation to reproduce the problem. The scenarios that I could not reliably simulate were 2 and 3.1 but the fix should cover both of them. Tim's patch had a bug where openSearcher was called without returnSearcher=true. This can return a null future sometimes if there is an on deck searcher already and no new registration is needed. The correct fix is to send returnSearcher=true as well as waitFuture and check for both.
was (Author: shalinmangar):
Here is patch with the fix and tests.
All in all, the following are the ways which cause this issue:
# Replica is the only one in the shard -- hence on startup becomes leader, skips recovery, becomes active without waiting for warming to complete
# Replica goes into recovery, discovers that it is the leader, becomes active without waiting for warming to complete
# When peersync fails but replication reports that there is nothing to replicate then replica becomes active without waiting for warming to complete. This can happen in the following ways:
## peersync could be skipped if firstTime=false which can happen if recovering on startup we discover that last operation in ulog had flag gap set or if the previous recovery attempt failed and we're retrying recovery
## peersync could fail if the peersync request failed due to exceptions
The fix in this patch (based on Tim's patch) ensure that there is a registered searcher before we publish any replica as active. Doing it inside RecoveryStrategy was not sufficient because that does not cover scenario 1. The tests in this patch inject failure into peer sync and simulate wrong index fingerprint computation to reproduce the problem. The one scenario that I could not reliably simulate was 2 and 3.1 but the fix should cover both of them. Tim's patch had a bug where openSearcher was called without returnSearcher=true. This can return a null future sometimes if there is an on deck searcher already and no new registration is needed. The correct fix is to send returnSearcher=true as well as waitFuture and check for both.
> Replica active during Warming
> -----------------------------
>
> Key: SOLR-6086
> URL: https://issues.apache.org/jira/browse/SOLR-6086
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.6.1, 4.8.1
> Reporter: Ludovic Boutros
> Assignee: Shalin Shekhar Mangar
> Labels: difficulty-medium, impact-medium
> Attachments: SOLR-6086.patch, SOLR-6086.patch, SOLR-6086.patch, SOLR-6086-temp.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> At least with Solr 4.6.1, replica are considered as active during the warming process.
> This means that if you restart a replica or create a new one, queries will
> be send to this replica and the query will hang until the end of the warming
> process (If cold searchers are not used).
> You cannot add or restart a node silently anymore.
> I think that the fact that the replica is active is not a bad thing.
> But, the HttpShardHandler and the CloudSolrServer class should take the warming process in account.
> Currently, I have developped a new very simple component which check that a searcher is registered.
> I am also developping custom HttpShardHandler and CloudSolrServer classes which will check the warming process in addition to the ACTIVE status in the cluster state.
> This seems to be more a workaround than a solution but that's all I can do in this version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org