You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Kelly, Frank" <fr...@here.com> on 2017/01/17 17:01:05 UTC
SolrCloud: ClusterState says we are the leader but locally we don't
think so
Solr Version: 5.3.1
Configuration: 3 shards, 3 replicas each
After running out of heap memory recently (cause unknown) we've been successfully restarting nodes to recover.
Finally we did one restart and one of the nodes now says the following
2017-01-17 16:57:16.835 ERROR (qtp1395089624-17) [c:prod_us-east-1_here_account s:shard3 r:core_node26 x:prod_us-east-1_here_account_shard3_replica3] o.a.s.c.SolrCore org.apache.solr.common.SolrException: ClusterState says we are the leader (http://10.255.6.196:8983/solr/prod_us-east-1_here_account_shard3_replica3), but locally we don't think so. Request came from null
How can we recover from this (for Solr 5.3.1)?
Is there someway to force a new leader (I know the following feature exists but in 5.4.0 https://issues.apache.org/jira/browse/SOLR-7569)
Thanks!
-Frank
[Description: Macintosh HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]
Frank Kelly
Principal Software Engineer
HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
[Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]<http://360.here.com/> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif] <https://www.twitter.com/here> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif] <https://www.facebook.com/here> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif] <https://www.linkedin.com/company/heremaps> [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif] <https://www.instagram.com/here/>
Re: SolrCloud: ClusterState says we are the leader but locally we
don't think so
Posted by "Kelly, Frank" <fr...@here.com>.
We bounced ZooKeeper nodes one by one but not change
Since this is our Prod server (100M+ docs) we don¹t want to have to
reindex from scratch (takes 7+ days)
So we¹re considering editing /collections/<collection>/state.json via
zkcli.sh
Thoughts?
-Frank
On 1/17/17, 5:49 PM, "Pushkar Raste" <pu...@gmail.com> wrote:
>Try bouncing the overseer for your cluster.
>
>On Jan 17, 2017 12:01 PM, "Kelly, Frank" <fr...@here.com> wrote:
>
>> Solr Version: 5.3.1
>>
>> Configuration: 3 shards, 3 replicas each
>>
>> After running out of heap memory recently (cause unknown) we¹ve been
>> successfully restarting nodes to recover.
>>
>> Finally we did one restart and one of the nodes now says the following
>> 2017-01-17 16:57:16.835 ERROR (qtp1395089624-17)
>> [c:prod_us-east-1_here_account s:shard3 r:core_node26
>> x:prod_us-east-1_here_account_shard3_replica3] o.a.s.c.SolrCore
>> org.apache.solr.common.SolrException: ClusterState says we are the
>>leader
>>
>>(https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.255
>>.6.196%3A8983%2Fsolr%2Fprod_us-east-1_here_account_shard3_replica3&data=0
>>1%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d4034cd72254f72b85391feae
>>a64919%7C1&sdata=iN%2FH3MxId9b6aKcch%2FKSHYncsM5Ug6QJhRfGJ59G7uo%3D&reser
>>ved=0),
>> but locally we don't think so. Request came from null
>>
>> How can we recover from this (for Solr 5.3.1)?
>> Is there someway to force a new leader (I know the following feature
>> exists but in 5.4.0
>>https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues
>>.apache.org%2Fjira%2Fbrowse%2FSOLR-7569&data=01%7C01%7C%7Ce6ff77fca8124af
>>5cd7c08d43f2b2cf6%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=O7Ey7mUyuy
>>Kv%2FEPQvewITfxw8cmTEpUlV18hXGeIKUI%3D&reserved=0)
>>
>> Thanks!
>>
>> -Frank
>>
>> [image: Description: Macintosh
>>
>>HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PD
>>F:HERE_Logo_2016_POS_sRGB.pdf]
>>
>>
>>
>> *Frank Kelly*
>>
>> *Principal Software Engineer*
>>
>>
>>
>> HERE
>>
>> 5 Wayside Rd, Burlington, MA 01803, USA
>>
>> *42° 29' 7" N 71° 11' 32" W*
>>
>>
>> [image: Description:
>>
>>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima
>>ges/20160726_HERE_EMail_Signature_360.gif]
>>
>><https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.he
>>re.com%2F&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d4034cd72
>>254f72b85391feaea64919%7C1&sdata=wDqPoxFI%2BLRB7kZ%2FVkvGPZnz8vHK4VoeEjcK
>>%2FXxTo%2BU%3D&reserved=0> [image: Description:
>>
>>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima
>>ges/20160726_HERE_EMail_Signature_Twitter.gif]
>>
>><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.t
>>witter.com%2Fhere&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d
>>4034cd72254f72b85391feaea64919%7C1&sdata=La8C0bkCRhubUHz6KwLtMnLcv1Txglck
>>ua2YKDldM84%3D&reserved=0> [image: Description:
>>
>>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima
>>ges/20160726_HERE_EMail_Signature_FB.gif]
>>
>><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.f
>>acebook.com%2Fhere&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6
>>d4034cd72254f72b85391feaea64919%7C1&sdata=ngYNMYgIra7Kb1FPus5sp1b%2B5EBis
>>ws9%2Fsy10K%2B4l9I%3D&reserved=0> [image: Description:
>>
>>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima
>>ges/20160726_HERE_EMail_Signature_IN.gif]
>>
>><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.l
>>inkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08
>>d43f2b2cf6%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=KqQpmAcp2l2h4R%2B
>>slYZGf0%2Brwa8c9lzbzeIK%2B94ifrc%3D&reserved=0> [image: Description:
>>
>>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima
>>ges/20160726_HERE_EMail_Signature_Insta.gif]
>>
>><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.i
>>nstagram.com%2Fhere%2F&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6
>>%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=ys7%2FxZdOGXtlVMFF2%2FBn3At
>>pBN0GuOPePmf9sH4j8Dg%3D&reserved=0>
>>
Re: SolrCloud: ClusterState says we are the leader but locally we
don't think so
Posted by Pushkar Raste <pu...@gmail.com>.
Try bouncing the overseer for your cluster.
On Jan 17, 2017 12:01 PM, "Kelly, Frank" <fr...@here.com> wrote:
> Solr Version: 5.3.1
>
> Configuration: 3 shards, 3 replicas each
>
> After running out of heap memory recently (cause unknown) we’ve been
> successfully restarting nodes to recover.
>
> Finally we did one restart and one of the nodes now says the following
> 2017-01-17 16:57:16.835 ERROR (qtp1395089624-17)
> [c:prod_us-east-1_here_account s:shard3 r:core_node26
> x:prod_us-east-1_here_account_shard3_replica3] o.a.s.c.SolrCore
> org.apache.solr.common.SolrException: ClusterState says we are the leader
> (http://10.255.6.196:8983/solr/prod_us-east-1_here_account_shard3_replica3),
> but locally we don't think so. Request came from null
>
> How can we recover from this (for Solr 5.3.1)?
> Is there someway to force a new leader (I know the following feature
> exists but in 5.4.0 https://issues.apache.org/jira/browse/SOLR-7569)
>
> Thanks!
>
> -Frank
>
> [image: Description: Macintosh
> HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]
>
>
>
> *Frank Kelly*
>
> *Principal Software Engineer*
>
>
>
> HERE
>
> 5 Wayside Rd, Burlington, MA 01803, USA
>
> *42° 29' 7" N 71° 11' 32" W*
>
>
> [image: Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]
> <http://360.here.com/> [image: Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif]
> <https://www.twitter.com/here> [image: Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif]
> <https://www.facebook.com/here> [image: Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif]
> <https://www.linkedin.com/company/heremaps> [image: Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif]
> <https://www.instagram.com/here/>
>