You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by AlexeyK <le...@gmail.com> on 2012/10/23 13:43:36 UTC
SolrCloud - loop in recovery mode
I built a setup on my machine using SolrCloud release and Tomcat 6 (Windows
machine) for testing purposes.
I use solrconfig from the example, while solr.xml contains the following:
<cores adminPath="/admin/cores" defaultCoreName="collection1"
host="${host:}" hostPort="8080" hostContext="${hostContext:}"
zkClientTimeout="${zkClientTimeout:15000}">
<core name="collection1" collection="collection1"
instanceDir="collection1" />
<core name="core2" collection="collection1" instanceDir="core2"/>
<core name="core3" collection="collection1" instanceDir="core3"/>
<core name="core4" collection="collection1" instanceDir="core4"/>
<core name="core5" collection="collection2" instanceDir="core5"/>
<core name="core6" collection="collection2" instanceDir="core6"/>
</cores>
numShards = 2
ZooKeeper is embedded.
The index is empty.
*Sometimes *during the init phase the application is stuck trying to fetch
the leader:
SEVERE: Error while trying to recover.
core=core3:org.apache.solr.common.SolrException: No registered leader was
found, collection:collection1 slice:shard1
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderProps(ZkStateReader.java:413)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderProps(ZkStateReader.java:399)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:318)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:220)
This seems like an endless loop.
This happens for *all nodes of collection1*
After stop/start it *may* be OK.
The state in zookeeper is like this:
"collection1":{
"shard1":{
"range":"80000000-ffffffff",
"replicas":{
"akudinov-pc:8080_solr_collection1":{
"shard":"shard1",
"roles":null,
"state":"active",
"core":"collection1",
"collection":"collection1",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr",
"leader":"true"},
"akudinov-pc:8080_solr_core3":{
"shard":"shard1",
"roles":null,
"state":"active",
"core":"core3",
"collection":"collection1",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr"}}},
"shard2":{
"range":"0-7fffffff",
"replicas":{
"akudinov-pc:8080_solr_core2":{
"shard":"shard2",
"roles":null,
"state":"active",
"core":"core2",
"collection":"collection1",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr",
"leader":"true"},
"akudinov-pc:8080_solr_core4":{
"shard":"shard2",
"roles":null,
"state":"active",
"core":"core4",
"collection":"collection1",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr"}}}},
"collection2":{
"shard1":{
"range":"80000000-ffffffff",
"replicas":{"akudinov-pc:8080_solr_core5":{
"shard":"shard1",
"roles":null,
"state":"active",
"core":"core5",
"collection":"collection2",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr",
"leader":"true"}}},
"shard2":{
"range":"0-7fffffff",
"replicas":{"akudinov-pc:8080_solr_core6":{
"shard":"shard2",
"roles":null,
"state":"active",
"core":"core6",
"collection":"collection2",
"node_name":"akudinov-pc:8080_solr",
"base_url":"http://akudinov-pc:8080/solr",
"leader":"true"}}}}}
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
See https://issues.apache.org/jira/browse/SOLR-3991
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015772.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by Mark Miller <ma...@gmail.com>.
On a quick search, I didn't happen to see an open JIRA for this type of thing. Could you file one?
- Mark
On Oct 24, 2012, at 11:35 AM, AlexeyK <le...@gmail.com> wrote:
> The situation can be replayed on solr 4 (solrcloud):
> 1. Define the warmup query
> 2. Add spell checker configuration to the /select search handler
> 3. Set spellcheck.collation = true
>
> The server will stuck in init phase due to deadlock.
> Is there a bug open for this?
> Actually you cannot get collated spell check results together with a query
> result.
>
> The workaround is one of the following:
> 1. don't use warmup
> 2. don't use collation
> 3. don't define spell check for /select, but define a distinct handler and
> call it specifically
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015622.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
The situation can be replayed on solr 4 (solrcloud):
1. Define the warmup query
2. Add spell checker configuration to the /select search handler
3. Set spellcheck.collation = true
The server will stuck in init phase due to deadlock.
Is there a bug open for this?
Actually you cannot get collated spell check results together with a query
result.
The workaround is one of the following:
1. don't use warmup
2. don't use collation
3. don't define spell check for /select, but define a distinct handler and
call it specifically
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015622.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
It is actually connected to this:
https://gist.github.com/2880527
Once you have collation = true + warmup, the init is stuck on wait
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015593.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
After a little bit of investigation, it's about the searcher warmup that
doesn't happen.
I see the main thread waiting for the searcher. The warmup query handler is
stuck in another thread on the very same lock in getSearcher(), and no
notify() is called.
If I set the useColdSearcher = true, this obviously doesn't happen and the
application starts normally.
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015581.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
A small update: it seems that the zoo keeper cannot find children under the
/overseer/queue node
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015578.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by AlexeyK <le...@gmail.com>.
I only started learning the new features, so chances are it's about some
misconfiguration.
I removed the collection2 from the setup and indexed some files.
Now there is another pattern that stucks the init, and it's about the
overseer polling the queue:
Oct 24, 2012 2:18:52 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@5f00498c
main{StandardDirectoryReader(segments_2:3 _0(4.0.0.2):C8)}
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.ZooKeeperServer expire
INFO: Expiring session 0x13a92a392000000, timeout of 15000ms exceeded
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Processed session termination for sessionid: 0x13a92a392000000
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Got user-level KeeperException when processing
sessionid:0x13a92b5f1990000 type:delete cxid:0x282 zxid:0xfffffffffffffffe
txntype:unknown reqpath:n/a Error Path:/overseer_elect/leader
Error:KeeperErrorCode = NoNode for /overseer_elect/leader
Oct 24, 2012 2:19:04 PM org.apache.solr.common.cloud.SolrZkClient makePath
INFO: makePath: /overseer_elect/leader
Oct 24, 2012 2:19:04 PM org.apache.solr.cloud.Overseer start
INFO: Overseer (id=88544452827217920-akudinov-pc:8080_solr-n_0000000001)
starting
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Got user-level KeeperException when processing
sessionid:0x13a92b5f1990000 type:create cxid:0x287 zxid:0xfffffffffffffffe
txntype:unknown reqpath:n/a Error Path:/overseer Error:KeeperErrorCode =
NodeExists for /overseer
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Got user-level KeeperException when processing
sessionid:0x13a92b5f1990000 type:create cxid:0x288 zxid:0xfffffffffffffffe
txntype:unknown reqpath:n/a Error Path:/overseer Error:KeeperErrorCode =
NodeExists for /overseer
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Got user-level KeeperException when processing
sessionid:0x13a92b5f1990000 type:create cxid:0x289 zxid:0xfffffffffffffffe
txntype:unknown reqpath:n/a Error Path:/overseer Error:KeeperErrorCode =
NodeExists for /overseer
Oct 24, 2012 2:19:04 PM org.apache.zookeeper.server.PrepRequestProcessor
pRequest
INFO: Got user-level KeeperException when processing
sessionid:0x13a92b5f1990000 type:create cxid:0x28a zxid:0xfffffffffffffffe
txntype:unknown reqpath:n/a Error Path:/overseer Error:KeeperErrorCode =
NodeExists for /overseer
Oct 24, 2012 2:19:04 PM org.apache.solr.cloud.OverseerCollectionProcessor
run
INFO: Process current queue of collection creations
Oct 24, 2012 2:19:04 PM org.apache.solr.cloud.Overseer$ClusterStateUpdater
run
INFO: Starting to work on the main queue
Can you give a clue of what's happening with it?
Now my setup is:
collection1
2 shards
4 cores
There are several documents in both shards, automatically distributed by
solrcloud.
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-loop-in-recovery-mode-tp4015330p4015574.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud - loop in recovery mode
Posted by Mark Miller <ma...@gmail.com>.
This sounds like two issues. One, you are triggering a leader election
somehow. Do you see session expiration errors in the logs? You may
need to raise the timeout.
The second issue is probably related to what came out of
https://issues.apache.org/jira/browse/SOLR-3939
Shards that have no versions (because they are empty or just
replicated) cannot become the leader.
So if your indexes are empty and a leader election is triggered, no
one ends up becoming the leader.
This will be addressed in a release shortly, but the simple workaround
for now is simply to restart the shard and you will get a leader.
Next, you want to figure out why a leader election is being triggered
at that time (eg a zk session timeout?). Is the system under heavy
load at that point?
- Mark