You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by lansing <ha...@wantedanalytics.com> on 2014/01/30 16:26:36 UTC
Error when restarting solr servers
Hello,
Running solr cloud with 2 collections 5 shards and 3 replicas for each
collection, 5 zookeeper instance.
solr-4.6.0
apache-tomcat-7.0.39
zookeeper-3.4.5
jre1.7.0_21
When I try to restart a solr servers in my solr cloud I am receiving this
errors :
1861449 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.ShardLeaderElectionContext â Running the leader
process for shard shard1
1861451 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.ShardLeaderElectionContext â Checking if I should try
and be the leader.
1861451 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.ShardLeaderElectionContext â My last published State
was down, I won't be the leader.
1861451 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.ShardLeaderElectionContext â There may be a better
leader candidate than us - going back into recovery
1861452 [localhost-startStop-1-EventThread] INFO
org.apache.solr.update.DefaultSolrCoreState â Running recovery - first
canceling any ongoing recovery
1861452 [localhost-startStop-1-EventThread] WARN
org.apache.solr.cloud.RecoveryStrategy â Stopping recovery for
zkNodeName=core_node3core=Current1_shard1_replica3
1862223 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â
Finished recovery process. core=Current1_shard1_replica3
1862223 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â
Starting recovery process. core=Current1_shard1_replica3
recoveringAfterStartup=false
1862223 [RecoveryThread] ERROR org.apache.solr.update.UpdateLog â Exception
reading versions from log
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)
at sun.nio.ch.FileChannelImpl.read(Unknown Source)
at
org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:778)
at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:71)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog$FSReverseReader.<init>(TransactionLog.java:696)
at
org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:575)
at
org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:942)
at
org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:885)
at
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1042)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:280)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)
1862223 [RecoveryThread] ERROR org.apache.solr.cloud.RecoveryStrategy â
Error while trying to recover.
core=Current1_shard1_replica3:org.apache.solr.common.SolrException: Cloud
state sti
ll says we are leader.
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:354)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)
1862224 [RecoveryThread] ERRwwwwOR org.apache.solr.cloud.RecoveryStrategy â
Recovery failed - trying again... (0) core=Current1_shard1_replica3
1862224 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â
Wait 2.0 seconds before trying to recover again (1)
1862541 [localhost-startStop-1-SendThread(10.0.5.230:2281)] WARN
org.apache.zookeeper.ClientCnxn â Session 0x542fd3f2be100e6 for server
10.0.5.230/10.0.5.230:2281, unexpected error, clo
sing socket connection and attempting reconnect
java.io.IOException: Packet len11106511 is out of range!
at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
1862641 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO
org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state:
Disconnected type None
..............
1270268 [http-bio-8201-exec-26] INFO
org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for
coreNodeName: core_node10, state: recovering, checkLive: true, onlyIfLeader:
true
1270268 [http-bio-8201-exec-10] INFO
org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for
coreNodeName: core_node11, state: recovering, checkLive: true, onlyIfLeader:
true
1270268 [http-bio-8201-exec-26] ERROR org.apache.solr.core.SolrCore â
org.apache.solr.common.SolrException: We are not the leader
at
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
..............
1778261 [http-bio-8201-exec-1] INFO
org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for
coreNodeName: core_node15, state: recovering, checkLive: true, onlyIfLeader:
true
1778261 [http-bio-8201-exec-1] ERROR org.apache.solr.core.SolrCore â
org.apache.solr.common.SolrException: We are not the leader
at
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
1778261 [http-bio-8201-exec-1] INFO
org.apache.solr.servlet.SolrDispatchFilter â [admin] webapp=null
path=/admin/cores
params={coreNodeName=core_node15&state=recovering&nodeName=10.0.5.
230:8201_solr&action=PREPRECOVERY&checkLive=true&core=Current1_shard5_replica1&wt=javabin&onlyIfLeader=true&version=2}
status=400 QTime=0
1780238 [http-bio-8201-exec-29] ERROR org.apache.solr.core.SolrCore â
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:148)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
.............
2260241 [http-bio-8201-exec-32] ERROR org.apache.solr.core.SolrCore â
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1281)
at
org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:96)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:237)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.lucene.store.AlreadyClosedException: Already closed
at
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:251)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1372)
... 26 more
--
View this message in context: http://lucene.472066.n3.nabble.com/Error-when-restarting-solr-servers-tp4114461.html
Sent from the Solr - User mailing list archive at Nabble.com.