You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jam Luo <co...@gmail.com> on 2012/04/05 13:11:00 UTC
A endless loop in new SolrCloud probably
Hi
I deployed a solr cluster,the code version is
"NightlyBuilds apache-solr-4.0-2012-03-19_09-25-37".
Cluster has 4 nodes named "A", "B", "C", "D", "num_shards=2", A and
C in shard1 , B and D in shard2, A and B is the leader of their shard. It
has ran 2 days, added 20m docs, all of then are OK, but after this ,C
occured a Exception "org.apache.lucene.store.AlreadyClosedException: this
IndexWriter is closed", and jetty on C was not down, node C exist in
zookeeper in "/live_nodes". In this time, The A try to ask C to recover,
but C can't an response, so A get a Exception, the log is:
<D0><C5>Ϣ: try and ask http://node23:8983/solr to recover
<CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7>
org.apache.solr.update.processor.DistributedUpdateProcessor doFinish
<D0><C5>Ϣ: try and ask http://node23:8983/solr to recover
<CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7>
org.apache.solr.update.processor.DistributedUpdateProcessor doFinish
<D0><C5>Ϣ: Could not tell a replica to recover
org.apache.solr.client.solrj.SolrServerException: http://node23:8983/solr
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:496)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:347)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:816)
at
org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:176)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1549)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:441)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:262)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: <BE>ܾ<F8><C1><AC><BD><D3>
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:426)
... 34 more
There are 2400 lines log as above at 8:02:36 in 1 second, and it is not
over until I shut down the C's jetty server. The important is the client
can 't add any document to cluster in this time, it will recover after C
server shutdown also.
/clusterstate.json data in zookeeper is:
{"mblog":{
"shard1":{
"A:8983_solr_mblog":{
"shard":"shard1",
"leader":"true",
"state":"active",
"core":"mblog",
"collection":"mblog",
"node_name":"A:8983_solr",
"base_url":"http://A:8983/solr"},
"C:8983_solr_mblog":{
"shard":"shard1",
"state":"recovering",
"core":"mblog",
"collection":"mblog",
"node_name":"C:8983_solr",
"base_url":"http://C:8983/solr"}},
"shard2":{
"B:8983_solr_mblog":{
"shard":"shard2",
"state":"active",
"leader":"true",
"core":"mblog",
"collection":"mblog",
"node_name":"B:8983_solr",
"base_url":"http://B:8983/solr"},
"D:8983_solr_mblog":{
"shard":"shard2",
"state":"active",
"core":"mblog",
"collection":"mblog",
"node_name":"D:8983_solr",
"base_url":"http://D:8983/solr"}}}}
The last question is how to config the solr_cluster.properties file and
excute the Cluster Admin Commands described in
http://wiki.apache.org/solr/NewSolrCloudDesign ?
Re: A endless loop in new SolrCloud probably
Posted by Mark Miller <ma...@gmail.com>.
On Apr 5, 2012, at 7:11 AM, Jam Luo wrote:
> Hi
> I deployed a solr cluster,the code version is
> "NightlyBuilds apache-solr-4.0-2012-03-19_09-25-37".
> Cluster has 4 nodes named "A", "B", "C", "D", "num_shards=2", A and
> C in shard1 , B and D in shard2, A and B is the leader of their shard. It
> has ran 2 days, added 20m docs, all of then are OK, but after this ,C
> occured a Exception "org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter is closed", and jetty on C was not down, node C exist in
> zookeeper in "/live_nodes". In this time, The A try to ask C to recover,
> but C can't an response, so A get a Exception, the log is:
>
> <D0><C5>Ϣ: try and ask http://node23:8983/solr to recover
> <CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7>
> org.apache.solr.update.processor.DistributedUpdateProcessor doFinish
> <D0><C5>Ϣ: try and ask http://node23:8983/solr to recover
> <CB><C4><D4><C2> 04, 2012 8:02:36 <CF><C2><CE><E7>
> org.apache.solr.update.processor.DistributedUpdateProcessor doFinish
> <D0><C5>Ϣ: Could not tell a replica to recover
> org.apache.solr.client.solrj.SolrServerException: http://node23:8983/solr
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:496)
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:347)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:816)
> at
> org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:176)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1549)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:441)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:262)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:351)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.net.ConnectException: <BE>ܾ<F8><C1><AC><BD><D3>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
> at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:426)
> ... 34 more
>
> There are 2400 lines log as above at 8:02:36 in 1 second, and it is not
> over until I shut down the C's jetty server. The important is the client
> can 't add any document to cluster in this time, it will recover after C
> server shutdown also.
> /clusterstate.json data in zookeeper is:
> {"mblog":{
> "shard1":{
> "A:8983_solr_mblog":{
> "shard":"shard1",
> "leader":"true",
> "state":"active",
> "core":"mblog",
> "collection":"mblog",
> "node_name":"A:8983_solr",
> "base_url":"http://A:8983/solr"},
> "C:8983_solr_mblog":{
> "shard":"shard1",
> "state":"recovering",
> "core":"mblog",
> "collection":"mblog",
> "node_name":"C:8983_solr",
> "base_url":"http://C:8983/solr"}},
> "shard2":{
> "B:8983_solr_mblog":{
> "shard":"shard2",
> "state":"active",
> "leader":"true",
> "core":"mblog",
> "collection":"mblog",
> "node_name":"B:8983_solr",
> "base_url":"http://B:8983/solr"},
> "D:8983_solr_mblog":{
> "shard":"shard2",
> "state":"active",
> "core":"mblog",
> "collection":"mblog",
> "node_name":"D:8983_solr",
> "base_url":"http://D:8983/solr"}}}}
I'd try bouncing the node that had the error. Sounds like there is some bug involved, unless it's been fixed since that build - this should not occur (an IndexWriter being closed while something tries to use it).
>
>
> The last question is how to config the solr_cluster.properties file and
> excute the Cluster Admin Commands described in
> http://wiki.apache.org/solr/NewSolrCloudDesign ?
We didn't end up using that design for a lot of things - we should probably note that at the top of that doc.
- Mark Miller
lucidimagination.com