Connection Reset Errors on a Distributed Index

["harish.agarwal" <ha...@gmail.com>]

Hello,

For reference, I've posted about this before (but have new information now):
http://lucene.472066.n3.nabble.com/Connection-reset-errors-during-commits-optimize-td484058.html#a484058

and have seen other similar posts as well:
http://lucene.472066.n3.nabble.com/Question-on-Solr-Distributed-Search-td495188.html#a495191

During the aftermath of commits on a distributed index (3 shards, about 3M
documents each with many many facets), I'm getting ConnectionReset errors
(see below for the full trace).  The place in the code where it happens is
where the 'master server' is waiting on results from the other shards.  I've
been combing through the logs of all the shards at the time of the
exceptions and have noticed that every exception is thrown on a search which
does not appear in one of the other shard's logs.  In addition, the shard
which doesn't record the search is usually busy warming up uninverted fields
for facet searches, via a newSearcher query, at the time.  These exceptions
are problematic because they tend to slow the server down to a crawl,
sometimes permanently.

Does anyone have any advice for me on how to proceed?  Is it possible that a
shard would stop responding while uninverting fields?

Thanks,
-Harish

Trace:

SEVERE: org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: java.net.SocketException:
Connection reset
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:282)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.client.solrj.SolrServerException:
java.net.SocketException: Connection reset        at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:472)       
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)       
at
org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:422)       
at
org.apache.solr.handler.component.HttpCommComponent$1.call(SearchHandler.java:394)       
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)       
at java.util.concurrent.FutureTask.run(FutureTask.java:138)        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)       
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)       
at java.util.concurrent.FutureTask.run(FutureTask.java:138)        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)       
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)       
... 1 moreCaused by: java.net.SocketException: Connection reset        at
java.net.SocketInputStream.read(SocketInputStream.java:168)        at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)        at
java.io.BufferedInputStream.read(BufferedInputStream.java:237)        at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)       
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
        at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
        at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)       
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)       
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)       
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)       
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)       
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)       
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)       
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)       
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:416)       
... 11 more


-- 
View this message in context: http://lucene.472066.n3.nabble.com/Connection-Reset-Errors-on-a-Distributed-Index-tp822049p822049.html
Sent from the Solr - User mailing list archive at Nabble.com.