Recover - Read timed out

["Trym R. Møller" <tr...@sigmat.dk>]

Hi

I experience that a Solr looses its connection with Zookeeper and 
re-establish it. After Solr is reconnection to Zookeeper it begins to 
recover its replicas. It has been missing the connection approximately 
10 seconds and meanwhile the leader slice has received some documents 
(maybe about 1000 documents). Solr fails to update using peer sync and 
fails afterwards to do a full replicate with the log message below. The 
Solr from where the documents are replicated doesn't log anything when 
the replication is in progress. The full replica continues to fail with 
the "read time out" for about 10 hours and then Solr gives up.

1. How can I get more information about why the Read time out happens?
2. It seems like the Solr from where it replicates leaks a http 
connection each time (and a thread) having about 18.000 threads in 8 hours.

Any comments are welcome.

Best regards Trym

Apr 21, 2012 10:14:11 AM org.apache.solr.common.SolrException log
SEVERE: Error while trying to 
recover:org.apache.solr.client.solrj.SolrServerException: 
http://solr-ip:8983/solr/mycollection_slice21_shard2
         at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:493)
         at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:264)
         at 
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:103)
         at 
org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:180)
         at 
org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:156)
         at 
org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:170)
         at 
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:120)
         at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:341)
         at 
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:206)
Caused by: java.net.SocketTimeoutException: Read timed out
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
         at 
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
         at 
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
         at 
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
         at 
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
         at 
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
         at 
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
         at 
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
         at 
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
         at 
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
         at 
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
         at 
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
         at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:440)
         ... 8 more