[jira] [Created] (SOLR-13793) HTTPSolrCall makes cascading calls even when all replicas are down for a collection

["Kesharee Nandan Vishwakarma (Jira)" <ji...@apache.org>]

Kesharee Nandan Vishwakarma created SOLR-13793:
--------------------------------------------------

             Summary: HTTPSolrCall makes cascading calls even when all replicas are down for a collection
                 Key: SOLR-13793
                 URL: https://issues.apache.org/jira/browse/SOLR-13793
             Project: Solr
          Issue Type: Bug
      Security Level: Public (Default Security Level. Issues are Public)
          Components: SolrCloud
    Affects Versions: 6.6
            Reporter: Kesharee Nandan Vishwakarma


REMOTEQUERY action in HTTPSolrCall ends up making too many cascading remoteQuery calls when all all the replicas of a collection are in down state. 
This results in increase in thread count, unresponsive solr nodes and eventually node (one's which have this collection) going out of live nodes.

*Example scenario*: Consider a cluster with 3 nodes(solr1, solrw1, solr-overseer1). A collection is present on solr1, solrw1 but both replicas are in down state. When a search request is made to solr-overseer1, since replica is not present locally a remote query is made to solr1 (we also consider inactive slices/coreUrls), solr1 also doesn't see an active replica present locally, it forwards to solrw1, again solrw1 will forward request to solr1. This goes on till both of solr1, solrw1 become unresponsive. Attached logs for this.

This is happening because we are considering [inactive slices|https://github.com/apache/lucene-solr/blob/68fa249034ba8b273955f20097700dc2fbb7a800/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L913 ], [inactive coreUrl| https://github.com/apache/lucene-solr/blob/68fa249034ba8b273955f20097700dc2fbb7a800/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L929] while forwarding requests to nodes.

*Steps to reproduce*:
#  Bring down all replicas of a collection but ensure nodes containing them are up 
# Make any search call to any of solr nodes for this collection. 
 
*Possible fixes*: 
# Ensure we select only active slices/coreUrls before making remote queries
# Put a limit on cascading calls probably limit to number of replicas 
 
{noformat} 
solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:14.458 ERROR (qtp762152757-8772) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:14.451 ERROR (qtp762152757-8774) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:14.333 ERROR (qtp762152757-8771) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:12.661 INFO (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) solrw1_1 | 2019-09-24 09:35:12.284 ERROR (qtp762152757-8767) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:12.275 ERROR (qtp762152757-8768) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:12.274 ERROR (qtp762152757-8766) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more solrw1_1 | solrw1_1 | 2019-09-24 09:35:12.054 ERROR (qtp762152757-8764) [ ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) solrw1_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) solrw1_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) solrw1_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) solrw1_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) solrw1_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) solrw1_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) solrw1_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) solrw1_1 | at org.eclipse.jetty.server.Server.handle(Server.java:534) solrw1_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) solrw1_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) solrw1_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) solrw1_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) solrw1_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) solrw1_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) solrw1_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) solrw1_1 | at java.lang.Thread.run(Thread.java:748) solrw1_1 | Caused by: java.net.SocketTimeoutException: Read timed out solrw1_1 | at java.net.SocketInputStream.socketRead0(Native Method) solrw1_1 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:171) solrw1_1 | at java.net.SocketInputStream.read(SocketInputStream.java:141) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) solrw1_1 | at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) solrw1_1 | at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) solrw1_1 | at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) solrw1_1 | at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) solrw1_1 | at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) solrw1_1 | at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) solrw1_1 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) solrw1_1 | at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) solrw1_1 | at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) solrw1_1 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) solrw1_1 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) solrw1_1 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) solrw1_1 | at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632) solrw1_1 | ... 30 more
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org