Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

We are running SolrCloud in AWS and using their auto scaling groups to spin
up new Solr replicas when CPU utilization exceeds a threshold for a period
of time. All is well until the replicas are terminated when CPU utilization
falls below another threshold. What happens is that index updates sent to
the Solr leader hang forever in both the Solr leader and the SolrJ client
app. Searches work fine.  Here are 2 thread stack traces from the Solr
leader and 2 from the client app:

1) Solr-leader thread doing a distributed commit:

Thread 23527: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
(Compiled frame)
 - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
 - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
(Compiled frame)
 -
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=61 (Compiled frame)
 - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
 -
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
 -
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=62, line=300 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=574, line=520 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
(Interpreted frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Interpreted frame)
 -
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=101, line=293 (Compiled frame)
 -
org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
@bci=127, line=226 (Interpreted frame)
 -
org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
@bci=112, line=195 (Interpreted frame)
 -
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
@bci=174, line=1250 (Interpreted frame)
 -
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
@bci=61, line=157 (Interpreted frame)
 -
org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.update.processor.UpdateRequestProcessor,
org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
(Interpreted frame)
 -
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
frame)
 -
org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
frame)
 -
org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
@bci=1380, line=417 (Compiled frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
(Compiled frame)
 -
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
 - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
 - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
 - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
 -
org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled frame)
 - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
 -
org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled frame)
 - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
 -
org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
 - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
 -
org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=146, line=368 (Compiled frame)
 - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
@bci=323, line=489 (Compiled frame)
 - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() @bci=1,
line=53 (Interpreted frame)
 -
org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
@bci=13, line=953 (Interpreted frame)
 -
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
@bci=5, line=1014 (Interpreted frame)
 - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
(Compiled frame)
 - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
(Compiled frame)
 - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
line=72 (Compiled frame)
 - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
@bci=100, line=264 (Compiled frame)
 -
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=608 (Interpreted frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

 2) Solr-leader thread doing a delete by query:

Thread 19580: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=186 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=834 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
int) @bci=67, line=867 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
@bci=17, line=1197 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21,
line=214 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
(Compiled frame)
 -
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
@bci=81, line=391 (Compiled frame)
 - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
@bci=35, line=98 (Compiled frame)
 - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
(Compiled frame)
 -
org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
@bci=1088, line=1088 (Compiled frame)
 -
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
@bci=14, line=825 (Compiled frame)
 -
org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
@bci=61, line=121 (Compiled frame)
 -
org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.client.solrj.request.UpdateRequest,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208, line=150
(Compiled frame)
 -
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68, line=111
(Compiled frame)
 -
org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse,
org.apache.solr.common.util.ContentStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18, line=58
(Compiled frame)
 -
org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse,
org.apache.solr.common.util.ContentStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165, line=92
(Compiled frame)
 -
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled
frame)
 -
org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
frame)
 -
org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
@bci=1380, line=417 (Compiled frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
(Compiled frame)
 -
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
 - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
 - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
 - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
 -
org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled frame)
 - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
 -
org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled frame)
 - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
 -
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
 -
org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
 - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
 -
org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=146, line=368 (Compiled frame)
 - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
@bci=323, line=489 (Compiled frame)
 - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
(Compiled frame)
 - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
(Compiled frame)
 - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
line=72 (Compiled frame)
 - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
@bci=100, line=264 (Compiled frame)
 -
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=608 (Interpreted frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

 3) SolrJ client thread doing commit:

 Thread 1872: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
(Compiled frame)
 - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
 - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
(Compiled frame)
 -
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
 -
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=5, line=115 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=62 (Compiled frame)
 - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
 -
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
 -
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=73, line=301 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=597, line=522 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
@bci=132, line=285 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=838, line=640 (Compiled frame)
 -
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
@bci=17, line=117 (Interpreted frame)
 - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
@bci=16, line=168 (Interpreted frame)
 - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
(Interpreted frame)

 4) SolrJ client thread doing delete by query:

 Thread 1873: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
(Compiled frame)
 - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
 - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
(Compiled frame)
 -
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
 -
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=5, line=115 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=62 (Compiled frame)
 - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
 -
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
 -
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
 -
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=73, line=301 (Compiled frame)
 -
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
 -
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=597, line=522 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
 -
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
frame)
 -
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
@bci=132, line=285 (Compiled frame)
 -
org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=838, line=640 (Compiled frame)
 -
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
@bci=17, line=117 (Compiled frame)
 - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
int) @bci=21, line=285 (Compiled frame)
 - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
@bci=3, line=271 (Compiled frame)

It looks as if the Solr-leader threads are waiting on a response from the
terminated replicas, but I would have expected an IOException to occur. Any
ideas on what may be happening?

 Thanks,
 Peter

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

> As of 4.10, commits/optimize etc are executed in parallel.
Excellent - thanks.

On Tue, Dec 16, 2014 at 6:51 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
>
> On Tue, Dec 16, 2014 at 11:34 AM, Peter Keegan <pe...@gmail.com>
> wrote:
> >
> > > A distributed update is streamed to all available replicas in parallel.
> >
> > Hmm, that's not what I'm seeing with 4.6.1, as I tail the logs on leader
> > and replicas. Mark Miller comments on this last May:
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201404.mbox/%3CetPan.534d8d6d.74b0dc51.13a79@airmetal.local%3E
> >
> >
> Yes, sorry I didn't notice that you are on 4.6.1. This was changed in 4.10
> with https://issues.apache.org/jira/browse/SOLR-6264
>
> As of 4.10, commits/optimize etc are executed in parallel.
>
>
> > On Mon, Dec 15, 2014 at 8:11 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> > >
> > > On Mon, Dec 15, 2014 at 8:41 PM, Peter Keegan <pe...@gmail.com>
> > > wrote:
> > > >
> > > > If a timeout occurs, does the distributed update then go to the next
> > > > replica?
> > > >
> > >
> > > A distributed update is streamed to all available replicas in parallel.
> > >
> > >
> > > >
> > > > On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
> > > > shalinmangar@gmail.com> wrote:
> > > > >
> > > > > Sorry I should have specified. These timeouts go inside the
> > <solrcloud>
> > > > > section and apply for inter-shard update requests only. The socket
> > and
> > > > > connection timeout inside the shardHandlerFactory section apply for
> > > > > inter-shard search requests.
> > > > >
> > > > > On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <
> > peterlkeegan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Btw, are the following timeouts still supported in solr.xml, and
> do
> > > > they
> > > > > > only apply to distributed search?
> > > > > >
> > > > > >   <shardHandlerFactory name="shardHandlerFactory"
> > > > > >     class="HttpShardHandlerFactory">
> > > > > >     <int name="socketTimeout">${socketTimeout:0}</int>
> > > > > >     <int name="connTimeout">${connTimeout:0}</int>
> > > > > >   </shardHandlerFactory>
> > > > > >
> > > > > > Thanks,
> > > > > > Peter
> > > > > >
> > > > > > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <
> > > peterlkeegan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > No, I wasn't aware of these. I will give that a try. If I stop
> > the
> > > > Solr
> > > > > > > jetty service manually, things recover fine, but the hang
> occurs
> > > > when I
> > > > > > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader
> > > reports
> > > > a
> > > > > > > 15-sec timeout from the stopped node, and expires the session,
> > but
> > > > the
> > > > > > Solr
> > > > > > > leader never gets notified. This seems like a bug in ZK.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Peter
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > > > > > shalinmangar@gmail.com> wrote:
> > > > > > >
> > > > > > >> Do you have distribUpdateConnTimeout and
> distribUpdateSoTimeout
> > > set
> > > > to
> > > > > > >> reasonable values in your solr.xml? These are the timeouts
> used
> > > for
> > > > > > >> inter-shard update requests.
> > > > > > >>
> > > > > > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <
> > > > peterlkeegan@gmail.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > We are running SolrCloud in AWS and using their auto scaling
> > > > groups
> > > > > to
> > > > > > >> spin
> > > > > > >> > up new Solr replicas when CPU utilization exceeds a
> threshold
> > > for
> > > > a
> > > > > > >> period
> > > > > > >> > of time. All is well until the replicas are terminated when
> > CPU
> > > > > > >> utilization
> > > > > > >> > falls below another threshold. What happens is that index
> > > updates
> > > > > sent
> > > > > > >> to
> > > > > > >> > the Solr leader hang forever in both the Solr leader and the
> > > SolrJ
> > > > > > >> client
> > > > > > >> > app. Searches work fine.  Here are 2 thread stack traces
> from
> > > the
> > > > > Solr
> > > > > > >> > leader and 2 from the client app:
> > > > > > >> >
> > > > > > >> > 1) Solr-leader thread doing a distributed commit:
> > > > > > >> >
> > > > > > >> > Thread 23527: (state = IN_NATIVE)
> > > > > > >> >  -
> > > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > > >> byte[],
> > > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > > imprecise)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > > @bci=79,
> > > > > > >> line=150
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int)
> @bci=11,
> > > > > line=121
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > > >> @bci=71,
> > > > > > >> > line=166 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > > @bci=1,
> > > > > > >> line=90
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=2, line=61 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > > @bci=38,
> > > > > > >> line=254
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=62, line=300 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715
> > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=574, line=520 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > > (Compiled
> > > > > > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175,
> > line=395
> > > > > > >> > (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=17, line=199 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=101, line=293 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > > > > >> > @bci=127, line=226 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > > > > >> > java.util.List,
> > > > org.apache.solr.common.params.ModifiableSolrParams)
> > > > > > >> > @bci=112, line=195 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > > > >> > @bci=174, line=1250 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > > > >> > @bci=61, line=157 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > > > > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100,
> > > > line=69
> > > > > > >> > (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43,
> line=135
> > > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115,
> > line=1859
> > > > > > >> (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31,
> line=721
> > > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> > > boolean)
> > > > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> > > @bci=5,
> > > > > > >> line=201
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > > > frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > > > >> > @bci=146, line=368 (Compiled frame)
> > > > > > >> >  -
> > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > > > >> > @bci=323, line=489 (Compiled frame)
> > > > > > >> >  -
> > > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > > > > > >> @bci=1,
> > > > > > >> > line=53 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > > > > >> > @bci=13, line=953 (Interpreted frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > > > > >> > @bci=5, line=1014 (Interpreted frame)
> > > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948,
> > > > line=921
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable()
> @bci=51,
> > > > > > line=240
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > > > @bci=51,
> > > > > > >> > line=72 (Compiled frame)
> > > > > > >> >  -
> > > > > >
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > > > >> > @bci=100, line=264 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> > > @bci=47,
> > > > > > >> line=543
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted
> > frame)
> > > > > > >> >
> > > > > > >> >  2) Solr-leader thread doing a delete by query:
> > > > > > >> >
> > > > > > >> > Thread 19580: (state = BLOCKED)
> > > > > > >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled
> frame;
> > > > > > >> information
> > > > > > >> > may be imprecise)
> > > > > > >> >  -
> > java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > > > > > >> @bci=14,
> > > > > > >> > line=186 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > > > > >> > @bci=1, line=834 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > > > > >> > int) @bci=67, line=867 (Compiled frame)
> > > > > > >> >  -
> > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > > > > >> > @bci=17, line=1197 (Compiled frame)
> > > > > > >> >  -
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > > > > > @bci=21,
> > > > > > >> > line=214 (Compiled frame)
> > > > > > >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4,
> > > > line=290
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > > > > >> > @bci=81, line=391 (Compiled frame)
> > > > > > >> >  -
> > > > org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > > > > >> > @bci=35, line=98 (Compiled frame)
> > > > > > >> >  - org.apache.solr.update.SolrCmdDistributor.finish()
> @bci=8,
> > > > > line=62
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > > > > >> > @bci=1088, line=1088 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > > > >> > @bci=14, line=825 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > > > >> > @bci=61, line=121 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > > @bci=208,
> > > > > > >> line=150
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse,
> > java.io.InputStream,
> > > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > > @bci=68,
> > > > > > >> line=111
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > > > >> > org.apache.solr.common.util.ContentStream,
> > > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > > @bci=18,
> > > > > > >> line=58
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > > > >> > org.apache.solr.common.util.ContentStream,
> > > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > > @bci=165,
> > > > > > >> line=92
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=129,
> line=74
> > > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43,
> line=135
> > > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115,
> > line=1859
> > > > > > >> (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31,
> line=721
> > > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> > > boolean)
> > > > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> > > @bci=5,
> > > > > > >> line=201
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > > > frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > > > (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > > > >> > org.eclipse.jetty.server.Request,
> > > > > > javax.servlet.http.HttpServletRequest,
> > > > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > > > >> > @bci=146, line=368 (Compiled frame)
> > > > > > >> >  -
> > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > > > >> > @bci=323, line=489 (Compiled frame)
> > > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597,
> > > > line=866
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable()
> @bci=51,
> > > > > > line=240
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > > > @bci=51,
> > > > > > >> > line=72 (Compiled frame)
> > > > > > >> >  -
> > > > > >
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > > > >> > @bci=100, line=264 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> > > @bci=47,
> > > > > > >> line=543
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted
> > frame)
> > > > > > >> >
> > > > > > >> >  3) SolrJ client thread doing commit:
> > > > > > >> >
> > > > > > >> >  Thread 1872: (state = IN_NATIVE)
> > > > > > >> >  -
> > > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > > >> byte[],
> > > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > > imprecise)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > > @bci=79,
> > > > > > >> line=150
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int)
> @bci=11,
> > > > > line=121
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > > >> @bci=71,
> > > > > > >> > line=166 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > > @bci=1,
> > > > > > >> line=90
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > > >> > @bci=5, line=115 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=2, line=62 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > > @bci=38,
> > > > > > >> line=254
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=73, line=301 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=597, line=522 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > > (Compiled
> > > > > > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184,
> > line=396
> > > > > > >> (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=17, line=199 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > > > >> > @bci=132, line=285 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=838, line=640 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > > > >> > @bci=17, line=117 (Interpreted frame)
> > > > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean,
> > > > boolean)
> > > > > > >> > @bci=16, line=168 (Interpreted frame)
> > > > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3,
> > > > line=146
> > > > > > >> > (Interpreted frame)
> > > > > > >> >
> > > > > > >> >  4) SolrJ client thread doing delete by query:
> > > > > > >> >
> > > > > > >> >  Thread 1873: (state = IN_NATIVE)
> > > > > > >> >  -
> > > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > > >> byte[],
> > > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > > imprecise)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > > @bci=79,
> > > > > > >> line=150
> > > > > > >> > (Compiled frame)
> > > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int)
> @bci=11,
> > > > > line=121
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > > >> @bci=71,
> > > > > > >> > line=166 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > > @bci=1,
> > > > > > >> line=90
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > > >> > @bci=5, line=115 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > > >> > @bci=2, line=62 (Compiled frame)
> > > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > > @bci=38,
> > > > > > >> line=254
> > > > > > >> > (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=73, line=301 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > > >> > org.apache.http.HttpClientConnection,
> > > > > > >> org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> > > (Compiled
> > > > > > >> frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=597, line=522 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > > >> > org.apache.http.HttpRequest,
> > > org.apache.http.protocol.HttpContext)
> > > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > > (Compiled
> > > > > > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184,
> > line=396
> > > > > > >> (Compiled
> > > > > > >> > frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=17, line=199 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > > > >> > @bci=132, line=285 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > > >> > @bci=838, line=640 (Compiled frame)
> > > > > > >> >  -
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > > > >> > @bci=17, line=117 (Compiled frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > > > > >> > int) @bci=21, line=285 (Compiled frame)
> > > > > > >> >  -
> > > > > > >>
> > > > >
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > > > > >> > @bci=3, line=271 (Compiled frame)
> > > > > > >> >
> > > > > > >> > It looks as if the Solr-leader threads are waiting on a
> > response
> > > > > from
> > > > > > >> the
> > > > > > >> > terminated replicas, but I would have expected an
> IOException
> > to
> > > > > > occur.
> > > > > > >> Any
> > > > > > >> > ideas on what may be happening?
> > > > > > >> >
> > > > > > >> >  Thanks,
> > > > > > >> >  Peter
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Regards,
> > > > > > >> Shalin Shekhar Mangar.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Shalin Shekhar Mangar.
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> > >
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr hangs on distributed updates

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Tue, Dec 16, 2014 at 11:34 AM, Peter Keegan <pe...@gmail.com>
wrote:
>
> > A distributed update is streamed to all available replicas in parallel.
>
> Hmm, that's not what I'm seeing with 4.6.1, as I tail the logs on leader
> and replicas. Mark Miller comments on this last May:
>
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201404.mbox/%3CetPan.534d8d6d.74b0dc51.13a79@airmetal.local%3E
>
>
Yes, sorry I didn't notice that you are on 4.6.1. This was changed in 4.10
with https://issues.apache.org/jira/browse/SOLR-6264

As of 4.10, commits/optimize etc are executed in parallel.


> On Mon, Dec 15, 2014 at 8:11 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
> >
> > On Mon, Dec 15, 2014 at 8:41 PM, Peter Keegan <pe...@gmail.com>
> > wrote:
> > >
> > > If a timeout occurs, does the distributed update then go to the next
> > > replica?
> > >
> >
> > A distributed update is streamed to all available replicas in parallel.
> >
> >
> > >
> > > On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
> > > shalinmangar@gmail.com> wrote:
> > > >
> > > > Sorry I should have specified. These timeouts go inside the
> <solrcloud>
> > > > section and apply for inter-shard update requests only. The socket
> and
> > > > connection timeout inside the shardHandlerFactory section apply for
> > > > inter-shard search requests.
> > > >
> > > > On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <
> peterlkeegan@gmail.com>
> > > > wrote:
> > > >
> > > > > Btw, are the following timeouts still supported in solr.xml, and do
> > > they
> > > > > only apply to distributed search?
> > > > >
> > > > >   <shardHandlerFactory name="shardHandlerFactory"
> > > > >     class="HttpShardHandlerFactory">
> > > > >     <int name="socketTimeout">${socketTimeout:0}</int>
> > > > >     <int name="connTimeout">${connTimeout:0}</int>
> > > > >   </shardHandlerFactory>
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > > > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <
> > peterlkeegan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > No, I wasn't aware of these. I will give that a try. If I stop
> the
> > > Solr
> > > > > > jetty service manually, things recover fine, but the hang occurs
> > > when I
> > > > > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader
> > reports
> > > a
> > > > > > 15-sec timeout from the stopped node, and expires the session,
> but
> > > the
> > > > > Solr
> > > > > > leader never gets notified. This seems like a bug in ZK.
> > > > > >
> > > > > > Thanks,
> > > > > > Peter
> > > > > >
> > > > > >
> > > > > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > > > > shalinmangar@gmail.com> wrote:
> > > > > >
> > > > > >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout
> > set
> > > to
> > > > > >> reasonable values in your solr.xml? These are the timeouts used
> > for
> > > > > >> inter-shard update requests.
> > > > > >>
> > > > > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <
> > > peterlkeegan@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > We are running SolrCloud in AWS and using their auto scaling
> > > groups
> > > > to
> > > > > >> spin
> > > > > >> > up new Solr replicas when CPU utilization exceeds a threshold
> > for
> > > a
> > > > > >> period
> > > > > >> > of time. All is well until the replicas are terminated when
> CPU
> > > > > >> utilization
> > > > > >> > falls below another threshold. What happens is that index
> > updates
> > > > sent
> > > > > >> to
> > > > > >> > the Solr leader hang forever in both the Solr leader and the
> > SolrJ
> > > > > >> client
> > > > > >> > app. Searches work fine.  Here are 2 thread stack traces from
> > the
> > > > Solr
> > > > > >> > leader and 2 from the client app:
> > > > > >> >
> > > > > >> > 1) Solr-leader thread doing a distributed commit:
> > > > > >> >
> > > > > >> > Thread 23527: (state = IN_NATIVE)
> > > > > >> >  -
> > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > >> byte[],
> > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > imprecise)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > @bci=79,
> > > > > >> line=150
> > > > > >> > (Compiled frame)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > > line=121
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > >> @bci=71,
> > > > > >> > line=166 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > @bci=1,
> > > > > >> line=90
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=2, line=61 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > @bci=38,
> > > > > >> line=254
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=62, line=300 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715
> > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=574, line=520 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > (Compiled
> > > > > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175,
> line=395
> > > > > >> > (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=17, line=199 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=101, line=293 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > > > >> > @bci=127, line=226 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > > > >> > java.util.List,
> > > org.apache.solr.common.params.ModifiableSolrParams)
> > > > > >> > @bci=112, line=195 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > > >> > @bci=174, line=1250 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > > >> > @bci=61, line=157 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > > > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100,
> > > line=69
> > > > > >> > (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115,
> line=1859
> > > > > >> (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> > boolean)
> > > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> > @bci=5,
> > > > > >> line=201
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > > frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >>
> > > >
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > > >> > @bci=146, line=368 (Compiled frame)
> > > > > >> >  -
> > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > > >> > @bci=323, line=489 (Compiled frame)
> > > > > >> >  -
> > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > > > > >> @bci=1,
> > > > > >> > line=53 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > > > >> > @bci=13, line=953 (Interpreted frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > > > >> > @bci=5, line=1014 (Interpreted frame)
> > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948,
> > > line=921
> > > > > >> > (Compiled frame)
> > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > > > line=240
> > > > > >> > (Compiled frame)
> > > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > > @bci=51,
> > > > > >> > line=72 (Compiled frame)
> > > > > >> >  -
> > > > >
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > > >> > @bci=100, line=264 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> > @bci=47,
> > > > > >> line=543
> > > > > >> > (Compiled frame)
> > > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted
> frame)
> > > > > >> >
> > > > > >> >  2) Solr-leader thread doing a delete by query:
> > > > > >> >
> > > > > >> > Thread 19580: (state = BLOCKED)
> > > > > >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > > > > >> information
> > > > > >> > may be imprecise)
> > > > > >> >  -
> java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > > > > >> @bci=14,
> > > > > >> > line=186 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > > > >> > @bci=1, line=834 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > > > >> > int) @bci=67, line=867 (Compiled frame)
> > > > > >> >  -
> > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > > > >> > @bci=17, line=1197 (Compiled frame)
> > > > > >> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > > > > @bci=21,
> > > > > >> > line=214 (Compiled frame)
> > > > > >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4,
> > > line=290
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > > > >> > @bci=81, line=391 (Compiled frame)
> > > > > >> >  -
> > > org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > > > >> > @bci=35, line=98 (Compiled frame)
> > > > > >> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8,
> > > > line=62
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > > > >> > @bci=1088, line=1088 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > > >> > @bci=14, line=825 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > > >> > @bci=61, line=121 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > @bci=208,
> > > > > >> line=150
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse,
> java.io.InputStream,
> > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > @bci=68,
> > > > > >> line=111
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > > >> > org.apache.solr.common.util.ContentStream,
> > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > @bci=18,
> > > > > >> line=58
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > > >> > org.apache.solr.common.util.ContentStream,
> > > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> > @bci=165,
> > > > > >> line=92
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> > > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115,
> line=1859
> > > > > >> (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> > boolean)
> > > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> > @bci=5,
> > > > > >> line=201
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > > frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > > (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >>
> > > >
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > > >> > org.eclipse.jetty.server.Request,
> > > > > javax.servlet.http.HttpServletRequest,
> > > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > > >> > @bci=146, line=368 (Compiled frame)
> > > > > >> >  -
> > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > > >> > @bci=323, line=489 (Compiled frame)
> > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597,
> > > line=866
> > > > > >> > (Compiled frame)
> > > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > > > line=240
> > > > > >> > (Compiled frame)
> > > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > > @bci=51,
> > > > > >> > line=72 (Compiled frame)
> > > > > >> >  -
> > > > >
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > > >> > @bci=100, line=264 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> > @bci=47,
> > > > > >> line=543
> > > > > >> > (Compiled frame)
> > > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted
> frame)
> > > > > >> >
> > > > > >> >  3) SolrJ client thread doing commit:
> > > > > >> >
> > > > > >> >  Thread 1872: (state = IN_NATIVE)
> > > > > >> >  -
> > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > >> byte[],
> > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > imprecise)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > @bci=79,
> > > > > >> line=150
> > > > > >> > (Compiled frame)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > > line=121
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > >> @bci=71,
> > > > > >> > line=166 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > @bci=1,
> > > > > >> line=90
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > >> > @bci=5, line=115 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=2, line=62 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > @bci=38,
> > > > > >> line=254
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=73, line=301 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=597, line=522 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > (Compiled
> > > > > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184,
> line=396
> > > > > >> (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=17, line=199 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > > >> > @bci=132, line=285 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=838, line=640 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > > >> > @bci=17, line=117 (Interpreted frame)
> > > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean,
> > > boolean)
> > > > > >> > @bci=16, line=168 (Interpreted frame)
> > > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3,
> > > line=146
> > > > > >> > (Interpreted frame)
> > > > > >> >
> > > > > >> >  4) SolrJ client thread doing delete by query:
> > > > > >> >
> > > > > >> >  Thread 1873: (state = IN_NATIVE)
> > > > > >> >  -
> > java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > > >> byte[],
> > > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > > imprecise)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> > @bci=79,
> > > > > >> line=150
> > > > > >> > (Compiled frame)
> > > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > > line=121
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > > >> @bci=71,
> > > > > >> > line=166 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> > @bci=1,
> > > > > >> line=90
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > >> > @bci=137, line=281 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > > >> > @bci=5, line=115 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=16, line=92 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > > >> > @bci=2, line=62 (Compiled frame)
> > > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> > @bci=38,
> > > > > >> line=254
> > > > > >> > (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > > >> > @bci=8, line=289 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >>
> > > > >
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > > >> > @bci=1, line=252 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > > >> > @bci=6, line=191 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=73, line=301 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > > >> > org.apache.http.HttpClientConnection,
> > > > > >> org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=60, line=127 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> > (Compiled
> > > > > >> frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=597, line=522 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > > >> > org.apache.http.HttpRequest,
> > org.apache.http.protocol.HttpContext)
> > > > > >> > @bci=344, line=906 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> > (Compiled
> > > > > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > > >> > @bci=6, line=784 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184,
> line=396
> > > > > >> (Compiled
> > > > > >> > frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=17, line=199 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > > >> > @bci=132, line=285 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > > >> > @bci=838, line=640 (Compiled frame)
> > > > > >> >  -
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > > >> > @bci=17, line=117 (Compiled frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > > > >> > int) @bci=21, line=285 (Compiled frame)
> > > > > >> >  -
> > > > > >>
> > > >
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > > > >> > @bci=3, line=271 (Compiled frame)
> > > > > >> >
> > > > > >> > It looks as if the Solr-leader threads are waiting on a
> response
> > > > from
> > > > > >> the
> > > > > >> > terminated replicas, but I would have expected an IOException
> to
> > > > > occur.
> > > > > >> Any
> > > > > >> > ideas on what may be happening?
> > > > > >> >
> > > > > >> >  Thanks,
> > > > > >> >  Peter
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Regards,
> > > > > >> Shalin Shekhar Mangar.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Shalin Shekhar Mangar.
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

> A distributed update is streamed to all available replicas in parallel.

Hmm, that's not what I'm seeing with 4.6.1, as I tail the logs on leader
and replicas. Mark Miller comments on this last May:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201404.mbox/%3CetPan.534d8d6d.74b0dc51.13a79@airmetal.local%3E

On Mon, Dec 15, 2014 at 8:11 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
>
> On Mon, Dec 15, 2014 at 8:41 PM, Peter Keegan <pe...@gmail.com>
> wrote:
> >
> > If a timeout occurs, does the distributed update then go to the next
> > replica?
> >
>
> A distributed update is streamed to all available replicas in parallel.
>
>
> >
> > On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> > >
> > > Sorry I should have specified. These timeouts go inside the <solrcloud>
> > > section and apply for inter-shard update requests only. The socket and
> > > connection timeout inside the shardHandlerFactory section apply for
> > > inter-shard search requests.
> > >
> > > On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <pe...@gmail.com>
> > > wrote:
> > >
> > > > Btw, are the following timeouts still supported in solr.xml, and do
> > they
> > > > only apply to distributed search?
> > > >
> > > >   <shardHandlerFactory name="shardHandlerFactory"
> > > >     class="HttpShardHandlerFactory">
> > > >     <int name="socketTimeout">${socketTimeout:0}</int>
> > > >     <int name="connTimeout">${connTimeout:0}</int>
> > > >   </shardHandlerFactory>
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <
> peterlkeegan@gmail.com>
> > > > wrote:
> > > >
> > > > > No, I wasn't aware of these. I will give that a try. If I stop the
> > Solr
> > > > > jetty service manually, things recover fine, but the hang occurs
> > when I
> > > > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader
> reports
> > a
> > > > > 15-sec timeout from the stopped node, and expires the session, but
> > the
> > > > Solr
> > > > > leader never gets notified. This seems like a bug in ZK.
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > > >
> > > > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > > > shalinmangar@gmail.com> wrote:
> > > > >
> > > > >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout
> set
> > to
> > > > >> reasonable values in your solr.xml? These are the timeouts used
> for
> > > > >> inter-shard update requests.
> > > > >>
> > > > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <
> > peterlkeegan@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > We are running SolrCloud in AWS and using their auto scaling
> > groups
> > > to
> > > > >> spin
> > > > >> > up new Solr replicas when CPU utilization exceeds a threshold
> for
> > a
> > > > >> period
> > > > >> > of time. All is well until the replicas are terminated when CPU
> > > > >> utilization
> > > > >> > falls below another threshold. What happens is that index
> updates
> > > sent
> > > > >> to
> > > > >> > the Solr leader hang forever in both the Solr leader and the
> SolrJ
> > > > >> client
> > > > >> > app. Searches work fine.  Here are 2 thread stack traces from
> the
> > > Solr
> > > > >> > leader and 2 from the client app:
> > > > >> >
> > > > >> > 1) Solr-leader thread doing a distributed commit:
> > > > >> >
> > > > >> > Thread 23527: (state = IN_NATIVE)
> > > > >> >  -
> java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > >> byte[],
> > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > imprecise)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> @bci=79,
> > > > >> line=150
> > > > >> > (Compiled frame)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > line=121
> > > > >> > (Compiled frame)
> > > > >> >  -
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > >> @bci=71,
> > > > >> > line=166 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> @bci=1,
> > > > >> line=90
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > >> > @bci=137, line=281 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=16, line=92 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=2, line=61 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> @bci=38,
> > > > >> line=254
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > >> > @bci=8, line=289 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > >> > @bci=1, line=252 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > >> > @bci=6, line=191 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=62, line=300 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=60, line=127 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715
> (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=574, line=520 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=344, line=906 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> (Compiled
> > > > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > >> > @bci=6, line=784 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > > > >> > (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=17, line=199 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=101, line=293 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > > >> > @bci=127, line=226 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > > >> > java.util.List,
> > org.apache.solr.common.params.ModifiableSolrParams)
> > > > >> > @bci=112, line=195 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > >> > @bci=174, line=1250 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > >> > @bci=61, line=157 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100,
> > line=69
> > > > >> > (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > > > >> (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> boolean)
> > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> @bci=5,
> > > > >> line=201
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > frame)
> > > > >> >  -
> > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >>
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >>
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >>
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > >> > @bci=146, line=368 (Compiled frame)
> > > > >> >  -
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > >> > @bci=323, line=489 (Compiled frame)
> > > > >> >  -
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > > > >> @bci=1,
> > > > >> > line=53 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > > >> > @bci=13, line=953 (Interpreted frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > > >> > @bci=5, line=1014 (Interpreted frame)
> > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948,
> > line=921
> > > > >> > (Compiled frame)
> > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > > line=240
> > > > >> > (Compiled frame)
> > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > @bci=51,
> > > > >> > line=72 (Compiled frame)
> > > > >> >  -
> > > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > >> > @bci=100, line=264 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> @bci=47,
> > > > >> line=543
> > > > >> > (Compiled frame)
> > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > > >> >
> > > > >> >  2) Solr-leader thread doing a delete by query:
> > > > >> >
> > > > >> > Thread 19580: (state = BLOCKED)
> > > > >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > > > >> information
> > > > >> > may be imprecise)
> > > > >> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > > > >> @bci=14,
> > > > >> > line=186 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > > >> > @bci=1, line=834 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > > >> > int) @bci=67, line=867 (Compiled frame)
> > > > >> >  -
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > > >> > @bci=17, line=1197 (Compiled frame)
> > > > >> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > > > @bci=21,
> > > > >> > line=214 (Compiled frame)
> > > > >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4,
> > line=290
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > > >> > @bci=81, line=391 (Compiled frame)
> > > > >> >  -
> > org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > > >> > @bci=35, line=98 (Compiled frame)
> > > > >> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8,
> > > line=62
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > > >> > @bci=1088, line=1088 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > >> > @bci=14, line=825 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > >> > @bci=61, line=121 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> @bci=208,
> > > > >> line=150
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> @bci=68,
> > > > >> line=111
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > >> > org.apache.solr.common.util.ContentStream,
> > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> @bci=18,
> > > > >> line=58
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse,
> > > > >> > org.apache.solr.common.util.ContentStream,
> > > > >> > org.apache.solr.update.processor.UpdateRequestProcessor)
> @bci=165,
> > > > >> line=92
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> > > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > > > >> (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > >> > org.apache.solr.request.SolrRequestHandler,
> > > > >> > org.apache.solr.request.SolrQueryRequest,
> > > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain,
> boolean)
> > > > >> > @bci=1380, line=417 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain)
> @bci=5,
> > > > >> line=201
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> > frame)
> > > > >> >  -
> > > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >>
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > > (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >>
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >>
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > >> > org.eclipse.jetty.server.Request,
> > > > javax.servlet.http.HttpServletRequest,
> > > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> > (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > >> > @bci=146, line=368 (Compiled frame)
> > > > >> >  -
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > >> > @bci=323, line=489 (Compiled frame)
> > > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597,
> > line=866
> > > > >> > (Compiled frame)
> > > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > > line=240
> > > > >> > (Compiled frame)
> > > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> > @bci=51,
> > > > >> > line=72 (Compiled frame)
> > > > >> >  -
> > > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > >> > @bci=100, line=264 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > >> > @bci=1, line=608 (Interpreted frame)
> > > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
> @bci=47,
> > > > >> line=543
> > > > >> > (Compiled frame)
> > > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > > >> >
> > > > >> >  3) SolrJ client thread doing commit:
> > > > >> >
> > > > >> >  Thread 1872: (state = IN_NATIVE)
> > > > >> >  -
> java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > >> byte[],
> > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > imprecise)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> @bci=79,
> > > > >> line=150
> > > > >> > (Compiled frame)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > line=121
> > > > >> > (Compiled frame)
> > > > >> >  -
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > >> @bci=71,
> > > > >> > line=166 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> @bci=1,
> > > > >> line=90
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > >> > @bci=137, line=281 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > >> > @bci=5, line=115 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=16, line=92 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=2, line=62 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> @bci=38,
> > > > >> line=254
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > >> > @bci=8, line=289 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > >> > @bci=1, line=252 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > >> > @bci=6, line=191 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=73, line=301 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=60, line=127 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=597, line=522 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=344, line=906 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> (Compiled
> > > > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > >> > @bci=6, line=784 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > > >> (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=17, line=199 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > >> > @bci=132, line=285 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=838, line=640 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > >> > @bci=17, line=117 (Interpreted frame)
> > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean,
> > boolean)
> > > > >> > @bci=16, line=168 (Interpreted frame)
> > > > >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3,
> > line=146
> > > > >> > (Interpreted frame)
> > > > >> >
> > > > >> >  4) SolrJ client thread doing delete by query:
> > > > >> >
> > > > >> >  Thread 1873: (state = IN_NATIVE)
> > > > >> >  -
> java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > > >> byte[],
> > > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> > imprecise)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int)
> @bci=79,
> > > > >> line=150
> > > > >> > (Compiled frame)
> > > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > > line=121
> > > > >> > (Compiled frame)
> > > > >> >  -
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > > >> @bci=71,
> > > > >> > line=166 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer()
> @bci=1,
> > > > >> line=90
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > >> > @bci=137, line=281 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > >> > @bci=5, line=115 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=16, line=92 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > >> > @bci=2, line=62 (Compiled frame)
> > > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse()
> @bci=38,
> > > > >> line=254
> > > > >> > (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > >> > @bci=8, line=289 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >>
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > >> > @bci=1, line=252 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > >> > @bci=6, line=191 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=73, line=301 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > >> > org.apache.http.HttpClientConnection,
> > > > >> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=60, line=127 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717
> (Compiled
> > > > >> frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=597, line=522 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > >> > org.apache.http.HttpRequest,
> org.apache.http.protocol.HttpContext)
> > > > >> > @bci=344, line=906 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805
> (Compiled
> > > > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > >> > @bci=6, line=784 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > > >> (Compiled
> > > > >> > frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=17, line=199 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > >> > @bci=132, line=285 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > >> > @bci=838, line=640 (Compiled frame)
> > > > >> >  -
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > >> > @bci=17, line=117 (Compiled frame)
> > > > >> >  -
> > > > >>
> > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > > >> > int) @bci=21, line=285 (Compiled frame)
> > > > >> >  -
> > > > >>
> > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > > >> > @bci=3, line=271 (Compiled frame)
> > > > >> >
> > > > >> > It looks as if the Solr-leader threads are waiting on a response
> > > from
> > > > >> the
> > > > >> > terminated replicas, but I would have expected an IOException to
> > > > occur.
> > > > >> Any
> > > > >> > ideas on what may be happening?
> > > > >> >
> > > > >> >  Thanks,
> > > > >> >  Peter
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >> Shalin Shekhar Mangar.
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> > >
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr hangs on distributed updates

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Mon, Dec 15, 2014 at 8:41 PM, Peter Keegan <pe...@gmail.com>
wrote:
>
> If a timeout occurs, does the distributed update then go to the next
> replica?
>

A distributed update is streamed to all available replicas in parallel.


>
> On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
> >
> > Sorry I should have specified. These timeouts go inside the <solrcloud>
> > section and apply for inter-shard update requests only. The socket and
> > connection timeout inside the shardHandlerFactory section apply for
> > inter-shard search requests.
> >
> > On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <pe...@gmail.com>
> > wrote:
> >
> > > Btw, are the following timeouts still supported in solr.xml, and do
> they
> > > only apply to distributed search?
> > >
> > >   <shardHandlerFactory name="shardHandlerFactory"
> > >     class="HttpShardHandlerFactory">
> > >     <int name="socketTimeout">${socketTimeout:0}</int>
> > >     <int name="connTimeout">${connTimeout:0}</int>
> > >   </shardHandlerFactory>
> > >
> > > Thanks,
> > > Peter
> > >
> > > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <pe...@gmail.com>
> > > wrote:
> > >
> > > > No, I wasn't aware of these. I will give that a try. If I stop the
> Solr
> > > > jetty service manually, things recover fine, but the hang occurs
> when I
> > > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports
> a
> > > > 15-sec timeout from the stopped node, and expires the session, but
> the
> > > Solr
> > > > leader never gets notified. This seems like a bug in ZK.
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > >
> > > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > > shalinmangar@gmail.com> wrote:
> > > >
> > > >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set
> to
> > > >> reasonable values in your solr.xml? These are the timeouts used for
> > > >> inter-shard update requests.
> > > >>
> > > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <
> peterlkeegan@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > We are running SolrCloud in AWS and using their auto scaling
> groups
> > to
> > > >> spin
> > > >> > up new Solr replicas when CPU utilization exceeds a threshold for
> a
> > > >> period
> > > >> > of time. All is well until the replicas are terminated when CPU
> > > >> utilization
> > > >> > falls below another threshold. What happens is that index updates
> > sent
> > > >> to
> > > >> > the Solr leader hang forever in both the Solr leader and the SolrJ
> > > >> client
> > > >> > app. Searches work fine.  Here are 2 thread stack traces from the
> > Solr
> > > >> > leader and 2 from the client app:
> > > >> >
> > > >> > 1) Solr-leader thread doing a distributed commit:
> > > >> >
> > > >> > Thread 23527: (state = IN_NATIVE)
> > > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > >> byte[],
> > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> imprecise)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > >> line=150
> > > >> > (Compiled frame)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > line=121
> > > >> > (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > >> @bci=71,
> > > >> > line=166 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > > >> line=90
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > >> > @bci=137, line=281 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=16, line=92 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=2, line=61 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > >> line=254
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > >> > @bci=8, line=289 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > >> > @bci=1, line=252 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > >> > @bci=6, line=191 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=62, line=300 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=60, line=127 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=574, line=520 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=344, line=906 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > >> > @bci=6, line=784 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > > >> > (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=17, line=199 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=101, line=293 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > >> > @bci=127, line=226 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > >> > java.util.List,
> org.apache.solr.common.params.ModifiableSolrParams)
> > > >> > @bci=112, line=195 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > >> > @bci=174, line=1250 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > >> > @bci=61, line=157 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100,
> line=69
> > > >> > (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > >> > org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > > >> (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > >> > org.apache.solr.request.SolrRequestHandler,
> > > >> > org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > >> > @bci=1380, line=417 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > > >> line=201
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> frame)
> > > >> >  -
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> (Compiled
> > > >> frame)
> > > >> >  -
> > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > (Compiled
> > > >> > frame)
> > > >> >  -
> > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > >> > @bci=146, line=368 (Compiled frame)
> > > >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > >> > @bci=323, line=489 (Compiled frame)
> > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > > >> @bci=1,
> > > >> > line=53 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > >> > @bci=13, line=953 (Interpreted frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > >> > @bci=5, line=1014 (Interpreted frame)
> > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948,
> line=921
> > > >> > (Compiled frame)
> > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > line=240
> > > >> > (Compiled frame)
> > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> @bci=51,
> > > >> > line=72 (Compiled frame)
> > > >> >  -
> > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > >> > @bci=100, line=264 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > >> > @bci=1, line=608 (Interpreted frame)
> > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > > >> line=543
> > > >> > (Compiled frame)
> > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > >> >
> > > >> >  2) Solr-leader thread doing a delete by query:
> > > >> >
> > > >> > Thread 19580: (state = BLOCKED)
> > > >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > > >> information
> > > >> > may be imprecise)
> > > >> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > > >> @bci=14,
> > > >> > line=186 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > >> > @bci=1, line=834 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > >> > int) @bci=67, line=867 (Compiled frame)
> > > >> >  -
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > >> > @bci=17, line=1197 (Compiled frame)
> > > >> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > > @bci=21,
> > > >> > line=214 (Compiled frame)
> > > >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4,
> line=290
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > >> > @bci=81, line=391 (Compiled frame)
> > > >> >  -
> org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > >> > @bci=35, line=98 (Compiled frame)
> > > >> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8,
> > line=62
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > >> > @bci=1088, line=1088 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > >> > @bci=14, line=825 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > >> > @bci=61, line=121 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> > > >> line=150
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> > > >> line=111
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse,
> > > >> > org.apache.solr.common.util.ContentStream,
> > > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> > > >> line=58
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse,
> > > >> > org.apache.solr.common.util.ContentStream,
> > > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> > > >> line=92
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> > > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > >> > org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > > >> (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > >> > org.apache.solr.request.SolrRequestHandler,
> > > >> > org.apache.solr.request.SolrQueryRequest,
> > > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > >> > @bci=1380, line=417 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > > >> line=201
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled
> frame)
> > > >> >  -
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137
> (Compiled
> > > >> frame)
> > > >> >  -
> > org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> > (Compiled
> > > >> > frame)
> > > >> >  -
> > org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> > (Compiled
> > > >> > frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >>
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > >> > org.eclipse.jetty.server.Request,
> > > javax.servlet.http.HttpServletRequest,
> > > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116
> (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > >> > @bci=146, line=368 (Compiled frame)
> > > >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > >> > @bci=323, line=489 (Compiled frame)
> > > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597,
> line=866
> > > >> > (Compiled frame)
> > > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > > line=240
> > > >> > (Compiled frame)
> > > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle()
> @bci=51,
> > > >> > line=72 (Compiled frame)
> > > >> >  -
> > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > >> > @bci=100, line=264 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > >> > @bci=1, line=608 (Interpreted frame)
> > > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > > >> line=543
> > > >> > (Compiled frame)
> > > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > >> >
> > > >> >  3) SolrJ client thread doing commit:
> > > >> >
> > > >> >  Thread 1872: (state = IN_NATIVE)
> > > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > >> byte[],
> > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> imprecise)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > >> line=150
> > > >> > (Compiled frame)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > line=121
> > > >> > (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > >> @bci=71,
> > > >> > line=166 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > > >> line=90
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > >> > @bci=137, line=281 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > >> > @bci=5, line=115 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=16, line=92 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=2, line=62 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > >> line=254
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > >> > @bci=8, line=289 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > >> > @bci=1, line=252 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > >> > @bci=6, line=191 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=73, line=301 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=60, line=127 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=597, line=522 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=344, line=906 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > >> > @bci=6, line=784 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > >> (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=17, line=199 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > >> > @bci=132, line=285 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=838, line=640 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > >> > @bci=17, line=117 (Interpreted frame)
> > > >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean,
> boolean)
> > > >> > @bci=16, line=168 (Interpreted frame)
> > > >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3,
> line=146
> > > >> > (Interpreted frame)
> > > >> >
> > > >> >  4) SolrJ client thread doing delete by query:
> > > >> >
> > > >> >  Thread 1873: (state = IN_NATIVE)
> > > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > > >> byte[],
> > > >> > int, int, int) @bci=0 (Compiled frame; information may be
> imprecise)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > >> line=150
> > > >> > (Compiled frame)
> > > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> > line=121
> > > >> > (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > >> @bci=71,
> > > >> > line=166 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > > >> line=90
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > >> > @bci=137, line=281 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > >> > @bci=5, line=115 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=16, line=92 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > >> > @bci=2, line=62 (Compiled frame)
> > > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > >> line=254
> > > >> > (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > >> > @bci=8, line=289 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >>
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > >> > @bci=1, line=252 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > >> > @bci=6, line=191 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=73, line=301 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > >> > org.apache.http.HttpClientConnection,
> > > >> org.apache.http.protocol.HttpContext)
> > > >> > @bci=60, line=127 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > > >> frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=597, line=522 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > >> > @bci=344, line=906 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > >> > @bci=6, line=784 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > >> (Compiled
> > > >> > frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=17, line=199 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > >> > @bci=132, line=285 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > >> > @bci=838, line=640 (Compiled frame)
> > > >> >  -
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > >> > @bci=17, line=117 (Compiled frame)
> > > >> >  -
> > > >>
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > >> > int) @bci=21, line=285 (Compiled frame)
> > > >> >  -
> > > >>
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > >> > @bci=3, line=271 (Compiled frame)
> > > >> >
> > > >> > It looks as if the Solr-leader threads are waiting on a response
> > from
> > > >> the
> > > >> > terminated replicas, but I would have expected an IOException to
> > > occur.
> > > >> Any
> > > >> > ideas on what may be happening?
> > > >> >
> > > >> >  Thanks,
> > > >> >  Peter
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Regards,
> > > >> Shalin Shekhar Mangar.
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

If a timeout occurs, does the distributed update then go to the next
replica?

On Fri, Dec 12, 2014 at 3:42 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
>
> Sorry I should have specified. These timeouts go inside the <solrcloud>
> section and apply for inter-shard update requests only. The socket and
> connection timeout inside the shardHandlerFactory section apply for
> inter-shard search requests.
>
> On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <pe...@gmail.com>
> wrote:
>
> > Btw, are the following timeouts still supported in solr.xml, and do they
> > only apply to distributed search?
> >
> >   <shardHandlerFactory name="shardHandlerFactory"
> >     class="HttpShardHandlerFactory">
> >     <int name="socketTimeout">${socketTimeout:0}</int>
> >     <int name="connTimeout">${connTimeout:0}</int>
> >   </shardHandlerFactory>
> >
> > Thanks,
> > Peter
> >
> > On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <pe...@gmail.com>
> > wrote:
> >
> > > No, I wasn't aware of these. I will give that a try. If I stop the Solr
> > > jetty service manually, things recover fine, but the hang occurs when I
> > > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> > > 15-sec timeout from the stopped node, and expires the session, but the
> > Solr
> > > leader never gets notified. This seems like a bug in ZK.
> > >
> > > Thanks,
> > > Peter
> > >
> > >
> > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > > shalinmangar@gmail.com> wrote:
> > >
> > >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> > >> reasonable values in your solr.xml? These are the timeouts used for
> > >> inter-shard update requests.
> > >>
> > >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <peterlkeegan@gmail.com
> >
> > >> wrote:
> > >>
> > >> > We are running SolrCloud in AWS and using their auto scaling groups
> to
> > >> spin
> > >> > up new Solr replicas when CPU utilization exceeds a threshold for a
> > >> period
> > >> > of time. All is well until the replicas are terminated when CPU
> > >> utilization
> > >> > falls below another threshold. What happens is that index updates
> sent
> > >> to
> > >> > the Solr leader hang forever in both the Solr leader and the SolrJ
> > >> client
> > >> > app. Searches work fine.  Here are 2 thread stack traces from the
> Solr
> > >> > leader and 2 from the client app:
> > >> >
> > >> > 1) Solr-leader thread doing a distributed commit:
> > >> >
> > >> > Thread 23527: (state = IN_NATIVE)
> > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=61 (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=62, line=300 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=574, line=520 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > >> > (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=101, line=293 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > >> > @bci=127, line=226 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > >> > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> > >> > @bci=112, line=195 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > >> > @bci=174, line=1250 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > >> > @bci=61, line=157 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> > >> > (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > >> (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > >> > org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > >> > @bci=1380, line=417 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > >> line=201
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >> >  -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > >> frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > >> frame)
> > >> >  -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> (Compiled
> > >> > frame)
> > >> >  -
> org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> (Compiled
> > >> > frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > >> frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > >> > @bci=146, line=368 (Compiled frame)
> > >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > >> > @bci=323, line=489 (Compiled frame)
> > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > >> @bci=1,
> > >> > line=53 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > >> > @bci=13, line=953 (Interpreted frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > >> > @bci=5, line=1014 (Interpreted frame)
> > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> > >> > (Compiled frame)
> > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > line=240
> > >> > (Compiled frame)
> > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > >> > line=72 (Compiled frame)
> > >> >  -
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > >> > @bci=100, line=264 (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > >> > @bci=1, line=608 (Interpreted frame)
> > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > >> line=543
> > >> > (Compiled frame)
> > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >> >
> > >> >  2) Solr-leader thread doing a delete by query:
> > >> >
> > >> > Thread 19580: (state = BLOCKED)
> > >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > >> information
> > >> > may be imprecise)
> > >> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > >> @bci=14,
> > >> > line=186 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > >> > @bci=1, line=834 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > >> > int) @bci=67, line=867 (Compiled frame)
> > >> >  -
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > >> > @bci=17, line=1197 (Compiled frame)
> > >> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> > @bci=21,
> > >> > line=214 (Compiled frame)
> > >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > >> > @bci=81, line=391 (Compiled frame)
> > >> >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > >> > @bci=35, line=98 (Compiled frame)
> > >> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8,
> line=62
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=1088, line=1088 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=14, line=825 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > >> > @bci=61, line=121 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.client.solrj.request.UpdateRequest,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> > >> line=150
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> > >> line=111
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse,
> > >> > org.apache.solr.common.util.ContentStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> > >> line=58
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse,
> > >> > org.apache.solr.common.util.ContentStream,
> > >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> > >> line=92
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> > (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> > (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > >> (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > >> > org.apache.solr.request.SolrRequestHandler,
> > >> > org.apache.solr.request.SolrQueryRequest,
> > >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> > (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > >> > @bci=1380, line=417 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > >> line=201
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >> >  -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > >> frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > >> frame)
> > >> >  -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075
> (Compiled
> > >> > frame)
> > >> >  -
> org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009
> (Compiled
> > >> > frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > >> frame)
> > >> >  -
> > >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > >> > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > >> > @bci=146, line=368 (Compiled frame)
> > >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > >> > @bci=323, line=489 (Compiled frame)
> > >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> > >> > (Compiled frame)
> > >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> > line=240
> > >> > (Compiled frame)
> > >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > >> > line=72 (Compiled frame)
> > >> >  -
> > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > >> > @bci=100, line=264 (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > >> > @bci=1, line=608 (Interpreted frame)
> > >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > >> line=543
> > >> > (Compiled frame)
> > >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >> >
> > >> >  3) SolrJ client thread doing commit:
> > >> >
> > >> >  Thread 1872: (state = IN_NATIVE)
> > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=5, line=115 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=62 (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=73, line=301 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=597, line=522 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > >> (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > >> > @bci=132, line=285 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=838, line=640 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > >> > @bci=17, line=117 (Interpreted frame)
> > >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> > >> > @bci=16, line=168 (Interpreted frame)
> > >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> > >> > (Interpreted frame)
> > >> >
> > >> >  4) SolrJ client thread doing delete by query:
> > >> >
> > >> >  Thread 1873: (state = IN_NATIVE)
> > >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > >> byte[],
> > >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > >> line=150
> > >> > (Compiled frame)
> > >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > >> > (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > >> @bci=71,
> > >> > line=166 (Compiled frame)
> > >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > >> line=90
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=137, line=281 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > >> > @bci=5, line=115 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=16, line=92 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > >> > @bci=2, line=62 (Compiled frame)
> > >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > >> line=254
> > >> > (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > >> > @bci=8, line=289 (Compiled frame)
> > >> >  -
> > >> >
> > >>
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > >> > @bci=1, line=252 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > >> > @bci=6, line=191 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=73, line=301 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > >> > org.apache.http.HttpClientConnection,
> > >> org.apache.http.protocol.HttpContext)
> > >> > @bci=60, line=127 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > >> frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=597, line=522 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > >> > @bci=344, line=906 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > >> > @bci=6, line=784 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > >> (Compiled
> > >> > frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=17, line=199 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > >> > @bci=132, line=285 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > >> > @bci=838, line=640 (Compiled frame)
> > >> >  -
> > >> >
> > >> >
> > >>
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > >> > @bci=17, line=117 (Compiled frame)
> > >> >  -
> > >>
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > >> > int) @bci=21, line=285 (Compiled frame)
> > >> >  -
> > >>
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > >> > @bci=3, line=271 (Compiled frame)
> > >> >
> > >> > It looks as if the Solr-leader threads are waiting on a response
> from
> > >> the
> > >> > terminated replicas, but I would have expected an IOException to
> > occur.
> > >> Any
> > >> > ideas on what may be happening?
> > >> >
> > >> >  Thanks,
> > >> >  Peter
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Shalin Shekhar Mangar.
> > >>
> > >
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr hangs on distributed updates

[Shalin Shekhar Mangar <sh...@gmail.com>]

Sorry I should have specified. These timeouts go inside the <solrcloud>
section and apply for inter-shard update requests only. The socket and
connection timeout inside the shardHandlerFactory section apply for
inter-shard search requests.

On Fri, Dec 12, 2014 at 8:38 PM, Peter Keegan <pe...@gmail.com>
wrote:

> Btw, are the following timeouts still supported in solr.xml, and do they
> only apply to distributed search?
>
>   <shardHandlerFactory name="shardHandlerFactory"
>     class="HttpShardHandlerFactory">
>     <int name="socketTimeout">${socketTimeout:0}</int>
>     <int name="connTimeout">${connTimeout:0}</int>
>   </shardHandlerFactory>
>
> Thanks,
> Peter
>
> On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <pe...@gmail.com>
> wrote:
>
> > No, I wasn't aware of these. I will give that a try. If I stop the Solr
> > jetty service manually, things recover fine, but the hang occurs when I
> > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> > 15-sec timeout from the stopped node, and expires the session, but the
> Solr
> > leader never gets notified. This seems like a bug in ZK.
> >
> > Thanks,
> > Peter
> >
> >
> > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> >
> >> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> >> reasonable values in your solr.xml? These are the timeouts used for
> >> inter-shard update requests.
> >>
> >> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
> >> wrote:
> >>
> >> > We are running SolrCloud in AWS and using their auto scaling groups to
> >> spin
> >> > up new Solr replicas when CPU utilization exceeds a threshold for a
> >> period
> >> > of time. All is well until the replicas are terminated when CPU
> >> utilization
> >> > falls below another threshold. What happens is that index updates sent
> >> to
> >> > the Solr leader hang forever in both the Solr leader and the SolrJ
> >> client
> >> > app. Searches work fine.  Here are 2 thread stack traces from the Solr
> >> > leader and 2 from the client app:
> >> >
> >> > 1) Solr-leader thread doing a distributed commit:
> >> >
> >> > Thread 23527: (state = IN_NATIVE)
> >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> >> byte[],
> >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> >> line=150
> >> > (Compiled frame)
> >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> >> > (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> >> @bci=71,
> >> > line=166 (Compiled frame)
> >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> >> line=90
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> >> > @bci=137, line=281 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=16, line=92 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=2, line=61 (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> >> line=254
> >> > (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> >> > @bci=8, line=289 (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> >> > @bci=1, line=252 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> >> > @bci=6, line=191 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=62, line=300 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=60, line=127 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=574, line=520 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=344, line=906 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> >> > @bci=6, line=784 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> >> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> >> > (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=17, line=199 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=101, line=293 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> >> > @bci=127, line=226 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> >> > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> >> > @bci=112, line=195 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> >> > @bci=174, line=1250 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> >> > @bci=61, line=157 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.update.processor.UpdateRequestProcessor,
> >> > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> >> > (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> >> > org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> >> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> >> > org.apache.solr.request.SolrRequestHandler,
> >> > org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> >> > @bci=1380, line=417 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> >> line=201
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> >> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> >> frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> >> frame)
> >> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> >> > frame)
> >> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> >> > frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> >> frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> >> > @bci=146, line=368 (Compiled frame)
> >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> >> > @bci=323, line=489 (Compiled frame)
> >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> >> @bci=1,
> >> > line=53 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> >> > @bci=13, line=953 (Interpreted frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> >> > @bci=5, line=1014 (Interpreted frame)
> >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> >> > (Compiled frame)
> >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> line=240
> >> > (Compiled frame)
> >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> >> > line=72 (Compiled frame)
> >> >  -
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> >> > @bci=100, line=264 (Compiled frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> >> > @bci=1, line=608 (Interpreted frame)
> >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> >> line=543
> >> > (Compiled frame)
> >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> >> >
> >> >  2) Solr-leader thread doing a delete by query:
> >> >
> >> > Thread 19580: (state = BLOCKED)
> >> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> >> information
> >> > may be imprecise)
> >> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> >> @bci=14,
> >> > line=186 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> >> > @bci=1, line=834 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> >> > int) @bci=67, line=867 (Compiled frame)
> >> >  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> >> > @bci=17, line=1197 (Compiled frame)
> >> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> @bci=21,
> >> > line=214 (Compiled frame)
> >> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> >> > @bci=81, line=391 (Compiled frame)
> >> >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> >> > @bci=35, line=98 (Compiled frame)
> >> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> >> > @bci=1088, line=1088 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> >> > @bci=14, line=825 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> >> > @bci=61, line=121 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.client.solrj.request.UpdateRequest,
> >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> >> line=150
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> >> line=111
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse,
> >> > org.apache.solr.common.util.ContentStream,
> >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> >> line=58
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse,
> >> > org.apache.solr.common.util.ContentStream,
> >> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> >> line=92
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> >> > org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> >> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> >> > org.apache.solr.request.SolrRequestHandler,
> >> > org.apache.solr.request.SolrQueryRequest,
> >> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> >> > @bci=1380, line=417 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> >> line=201
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> >> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> >> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> >> frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> >> frame)
> >> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> >> > frame)
> >> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> >> > frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> >> frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> >> frame)
> >> >  -
> >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> >> > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> >> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> >> > @bci=146, line=368 (Compiled frame)
> >> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> >> > @bci=323, line=489 (Compiled frame)
> >> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> >> > (Compiled frame)
> >> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> line=240
> >> > (Compiled frame)
> >> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> >> > line=72 (Compiled frame)
> >> >  -
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> >> > @bci=100, line=264 (Compiled frame)
> >> >  -
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> >> > @bci=1, line=608 (Interpreted frame)
> >> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> >> line=543
> >> > (Compiled frame)
> >> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> >> >
> >> >  3) SolrJ client thread doing commit:
> >> >
> >> >  Thread 1872: (state = IN_NATIVE)
> >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> >> byte[],
> >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> >> line=150
> >> > (Compiled frame)
> >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> >> > (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> >> @bci=71,
> >> > line=166 (Compiled frame)
> >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> >> line=90
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> >> > @bci=137, line=281 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> >> > @bci=5, line=115 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=16, line=92 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=2, line=62 (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> >> line=254
> >> > (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> >> > @bci=8, line=289 (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> >> > @bci=1, line=252 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> >> > @bci=6, line=191 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=73, line=301 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=60, line=127 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=597, line=522 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=344, line=906 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> >> > @bci=6, line=784 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> >> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=17, line=199 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> >> > @bci=132, line=285 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=838, line=640 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> >> > @bci=17, line=117 (Interpreted frame)
> >> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> >> > @bci=16, line=168 (Interpreted frame)
> >> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> >> > (Interpreted frame)
> >> >
> >> >  4) SolrJ client thread doing delete by query:
> >> >
> >> >  Thread 1873: (state = IN_NATIVE)
> >> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> >> byte[],
> >> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> >> line=150
> >> > (Compiled frame)
> >> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> >> > (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> >> @bci=71,
> >> > line=166 (Compiled frame)
> >> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> >> line=90
> >> > (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> >> > @bci=137, line=281 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> >> > @bci=5, line=115 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=16, line=92 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> >> > @bci=2, line=62 (Compiled frame)
> >> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> >> line=254
> >> > (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> >> > @bci=8, line=289 (Compiled frame)
> >> >  -
> >> >
> >>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> >> > @bci=1, line=252 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> >> > @bci=6, line=191 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=73, line=301 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> >> > org.apache.http.HttpClientConnection,
> >> org.apache.http.protocol.HttpContext)
> >> > @bci=60, line=127 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> >> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=597, line=522 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> >> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> >> > @bci=344, line=906 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> >> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> >> > @bci=6, line=784 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> >> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> >> (Compiled
> >> > frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=17, line=199 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> >> > @bci=132, line=285 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> >> > @bci=838, line=640 (Compiled frame)
> >> >  -
> >> >
> >> >
> >>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> >> > @bci=17, line=117 (Compiled frame)
> >> >  -
> >> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> >> > int) @bci=21, line=285 (Compiled frame)
> >> >  -
> >> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> >> > @bci=3, line=271 (Compiled frame)
> >> >
> >> > It looks as if the Solr-leader threads are waiting on a response from
> >> the
> >> > terminated replicas, but I would have expected an IOException to
> occur.
> >> Any
> >> > ideas on what may be happening?
> >> >
> >> >  Thanks,
> >> >  Peter
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
> >
> >
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

Btw, are the following timeouts still supported in solr.xml, and do they
only apply to distributed search?

  <shardHandlerFactory name="shardHandlerFactory"
    class="HttpShardHandlerFactory">
    <int name="socketTimeout">${socketTimeout:0}</int>
    <int name="connTimeout">${connTimeout:0}</int>
  </shardHandlerFactory>

Thanks,
Peter

On Fri, Dec 12, 2014 at 3:14 PM, Peter Keegan <pe...@gmail.com>
wrote:

> No, I wasn't aware of these. I will give that a try. If I stop the Solr
> jetty service manually, things recover fine, but the hang occurs when I
> 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> 15-sec timeout from the stopped node, and expires the session, but the Solr
> leader never gets notified. This seems like a bug in ZK.
>
> Thanks,
> Peter
>
>
> On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
>> reasonable values in your solr.xml? These are the timeouts used for
>> inter-shard update requests.
>>
>> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
>> wrote:
>>
>> > We are running SolrCloud in AWS and using their auto scaling groups to
>> spin
>> > up new Solr replicas when CPU utilization exceeds a threshold for a
>> period
>> > of time. All is well until the replicas are terminated when CPU
>> utilization
>> > falls below another threshold. What happens is that index updates sent
>> to
>> > the Solr leader hang forever in both the Solr leader and the SolrJ
>> client
>> > app. Searches work fine.  Here are 2 thread stack traces from the Solr
>> > leader and 2 from the client app:
>> >
>> > 1) Solr-leader thread doing a distributed commit:
>> >
>> > Thread 23527: (state = IN_NATIVE)
>> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
>> byte[],
>> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
>> line=150
>> > (Compiled frame)
>> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
>> > (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
>> @bci=71,
>> > line=166 (Compiled frame)
>> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
>> line=90
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
>> > @bci=137, line=281 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=16, line=92 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=2, line=61 (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
>> line=254
>> > (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
>> > @bci=8, line=289 (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
>> > @bci=1, line=252 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
>> > @bci=6, line=191 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=62, line=300 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=60, line=127 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
>> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=574, line=520 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=344, line=906 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
>> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
>> > @bci=6, line=784 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
>> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
>> > (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=17, line=199 (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=101, line=293 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
>> > @bci=127, line=226 (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
>> > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
>> > @bci=112, line=195 (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
>> > @bci=174, line=1250 (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
>> > @bci=61, line=157 (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.update.processor.UpdateRequestProcessor,
>> > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
>> > (Interpreted frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
>> > org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
>> (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
>> > org.apache.solr.request.SolrRequestHandler,
>> > org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
>> > @bci=1380, line=417 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
>> line=201
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
>> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
>> frame)
>> >  -
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
>> frame)
>> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
>> > frame)
>> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
>> > frame)
>> >  -
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
>> frame)
>> >  -
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
>> > @bci=146, line=368 (Compiled frame)
>> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
>> > @bci=323, line=489 (Compiled frame)
>> >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
>> @bci=1,
>> > line=53 (Interpreted frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
>> > @bci=13, line=953 (Interpreted frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
>> > @bci=5, line=1014 (Interpreted frame)
>> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
>> > (Compiled frame)
>> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
>> > (Compiled frame)
>> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
>> > line=72 (Compiled frame)
>> >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
>> > @bci=100, line=264 (Compiled frame)
>> >  -
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
>> > @bci=1, line=608 (Interpreted frame)
>> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
>> line=543
>> > (Compiled frame)
>> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
>> >
>> >  2) Solr-leader thread doing a delete by query:
>> >
>> > Thread 19580: (state = BLOCKED)
>> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
>> information
>> > may be imprecise)
>> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
>> @bci=14,
>> > line=186 (Compiled frame)
>> >  -
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
>> > @bci=1, line=834 (Compiled frame)
>> >  -
>> >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
>> > int) @bci=67, line=867 (Compiled frame)
>> >  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
>> > @bci=17, line=1197 (Compiled frame)
>> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21,
>> > line=214 (Compiled frame)
>> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
>> > @bci=81, line=391 (Compiled frame)
>> >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
>> > @bci=35, line=98 (Compiled frame)
>> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
>> > @bci=1088, line=1088 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
>> > @bci=14, line=825 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
>> > @bci=61, line=121 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.client.solrj.request.UpdateRequest,
>> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
>> line=150
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
>> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
>> line=111
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse,
>> > org.apache.solr.common.util.ContentStream,
>> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
>> line=58
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse,
>> > org.apache.solr.common.util.ContentStream,
>> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
>> line=92
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
>> > org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
>> (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
>> > org.apache.solr.request.SolrRequestHandler,
>> > org.apache.solr.request.SolrQueryRequest,
>> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
>> > @bci=1380, line=417 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
>> line=201
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
>> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
>> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
>> frame)
>> >  -
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
>> frame)
>> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
>> > frame)
>> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
>> > frame)
>> >  -
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
>> frame)
>> >  -
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
>> frame)
>> >  -
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
>> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
>> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
>> > @bci=146, line=368 (Compiled frame)
>> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
>> > @bci=323, line=489 (Compiled frame)
>> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
>> > (Compiled frame)
>> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
>> > (Compiled frame)
>> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
>> > line=72 (Compiled frame)
>> >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
>> > @bci=100, line=264 (Compiled frame)
>> >  -
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
>> > @bci=1, line=608 (Interpreted frame)
>> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
>> line=543
>> > (Compiled frame)
>> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
>> >
>> >  3) SolrJ client thread doing commit:
>> >
>> >  Thread 1872: (state = IN_NATIVE)
>> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
>> byte[],
>> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
>> line=150
>> > (Compiled frame)
>> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
>> > (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
>> @bci=71,
>> > line=166 (Compiled frame)
>> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
>> line=90
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
>> > @bci=137, line=281 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
>> > @bci=5, line=115 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=16, line=92 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=2, line=62 (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
>> line=254
>> > (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
>> > @bci=8, line=289 (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
>> > @bci=1, line=252 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
>> > @bci=6, line=191 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=73, line=301 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=60, line=127 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
>> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=597, line=522 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=344, line=906 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
>> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
>> > @bci=6, line=784 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
>> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
>> (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=17, line=199 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
>> > @bci=132, line=285 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=838, line=640 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
>> > @bci=17, line=117 (Interpreted frame)
>> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
>> > @bci=16, line=168 (Interpreted frame)
>> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
>> > (Interpreted frame)
>> >
>> >  4) SolrJ client thread doing delete by query:
>> >
>> >  Thread 1873: (state = IN_NATIVE)
>> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
>> byte[],
>> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
>> line=150
>> > (Compiled frame)
>> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
>> > (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
>> @bci=71,
>> > line=166 (Compiled frame)
>> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
>> line=90
>> > (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
>> > @bci=137, line=281 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
>> > @bci=5, line=115 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=16, line=92 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
>> > @bci=2, line=62 (Compiled frame)
>> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
>> line=254
>> > (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
>> > @bci=8, line=289 (Compiled frame)
>> >  -
>> >
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
>> > @bci=1, line=252 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
>> > @bci=6, line=191 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=73, line=301 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
>> > org.apache.http.HttpClientConnection,
>> org.apache.http.protocol.HttpContext)
>> > @bci=60, line=127 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
>> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
>> frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=597, line=522 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
>> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
>> > @bci=344, line=906 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
>> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
>> > @bci=6, line=784 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
>> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
>> (Compiled
>> > frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=17, line=199 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
>> > @bci=132, line=285 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
>> > @bci=838, line=640 (Compiled frame)
>> >  -
>> >
>> >
>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
>> > @bci=17, line=117 (Compiled frame)
>> >  -
>> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
>> > int) @bci=21, line=285 (Compiled frame)
>> >  -
>> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
>> > @bci=3, line=271 (Compiled frame)
>> >
>> > It looks as if the Solr-leader threads are waiting on a response from
>> the
>> > terminated replicas, but I would have expected an IOException to occur.
>> Any
>> > ideas on what may be happening?
>> >
>> >  Thanks,
>> >  Peter
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

I added distribUpdateConnTimeout and distribUpdateSoTimeout to solr.xml and
the commit did timeout.(btw, is there any way to view solr.xml in the admin
console?).

Also, although we do have an init.d start/stop script for Solr, the 'stop'
command was not executed during shutdown because there was no lock file for
the script in '/var/lock/subsys'. I didn't know about this until I google'd
around and found '
http://www.redhat.com/magazine/008jun05/departments/tips_tricks'. When I
added the lock file, both the AWS 'stop' and 'terminate' actions did result
in an orderly shutdown of the replica which caused the Solr-leader to get
an exception and update the live_nodes, gracefully.

So now, the timeouts should only play a backup role.

Thanks for the help,
Peter


On Fri, Dec 12, 2014 at 5:21 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
>
> : No, I wasn't aware of these. I will give that a try. If I stop the Solr
> : jetty service manually, things recover fine, but the hang occurs when I
> : 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
>
> I don't know squat about AWS Auto-Scaling, (and barely anything about AWS)
> but what you describe makes it sound like maybe your "machine" (ie AMI?)
> isn't really configured very well?
>
> Do you have some init.d/systemd type scripts to ensure a clean shutdown of
> Solr when the machine is shutdown/rebooted?  That seems like a pretty good
> idea in general (in dependent of wether you are using Auto-Scaling) and --
> assuming AWS auto-scaling does clean OS shutdowns when terminating
> instances -- would probably solve your problem.  It would help ensure you
> would never have to wait on the timeouts -- the nodes will each explicitly
> tell ZK they are going bye-bye.
>
> if you do have things setup so that *manually* shutting down your
> instances executes a clean shutdown of solr, but AWS Auto-Scaling is
> actaully totally brutal and doesn't even do a clean shutdown of your
> virtual machines -- just yanks the virtual power cord -- perhaps you could
> implement one of these "LifecycleHook" options that poped up when i did
> some googling for "AWS Auto-Scale termination" to explicitly do a clean
> shutdown of the Solr process" before the machine vanishes into thin air?
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

The AMIs are Red Hat (not Amazon's) and the instances are properly sized
for the environment (t1.micro for ZK, m3.xlarge for Solr). I do plan to add
hooks for a clean shutdown of Solr when the VM is shut down, but if Solr
takes too long, AWS may clobber it anyway. One frustrating part of auto
scaling shutdown is that you can't log into the 'vanishing machine' to view
the logs.

Peter

On Fri, Dec 12, 2014 at 5:21 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
>
> : No, I wasn't aware of these. I will give that a try. If I stop the Solr
> : jetty service manually, things recover fine, but the hang occurs when I
> : 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
>
> I don't know squat about AWS Auto-Scaling, (and barely anything about AWS)
> but what you describe makes it sound like maybe your "machine" (ie AMI?)
> isn't really configured very well?
>
> Do you have some init.d/systemd type scripts to ensure a clean shutdown of
> Solr when the machine is shutdown/rebooted?  That seems like a pretty good
> idea in general (in dependent of wether you are using Auto-Scaling) and --
> assuming AWS auto-scaling does clean OS shutdowns when terminating
> instances -- would probably solve your problem.  It would help ensure you
> would never have to wait on the timeouts -- the nodes will each explicitly
> tell ZK they are going bye-bye.
>
> if you do have things setup so that *manually* shutting down your
> instances executes a clean shutdown of solr, but AWS Auto-Scaling is
> actaully totally brutal and doesn't even do a clean shutdown of your
> virtual machines -- just yanks the virtual power cord -- perhaps you could
> implement one of these "LifecycleHook" options that poped up when i did
> some googling for "AWS Auto-Scale termination" to explicitly do a clean
> shutdown of the Solr process" before the machine vanishes into thin air?
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Solr hangs on distributed updates

[Chris Hostetter <ho...@fucit.org>]

: No, I wasn't aware of these. I will give that a try. If I stop the Solr
: jetty service manually, things recover fine, but the hang occurs when I
: 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a

I don't know squat about AWS Auto-Scaling, (and barely anything about AWS) 
but what you describe makes it sound like maybe your "machine" (ie AMI?) 
isn't really configured very well?

Do you have some init.d/systemd type scripts to ensure a clean shutdown of 
Solr when the machine is shutdown/rebooted?  That seems like a pretty good 
idea in general (in dependent of wether you are using Auto-Scaling) and -- 
assuming AWS auto-scaling does clean OS shutdowns when terminating 
instances -- would probably solve your problem.  It would help ensure you 
would never have to wait on the timeouts -- the nodes will each explicitly 
tell ZK they are going bye-bye.

if you do have things setup so that *manually* shutting down your 
instances executes a clean shutdown of solr, but AWS Auto-Scaling is 
actaully totally brutal and doesn't even do a clean shutdown of your 
virtual machines -- just yanks the virtual power cord -- perhaps you could 
implement one of these "LifecycleHook" options that poped up when i did 
some googling for "AWS Auto-Scale termination" to explicitly do a clean 
shutdown of the Solr process" before the machine vanishes into thin air?



-Hoss
http://www.lucidworks.com/

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

> The Solr leader should stop sending requests to the stopped replica once
> that replica's live node is removed from ZK (after session expiry).

Fwiw, here's the Zookeeper log entry for a graceful shutdown of the Solr
replica:

2014-12-12 15:04:21,304 [myid:2] - INFO  [ProcessThread(sid:2
cport:8181)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x34a1701a1df0037

And here's the Zookeeper log entry for a non-graceful shutdown via EC2 stop
or terminate of the replica:

2014-12-12 14:19:22,000 [myid:2] - INFO  [SessionTracker:ZooKeeperServer@325]
- Expiring session 0x14a1700c19c003f, timeout of 15000ms exceeded
2014-12-12 14:19:22,001 [myid:2] - INFO  [ProcessThread(sid:2
cport:8181)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x14a1700c19c003f

There was no hang in the graceful shutdown.
I'm running ZK version  3.4.5 and Solr 4.6.1

Peter




On Fri, Dec 12, 2014 at 3:39 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
>
> Okay, that should solve the hung threads on the leader.
>
> When you stop the jetty service then it is a graceful shutdown where
> existing requests finish before the searcher thread pool is shutdown
> completely. A EC2 terminate probably just kills the processes and leader
> threads just wait due to a lack of read/connection timeouts.
>
> The Solr leader should stop sending requests to the stopped replica once
> that replica's live node is removed from ZK (after session expiry). I think
> most of these issues are because of the lack of timeouts. Just add them and
> if there are more problems, we can discuss more.
>
> On Fri, Dec 12, 2014 at 8:14 PM, Peter Keegan <pe...@gmail.com>
> wrote:
>
> > No, I wasn't aware of these. I will give that a try. If I stop the Solr
> > jetty service manually, things recover fine, but the hang occurs when I
> > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> > 15-sec timeout from the stopped node, and expires the session, but the
> Solr
> > leader never gets notified. This seems like a bug in ZK.
> >
> > Thanks,
> > Peter
> >
> >
> > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> > shalinmangar@gmail.com> wrote:
> >
> > > Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> > > reasonable values in your solr.xml? These are the timeouts used for
> > > inter-shard update requests.
> > >
> > > On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
> > > wrote:
> > >
> > > > We are running SolrCloud in AWS and using their auto scaling groups
> to
> > > spin
> > > > up new Solr replicas when CPU utilization exceeds a threshold for a
> > > period
> > > > of time. All is well until the replicas are terminated when CPU
> > > utilization
> > > > falls below another threshold. What happens is that index updates
> sent
> > to
> > > > the Solr leader hang forever in both the Solr leader and the SolrJ
> > client
> > > > app. Searches work fine.  Here are 2 thread stack traces from the
> Solr
> > > > leader and 2 from the client app:
> > > >
> > > > 1) Solr-leader thread doing a distributed commit:
> > > >
> > > > Thread 23527: (state = IN_NATIVE)
> > > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > byte[],
> > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > line=150
> > > > (Compiled frame)
> > > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > > > (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > @bci=71,
> > > > line=166 (Compiled frame)
> > > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > line=90
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > @bci=137, line=281 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=16, line=92 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=2, line=61 (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > line=254
> > > > (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > @bci=8, line=289 (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > @bci=1, line=252 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > @bci=6, line=191 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=62, line=300 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=60, line=127 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=574, line=520 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=344, line=906 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > @bci=6, line=784 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > > > (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=17, line=199 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=101, line=293 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > > @bci=127, line=226 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> > > > @bci=112, line=195 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > @bci=174, line=1250 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > > @bci=61, line=157 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> > > > (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > org.apache.solr.request.SolrRequestHandler,
> > > > org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > > @bci=1380, line=417 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > > line=201
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > > >  -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > > frame)
> > > >  -
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > > frame)
> > > >  -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > > > frame)
> > > >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > > frame)
> > > >  -
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > > frame)
> > > >  -
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > > > frame)
> > > >  -
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > > frame)
> > > >  -
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > @bci=146, line=368 (Compiled frame)
> > > >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > @bci=323, line=489 (Compiled frame)
> > > >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > > @bci=1,
> > > > line=53 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > > @bci=13, line=953 (Interpreted frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > > @bci=5, line=1014 (Interpreted frame)
> > > >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> > > > (Compiled frame)
> > > >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> line=240
> > > > (Compiled frame)
> > > >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > > > line=72 (Compiled frame)
> > > >  -
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > @bci=100, line=264 (Compiled frame)
> > > >  -
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > @bci=1, line=608 (Interpreted frame)
> > > >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > > line=543
> > > > (Compiled frame)
> > > >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > >
> > > >  2) Solr-leader thread doing a delete by query:
> > > >
> > > > Thread 19580: (state = BLOCKED)
> > > >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > > information
> > > > may be imprecise)
> > > >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> > @bci=14,
> > > > line=186 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > > @bci=1, line=834 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > > int) @bci=67, line=867 (Compiled frame)
> > > >  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > > @bci=17, line=1197 (Compiled frame)
> > > >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
> @bci=21,
> > > > line=214 (Compiled frame)
> > > >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > > @bci=81, line=391 (Compiled frame)
> > > >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > > @bci=35, line=98 (Compiled frame)
> > > >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > > @bci=1088, line=1088 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > @bci=14, line=825 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > > @bci=61, line=121 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.client.solrj.request.UpdateRequest,
> > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> > > line=150
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> > > line=111
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse,
> > > > org.apache.solr.common.util.ContentStream,
> > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> > line=58
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse,
> > > > org.apache.solr.common.util.ContentStream,
> > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> > > line=92
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > > org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> > (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > > org.apache.solr.request.SolrRequestHandler,
> > > > org.apache.solr.request.SolrQueryRequest,
> > > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
> (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > > @bci=1380, line=417 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > > line=201
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > > >  -
> org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > > frame)
> > > >  -
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > > frame)
> > > >  -
> org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > > > frame)
> > > >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > > frame)
> > > >  -
> > > >
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > > frame)
> > > >  -
> > > >
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > > > frame)
> > > >  -
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > > frame)
> > > >  -
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > > frame)
> > > >  -
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > > org.eclipse.jetty.server.Request,
> > javax.servlet.http.HttpServletRequest,
> > > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > > @bci=146, line=368 (Compiled frame)
> > > >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > > @bci=323, line=489 (Compiled frame)
> > > >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> > > > (Compiled frame)
> > > >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
> line=240
> > > > (Compiled frame)
> > > >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > > > line=72 (Compiled frame)
> > > >  -
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > > @bci=100, line=264 (Compiled frame)
> > > >  -
> > > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > > @bci=1, line=608 (Interpreted frame)
> > > >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > > line=543
> > > > (Compiled frame)
> > > >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > > >
> > > >  3) SolrJ client thread doing commit:
> > > >
> > > >  Thread 1872: (state = IN_NATIVE)
> > > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > byte[],
> > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > line=150
> > > > (Compiled frame)
> > > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > > > (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > @bci=71,
> > > > line=166 (Compiled frame)
> > > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > line=90
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > @bci=137, line=281 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > @bci=5, line=115 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=16, line=92 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=2, line=62 (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > line=254
> > > > (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > @bci=8, line=289 (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > @bci=1, line=252 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > @bci=6, line=191 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=73, line=301 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=60, line=127 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=597, line=522 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=344, line=906 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > @bci=6, line=784 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=17, line=199 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > @bci=132, line=285 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=838, line=640 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > @bci=17, line=117 (Interpreted frame)
> > > >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> > > > @bci=16, line=168 (Interpreted frame)
> > > >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> > > > (Interpreted frame)
> > > >
> > > >  4) SolrJ client thread doing delete by query:
> > > >
> > > >  Thread 1873: (state = IN_NATIVE)
> > > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> > byte[],
> > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > > line=150
> > > > (Compiled frame)
> > > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11,
> line=121
> > > > (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > > @bci=71,
> > > > line=166 (Compiled frame)
> > > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> > line=90
> > > > (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > @bci=137, line=281 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > > @bci=5, line=115 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=16, line=92 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > > @bci=2, line=62 (Compiled frame)
> > > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > > line=254
> > > > (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > > @bci=8, line=289 (Compiled frame)
> > > >  -
> > > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > > @bci=1, line=252 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > > @bci=6, line=191 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=73, line=301 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > > org.apache.http.HttpClientConnection,
> > > org.apache.http.protocol.HttpContext)
> > > > @bci=60, line=127 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=597, line=522 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > > @bci=344, line=906 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > > @bci=6, line=784 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > > (Compiled
> > > > frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=17, line=199 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > > @bci=132, line=285 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > > @bci=838, line=640 (Compiled frame)
> > > >  -
> > > >
> > > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > > @bci=17, line=117 (Compiled frame)
> > > >  -
> > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > > int) @bci=21, line=285 (Compiled frame)
> > > >  -
> > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > > @bci=3, line=271 (Compiled frame)
> > > >
> > > > It looks as if the Solr-leader threads are waiting on a response from
> > the
> > > > terminated replicas, but I would have expected an IOException to
> occur.
> > > Any
> > > > ideas on what may be happening?
> > > >
> > > >  Thanks,
> > > >  Peter
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shalin Shekhar Mangar.
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr hangs on distributed updates

[Shalin Shekhar Mangar <sh...@gmail.com>]

Okay, that should solve the hung threads on the leader.

When you stop the jetty service then it is a graceful shutdown where
existing requests finish before the searcher thread pool is shutdown
completely. A EC2 terminate probably just kills the processes and leader
threads just wait due to a lack of read/connection timeouts.

The Solr leader should stop sending requests to the stopped replica once
that replica's live node is removed from ZK (after session expiry). I think
most of these issues are because of the lack of timeouts. Just add them and
if there are more problems, we can discuss more.

On Fri, Dec 12, 2014 at 8:14 PM, Peter Keegan <pe...@gmail.com>
wrote:

> No, I wasn't aware of these. I will give that a try. If I stop the Solr
> jetty service manually, things recover fine, but the hang occurs when I
> 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
> 15-sec timeout from the stopped node, and expires the session, but the Solr
> leader never gets notified. This seems like a bug in ZK.
>
> Thanks,
> Peter
>
>
> On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
> > Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> > reasonable values in your solr.xml? These are the timeouts used for
> > inter-shard update requests.
> >
> > On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
> > wrote:
> >
> > > We are running SolrCloud in AWS and using their auto scaling groups to
> > spin
> > > up new Solr replicas when CPU utilization exceeds a threshold for a
> > period
> > > of time. All is well until the replicas are terminated when CPU
> > utilization
> > > falls below another threshold. What happens is that index updates sent
> to
> > > the Solr leader hang forever in both the Solr leader and the SolrJ
> client
> > > app. Searches work fine.  Here are 2 thread stack traces from the Solr
> > > leader and 2 from the client app:
> > >
> > > 1) Solr-leader thread doing a distributed commit:
> > >
> > > Thread 23527: (state = IN_NATIVE)
> > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> byte[],
> > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > line=150
> > > (Compiled frame)
> > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > > (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > @bci=71,
> > > line=166 (Compiled frame)
> > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> line=90
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > @bci=137, line=281 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=16, line=92 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=2, line=61 (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > line=254
> > > (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > @bci=8, line=289 (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > @bci=1, line=252 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > @bci=6, line=191 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=62, line=300 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=60, line=127 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=574, line=520 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=344, line=906 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > @bci=6, line=784 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > > (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=17, line=199 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=101, line=293 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > > @bci=127, line=226 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> > > @bci=112, line=195 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > @bci=174, line=1250 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > > @bci=61, line=157 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.update.processor.UpdateRequestProcessor,
> > > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> > > (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > org.apache.solr.request.SolrRequestHandler,
> > > org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > @bci=1380, line=417 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > line=201
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > frame)
> > >  -
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > frame)
> > >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > > frame)
> > >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > frame)
> > >  -
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > frame)
> > >  -
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > > frame)
> > >  -
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > frame)
> > >  -
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > @bci=146, line=368 (Compiled frame)
> > >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > @bci=323, line=489 (Compiled frame)
> > >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> > @bci=1,
> > > line=53 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > > @bci=13, line=953 (Interpreted frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > > @bci=5, line=1014 (Interpreted frame)
> > >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> > > (Compiled frame)
> > >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> > > (Compiled frame)
> > >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > > line=72 (Compiled frame)
> > >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > @bci=100, line=264 (Compiled frame)
> > >  -
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > @bci=1, line=608 (Interpreted frame)
> > >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > line=543
> > > (Compiled frame)
> > >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >
> > >  2) Solr-leader thread doing a delete by query:
> > >
> > > Thread 19580: (state = BLOCKED)
> > >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> > information
> > > may be imprecise)
> > >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object)
> @bci=14,
> > > line=186 (Compiled frame)
> > >  -
> > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > > @bci=1, line=834 (Compiled frame)
> > >  -
> > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > > int) @bci=67, line=867 (Compiled frame)
> > >  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > > @bci=17, line=1197 (Compiled frame)
> > >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21,
> > > line=214 (Compiled frame)
> > >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > > @bci=81, line=391 (Compiled frame)
> > >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > > @bci=35, line=98 (Compiled frame)
> > >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > > @bci=1088, line=1088 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > @bci=14, line=825 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > > @bci=61, line=121 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.client.solrj.request.UpdateRequest,
> > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> > line=150
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> > line=111
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse,
> > > org.apache.solr.common.util.ContentStream,
> > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
> line=58
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse,
> > > org.apache.solr.common.util.ContentStream,
> > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> > line=92
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > > org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
> (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > > org.apache.solr.request.SolrRequestHandler,
> > > org.apache.solr.request.SolrQueryRequest,
> > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > > @bci=1380, line=417 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> > line=201
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> > >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> > frame)
> > >  -
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> > frame)
> > >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > > frame)
> > >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> > frame)
> > >  -
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> > frame)
> > >  -
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > > frame)
> > >  -
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> > frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> > frame)
> > >  -
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> > frame)
> > >  -
> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > > org.eclipse.jetty.server.Request,
> javax.servlet.http.HttpServletRequest,
> > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> > frame)
> > >  -
> > >
> > >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > > @bci=146, line=368 (Compiled frame)
> > >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > > @bci=323, line=489 (Compiled frame)
> > >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> > > (Compiled frame)
> > >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> > > (Compiled frame)
> > >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > > line=72 (Compiled frame)
> > >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > > @bci=100, line=264 (Compiled frame)
> > >  -
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > > @bci=1, line=608 (Interpreted frame)
> > >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> > line=543
> > > (Compiled frame)
> > >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> > >
> > >  3) SolrJ client thread doing commit:
> > >
> > >  Thread 1872: (state = IN_NATIVE)
> > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> byte[],
> > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > line=150
> > > (Compiled frame)
> > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > > (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > @bci=71,
> > > line=166 (Compiled frame)
> > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> line=90
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > @bci=137, line=281 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > @bci=5, line=115 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=16, line=92 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=2, line=62 (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > line=254
> > > (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > @bci=8, line=289 (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > @bci=1, line=252 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > @bci=6, line=191 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=73, line=301 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=60, line=127 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=597, line=522 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=344, line=906 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > @bci=6, line=784 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=17, line=199 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > @bci=132, line=285 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=838, line=640 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > @bci=17, line=117 (Interpreted frame)
> > >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> > > @bci=16, line=168 (Interpreted frame)
> > >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> > > (Interpreted frame)
> > >
> > >  4) SolrJ client thread doing delete by query:
> > >
> > >  Thread 1873: (state = IN_NATIVE)
> > >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
> byte[],
> > > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> > >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> > line=150
> > > (Compiled frame)
> > >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > > (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> > @bci=71,
> > > line=166 (Compiled frame)
> > >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
> line=90
> > > (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > @bci=137, line=281 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > > @bci=5, line=115 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=16, line=92 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > > @bci=2, line=62 (Compiled frame)
> > >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> > line=254
> > > (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > > @bci=8, line=289 (Compiled frame)
> > >  -
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > > @bci=1, line=252 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > > @bci=6, line=191 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=73, line=301 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > > org.apache.http.HttpClientConnection,
> > org.apache.http.protocol.HttpContext)
> > > @bci=60, line=127 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=597, line=522 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > > @bci=344, line=906 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
> frame)
> > >  -
> > >
> > >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > > @bci=6, line=784 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> > (Compiled
> > > frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=17, line=199 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > > @bci=132, line=285 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > > @bci=838, line=640 (Compiled frame)
> > >  -
> > >
> > >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > > @bci=17, line=117 (Compiled frame)
> > >  -
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > > int) @bci=21, line=285 (Compiled frame)
> > >  -
> > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > > @bci=3, line=271 (Compiled frame)
> > >
> > > It looks as if the Solr-leader threads are waiting on a response from
> the
> > > terminated replicas, but I would have expected an IOException to occur.
> > Any
> > > ideas on what may be happening?
> > >
> > >  Thanks,
> > >  Peter
> > >
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr hangs on distributed updates

[Peter Keegan <pe...@gmail.com>]

No, I wasn't aware of these. I will give that a try. If I stop the Solr
jetty service manually, things recover fine, but the hang occurs when I
'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
15-sec timeout from the stopped node, and expires the session, but the Solr
leader never gets notified. This seems like a bug in ZK.

Thanks,
Peter


On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:

> Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
> reasonable values in your solr.xml? These are the timeouts used for
> inter-shard update requests.
>
> On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
> wrote:
>
> > We are running SolrCloud in AWS and using their auto scaling groups to
> spin
> > up new Solr replicas when CPU utilization exceeds a threshold for a
> period
> > of time. All is well until the replicas are terminated when CPU
> utilization
> > falls below another threshold. What happens is that index updates sent to
> > the Solr leader hang forever in both the Solr leader and the SolrJ client
> > app. Searches work fine.  Here are 2 thread stack traces from the Solr
> > leader and 2 from the client app:
> >
> > 1) Solr-leader thread doing a distributed commit:
> >
> > Thread 23527: (state = IN_NATIVE)
> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150
> > (Compiled frame)
> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > (Compiled frame)
> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> @bci=71,
> > line=166 (Compiled frame)
> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> > (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > @bci=137, line=281 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=16, line=92 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=2, line=61 (Compiled frame)
> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> line=254
> > (Compiled frame)
> >  -
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > @bci=8, line=289 (Compiled frame)
> >  -
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > @bci=1, line=252 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > @bci=6, line=191 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=62, line=300 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=60, line=127 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=574, line=520 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=344, line=906 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > @bci=6, line=784 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> > (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=17, line=199 (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=101, line=293 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> > @bci=127, line=226 (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> > java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> > @bci=112, line=195 (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > @bci=174, line=1250 (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> > @bci=61, line=157 (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.update.processor.UpdateRequestProcessor,
> > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> > (Interpreted frame)
> >  -
> >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > org.apache.solr.request.SolrRequestHandler,
> > org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > @bci=1380, line=417 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> line=201
> > (Compiled frame)
> >  -
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> frame)
> >  -
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> frame)
> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > frame)
> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> frame)
> >  -
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> frame)
> >  -
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > frame)
> >  -
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> frame)
> >  -
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> frame)
> >  -
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> frame)
> >  -
> >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > @bci=146, line=368 (Compiled frame)
> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > @bci=323, line=489 (Compiled frame)
> >  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest()
> @bci=1,
> > line=53 (Interpreted frame)
> >  -
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> > @bci=13, line=953 (Interpreted frame)
> >  -
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> > @bci=5, line=1014 (Interpreted frame)
> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> > (Compiled frame)
> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> > (Compiled frame)
> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > line=72 (Compiled frame)
> >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > @bci=100, line=264 (Compiled frame)
> >  -
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > @bci=1, line=608 (Interpreted frame)
> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> line=543
> > (Compiled frame)
> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> >
> >  2) Solr-leader thread doing a delete by query:
> >
> > Thread 19580: (state = BLOCKED)
> >  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> information
> > may be imprecise)
> >  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> > line=186 (Compiled frame)
> >  -
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> > @bci=1, line=834 (Compiled frame)
> >  -
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> > int) @bci=67, line=867 (Compiled frame)
> >  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> > @bci=17, line=1197 (Compiled frame)
> >  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21,
> > line=214 (Compiled frame)
> >  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> > @bci=81, line=391 (Compiled frame)
> >  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> > @bci=35, line=98 (Compiled frame)
> >  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> > @bci=1088, line=1088 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > @bci=14, line=825 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> > @bci=61, line=121 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.client.solrj.request.UpdateRequest,
> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208,
> line=150
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68,
> line=111
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse,
> > org.apache.solr.common.util.ContentStream,
> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18, line=58
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse,
> > org.apache.solr.common.util.ContentStream,
> > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165,
> line=92
> > (Compiled frame)
> >  -
> >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> > org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> > org.apache.solr.request.SolrRequestHandler,
> > org.apache.solr.request.SolrQueryRequest,
> > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> > @bci=1380, line=417 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5,
> line=201
> > (Compiled frame)
> >  -
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
> >  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled
> frame)
> >  -
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled
> frame)
> >  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> > frame)
> >  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled
> frame)
> >  -
> > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled
> frame)
> >  -
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> > frame)
> >  -
> org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled
> frame)
> >  -
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled
> frame)
> >  -
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled
> frame)
> >  -
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> > org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled
> frame)
> >  -
> >
> >
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> > @bci=146, line=368 (Compiled frame)
> >  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> > @bci=323, line=489 (Compiled frame)
> >  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> > (Compiled frame)
> >  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> > (Compiled frame)
> >  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> > line=72 (Compiled frame)
> >  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> > @bci=100, line=264 (Compiled frame)
> >  -
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> > @bci=1, line=608 (Interpreted frame)
> >  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47,
> line=543
> > (Compiled frame)
> >  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
> >
> >  3) SolrJ client thread doing commit:
> >
> >  Thread 1872: (state = IN_NATIVE)
> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150
> > (Compiled frame)
> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > (Compiled frame)
> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> @bci=71,
> > line=166 (Compiled frame)
> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> > (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > @bci=137, line=281 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > @bci=5, line=115 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=16, line=92 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=2, line=62 (Compiled frame)
> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> line=254
> > (Compiled frame)
> >  -
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > @bci=8, line=289 (Compiled frame)
> >  -
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > @bci=1, line=252 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > @bci=6, line=191 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=73, line=301 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=60, line=127 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=597, line=522 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=344, line=906 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > @bci=6, line=784 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=17, line=199 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > @bci=132, line=285 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=838, line=640 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > @bci=17, line=117 (Interpreted frame)
> >  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> > @bci=16, line=168 (Interpreted frame)
> >  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> > (Interpreted frame)
> >
> >  4) SolrJ client thread doing delete by query:
> >
> >  Thread 1873: (state = IN_NATIVE)
> >  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> > int, int, int) @bci=0 (Compiled frame; information may be imprecise)
> >  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150
> > (Compiled frame)
> >  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> > (Compiled frame)
> >  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer()
> @bci=71,
> > line=166 (Compiled frame)
> >  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> > (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > @bci=137, line=281 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> > @bci=5, line=115 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=16, line=92 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> > @bci=2, line=62 (Compiled frame)
> >  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38,
> line=254
> > (Compiled frame)
> >  -
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> > @bci=8, line=289 (Compiled frame)
> >  -
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> > @bci=1, line=252 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> > @bci=6, line=191 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=73, line=301 (Compiled frame)
> >  -
> >
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> > org.apache.http.HttpClientConnection,
> org.apache.http.protocol.HttpContext)
> > @bci=60, line=127 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=597, line=522 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> > @bci=344, line=906 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
> >  -
> >
> >
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> > @bci=6, line=784 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396
> (Compiled
> > frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=17, line=199 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> > @bci=132, line=285 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> > @bci=838, line=640 (Compiled frame)
> >  -
> >
> >
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> > @bci=17, line=117 (Compiled frame)
> >  -
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> > int) @bci=21, line=285 (Compiled frame)
> >  -
> org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> > @bci=3, line=271 (Compiled frame)
> >
> > It looks as if the Solr-leader threads are waiting on a response from the
> > terminated replicas, but I would have expected an IOException to occur.
> Any
> > ideas on what may be happening?
> >
> >  Thanks,
> >  Peter
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr hangs on distributed updates

[Shalin Shekhar Mangar <sh...@gmail.com>]

Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
reasonable values in your solr.xml? These are the timeouts used for
inter-shard update requests.

On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <pe...@gmail.com>
wrote:

> We are running SolrCloud in AWS and using their auto scaling groups to spin
> up new Solr replicas when CPU utilization exceeds a threshold for a period
> of time. All is well until the replicas are terminated when CPU utilization
> falls below another threshold. What happens is that index updates sent to
> the Solr leader hang forever in both the Solr leader and the SolrJ client
> app. Searches work fine.  Here are 2 thread stack traces from the Solr
> leader and 2 from the client app:
>
> 1) Solr-leader thread doing a distributed commit:
>
> Thread 23527: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
> (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
> line=166 (Compiled frame)
>  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> (Compiled frame)
>  -
>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> @bci=137, line=281 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=16, line=92 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=2, line=61 (Compiled frame)
>  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
> (Compiled frame)
>  -
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> @bci=8, line=289 (Compiled frame)
>  -
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> @bci=1, line=252 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> @bci=6, line=191 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=62, line=300 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=60, line=127 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=574, line=520 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=344, line=906 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> @bci=6, line=784 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
> (Interpreted frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=17, line=199 (Interpreted frame)
>  -
>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=101, line=293 (Compiled frame)
>  -
>
> org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
> @bci=127, line=226 (Interpreted frame)
>  -
>
> org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
> java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
> @bci=112, line=195 (Interpreted frame)
>  -
>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> @bci=174, line=1250 (Interpreted frame)
>  -
>
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
> @bci=61, line=157 (Interpreted frame)
>  -
>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.update.processor.UpdateRequestProcessor,
> org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
> (Interpreted frame)
>  -
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled
> frame)
>  -
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> frame)
>  -
>
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
> frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> org.apache.solr.request.SolrRequestHandler,
> org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> @bci=1380, line=417 (Compiled frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
> (Compiled frame)
>  -
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
>  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
>  - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
>  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
>  -
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> frame)
>  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
>  -
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> frame)
>  - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
>  -
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
>  - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
>  -
>
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> @bci=146, line=368 (Compiled frame)
>  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> @bci=323, line=489 (Compiled frame)
>  - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() @bci=1,
> line=53 (Interpreted frame)
>  -
>
> org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
> @bci=13, line=953 (Interpreted frame)
>  -
>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
> @bci=5, line=1014 (Interpreted frame)
>  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
> (Compiled frame)
>  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> (Compiled frame)
>  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> line=72 (Compiled frame)
>  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> @bci=100, line=264 (Compiled frame)
>  -
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> @bci=1, line=608 (Interpreted frame)
>  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
> (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
>
>  2) Solr-leader thread doing a delete by query:
>
> Thread 19580: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=186 (Compiled frame)
>  -
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
> @bci=1, line=834 (Compiled frame)
>  -
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
> int) @bci=67, line=867 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
> @bci=17, line=1197 (Compiled frame)
>  - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21,
> line=214 (Compiled frame)
>  - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
> (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
> @bci=81, line=391 (Compiled frame)
>  - org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
> @bci=35, line=98 (Compiled frame)
>  - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
> (Compiled frame)
>  -
>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
> @bci=1088, line=1088 (Compiled frame)
>  -
>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> @bci=14, line=825 (Compiled frame)
>  -
>
> org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
> @bci=61, line=121 (Compiled frame)
>  -
>
> org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.client.solrj.request.UpdateRequest,
> org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208, line=150
> (Compiled frame)
>  -
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
> org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68, line=111
> (Compiled frame)
>  -
>
> org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse,
> org.apache.solr.common.util.ContentStream,
> org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18, line=58
> (Compiled frame)
>  -
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse,
> org.apache.solr.common.util.ContentStream,
> org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165, line=92
> (Compiled frame)
>  -
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled
> frame)
>  -
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled
> frame)
>  -
>
> org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
> org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled
> frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
> org.apache.solr.request.SolrRequestHandler,
> org.apache.solr.request.SolrQueryRequest,
> org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled
> frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
> @bci=1380, line=417 (Compiled frame)
>  -
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
> (Compiled frame)
>  -
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
> javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
>  - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
>  - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
>  - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
>  -
> org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
> frame)
>  - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
>  -
> org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
> frame)
>  - org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
>  -
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
>  -
> org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
>  - org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
> org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
> javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
>  -
>
> org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
> @bci=146, line=368 (Compiled frame)
>  - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
> @bci=323, line=489 (Compiled frame)
>  - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
> (Compiled frame)
>  - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
> (Compiled frame)
>  - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
> line=72 (Compiled frame)
>  - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
> @bci=100, line=264 (Compiled frame)
>  -
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
> @bci=1, line=608 (Interpreted frame)
>  - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
> (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
>
>  3) SolrJ client thread doing commit:
>
>  Thread 1872: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
> (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
> line=166 (Compiled frame)
>  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> (Compiled frame)
>  -
>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> @bci=137, line=281 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> @bci=5, line=115 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=16, line=92 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=2, line=62 (Compiled frame)
>  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
> (Compiled frame)
>  -
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> @bci=8, line=289 (Compiled frame)
>  -
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> @bci=1, line=252 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> @bci=6, line=191 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=73, line=301 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=60, line=127 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=597, line=522 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=344, line=906 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> @bci=6, line=784 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
> frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=17, line=199 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> @bci=132, line=285 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=838, line=640 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> @bci=17, line=117 (Interpreted frame)
>  - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
> @bci=16, line=168 (Interpreted frame)
>  - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
> (Interpreted frame)
>
>  4) SolrJ client thread doing delete by query:
>
>  Thread 1873: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
> (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
> line=166 (Compiled frame)
>  - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, line=90
> (Compiled frame)
>  -
>
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> @bci=137, line=281 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
> @bci=5, line=115 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=16, line=92 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
> @bci=2, line=62 (Compiled frame)
>  - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
> (Compiled frame)
>  -
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
> @bci=8, line=289 (Compiled frame)
>  -
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
> @bci=1, line=252 (Compiled frame)
>  -
>
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
> @bci=6, line=191 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=73, line=301 (Compiled frame)
>  -
>
> org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
> org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext)
> @bci=60, line=127 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
> org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled frame)
>  -
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=597, line=522 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
> org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
> @bci=344, line=906 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
> org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled frame)
>  -
>
> org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
> @bci=6, line=784 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
> org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
> frame)
>  -
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=17, line=199 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
> @bci=132, line=285 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
> @bci=838, line=640 (Compiled frame)
>  -
>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
> @bci=17, line=117 (Compiled frame)
>  - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
> int) @bci=21, line=285 (Compiled frame)
>  - org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
> @bci=3, line=271 (Compiled frame)
>
> It looks as if the Solr-leader threads are waiting on a response from the
> terminated replicas, but I would have expected an IOException to occur. Any
> ideas on what may be happening?
>
>  Thanks,
>  Peter
>



-- 
Regards,
Shalin Shekhar Mangar.