You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Ronald Braun (JIRA)" <ji...@apache.org> on 2016/05/19 23:14:12 UTC
[jira] [Created] (SOLR-9135) SystemInfoHandler can poison / consume
Jetty thread pool
Ronald Braun created SOLR-9135:
----------------------------------
Summary: SystemInfoHandler can poison / consume Jetty thread pool
Key: SOLR-9135
URL: https://issues.apache.org/jira/browse/SOLR-9135
Project: Solr
Issue Type: Bug
Environment: Solr 6.0.0
Reporter: Ronald Braun
Priority: Minor
We are running solr 6.0.0 in solr cloud mode within a docker container. We encountered an issue whereby the SystemInfoHandler was forking out processes that would immediately enter D (uninterruputable sleep) due to a container volume issue after hitting the admin manager in a browser. The thread stays in runnable state:
{noformat}
"qtp43368234-13611" #13611 prio=5 os_prio=0 tid=0x00007f0260011800 nid=0x36fb ru
nnable [0x00007efa0bce1000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:450)
at java.lang.Runtime.exec(Runtime.java:347)
at org.apache.solr.handler.admin.SystemInfoHandler.execute(SystemInfoHan
dler.java:244)
at org.apache.solr.handler.admin.SystemInfoHandler.getSystemInfo(SystemI
nfoHandler.java:198)
at org.apache.solr.handler.admin.SystemInfoHandler.handleRequestBody(Sys
temInfoHandler.java:111)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl
erBase.java:155)
at org.apache.solr.handler.admin.InfoHandler.handleRequestBody(InfoHandl
er.java:86)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl
erBase.java:155)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.
java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte
r.java:229)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte
r.java:184)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java
:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav
a:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl
er.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
er.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle
r.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle
r.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont
extHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl
ection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav
a:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstra
ctConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoin
t.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceA
ndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Exec
uteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo
l.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool
.java:572)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The problematic command being executed was 'uname -a'. The admin manager would throw up a "Lost connection to solr" message but presumably retries the connection periodically (at least a couple of times a minute). Before we figured out what was going on, we had 600+ threads in D state:
{noformat}
4433 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4434 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4439 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4440 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.04 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4461 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4462 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4467 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4470 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4486 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4487 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.06 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4488 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4489 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4496 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4497 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
4501 solr 20 0 0.399t 0.105t 0.057t D 0.0 86.2 0:00.00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms48g -Xmx48g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThre+
etc.
{noformat}
An OS exec call is a bit heavy for loading the admin page... Might you consider either:
- load this info once at startup and store
- use a collapsed panel for display and fetch only on expansion / request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org