You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ravi Solr <ra...@gmail.com> on 2013/05/01 23:14:07 UTC
Server inconsistent state & Core Reload issue
We are using Solr 3.6.2 with a single core setup on a glassfish server,
every 4-5 hours the server gradually gets into a some kind of a
inconsistent state and stops accepting any queries giving back cached
results. Even the core reload fails giving the following. Has anybody
experienced such behavior ? Can anybody help me understand why this might
happen ?
http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">9</int>
</lst>
<lst name="status">
<lst name="core1">
<str name="name">core1</str>
<str name="instanceDir">/data/solr/core1-home/</str>
<str name="dataDir">/data/solr/core/core1-data/</str>
<date name="startTime">2013-05-01T19:16:31.32Z</date>
<long name="uptime">137850</long>
<lst name="index">
<int name="numDocs">21479</int>
<int name="maxDoc">25170</int>
<long name="version">1367184551418</long>
<int name="segmentCount">4</int>
<bool name="current">true</bool>
<bool name="hasDeletions">true</bool>
<str
name="directory">org.apache.lucene.store.MMapDirectory:org.apache.lucene.store.MMapDirectory@/data/solr/core/core1-data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@71d9673</str>
<date name="lastModified">2013-05-01T19:15:04Z</date>
</lst>
</lst>
</lst>
</response>
During the inconsistent state any queries being issued to the server loose
the query parameters. We can see the proper queries in the container's http
access logs but solr somehow solr doesn't get the query params at all. Also
note that "content length" on the container's access logs is always 68935,
which implies its always giving the same docs irrespective of the query.
If we restart the server everything is back to normal and the same queries
run properly.
SOLR Log
--------------
[#|2013-05-01T15:20:02.031-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=17 |#]
[#|2013-05-01T15:20:02.034-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#]
[#|2013-05-01T15:20:02.055-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#]
[#|2013-05-01T15:20:02.081-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]
[#|2013-05-01T15:20:02.106-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]
[#|2013-05-01T15:20:02.136-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#]
[#|2013-05-01T15:20:02.161-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]
[#|2013-05-01T15:20:02.185-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]
[#|2013-05-01T15:20:02.209-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]
[#|2013-05-01T15:20:02.241-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#]
[#|2013-05-01T15:20:02.266-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]
[#|2013-05-01T15:20:02.288-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]
[#|2013-05-01T15:20:02.291-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]
Container Access Logs
---------------------------------
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Music+Venues%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=5&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fq=types%3A%28%22Painting%2FDrawing%22%29&sort=closingdate+asc%2C+openingdate+desc%2C+namestring+asc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Theaters%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Events%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Locations%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Restaurants%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935
Thanks,
Ravi
Re: Server inconsistent state & Core Reload issue
Posted by Chris Hostetter <ho...@fucit.org>.
: If you look at my email the container that is running SOLR got the request
: params (http access logs provided in first email) but when it goes through
: the SOLR app/code on the container (probably through request filters or
: dispatchers..I don't know exactly) its getting lost, which is what I am
: trying to understand. I want to understand under what situations this mat
: happen.
i can't think of any situation where anything in solr would do this ... i
do however note that in the stacktraces you provided from your threaddump,
there is a *lot* more forwarding and dispatching going on then i have ever
seen in a typical jetty or tomcat type setup -- aprently because of how
you have glassfish running
you've got the usual stuff prior to the SolrDispatchFilter
(org.apache.catalina.*) but you've also got things i have never seen
before...
com.sun.enterprise.web.WebPipeline
com.sun.enterprise.web.PESessionLockingStandardPipeline
com.sun.enterprise.web.connector.grizzly.*
...i would bet money that whatever is fausing problems with your request
paramaters is happening somewhere in there.
It would be trivial to test: just add some logging of the full request
params in SolrDispatchFilter and see what it's getting from our servlet
container when the doFilter method is called.
FWIW, some sporadic googling has lead to a handful of indications that
you are not alone in encountering problems with glassfish sporadically
swallowing request params (or POST bodies) so that applications running
in glassfish don't get them...
http://jira.icesoft.org/browse/ICE-5298
http://www.coderanch.com/t/601685/glassfish/Body-Request-occasionally-lost
-Hoss
Re: Server inconsistent state & Core Reload issue
Posted by Ravi Solr <ra...@gmail.com>.
I replayed the logs (roughly 255000 queries) with 100 threads to SOLR 4.3.0
and it did not crash!!! However when I replay the same logs to SOLR3.6.2 it
failed within a minute. Surely there is something in SOLR 3.6.2 which
causes the server to seize up.
Thanks,
Ravi Kiran Bhaskar
On Fri, May 3, 2013 at 9:47 AM, Ravi Solr <ra...@gmail.com> wrote:
> Hello,
> I managed to get the thread dump from solr admin console during the
> time it was failing. I must admit I am not that great with reading thread
> dumps, can somebody help me figure out anything weird in this thread dump
>
> ava HotSpot(TM) 64-Bit Server VM20.4-b02
> Thread Count: current=109, peak=130, daemon=85
> 'RMI TCP Connection(328)-10.112.36.135' Id=960, RUNNABLE on lock=, total
> cpu time=0.0000ms user time=0.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.FilterInputStream.read(FilterInputStream.java:66)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 950' Id=950, TIMED_WAITING on
> lock=[I@652381db, total cpu time=320.0000ms user time=230.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Connection(313)-10.8.7.69' Id=949, RUNNABLE on lock=, total cpu
> time=8530.0000ms user time=7980.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.FilterInputStream.read(FilterInputStream.java:66)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Connection(327)-10.112.36.135' Id=948, RUNNABLE on lock=, total
> cpu time=60.0000ms user time=60.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at java.io.FilterInputStream.read(FilterInputStream.java:66)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Thread-67' Id=942, TIMED_WAITING on
> lock=com.sun.appserv.management.util.jmx.NotificationEmitterSupport$SenderThread@7e2284bc,
> total cpu time=210.0000ms user time=150.0000ms
> at java.lang.Object.wait(Native Method)
>
> at
> com.sun.appserv.management.util.jmx.NotificationEmitterSupport$SenderThread.run(NotificationEmitterSupport.java:414)
>
> 'httpSSLWorkerThread-9001-5' Id=930, WAITING on
> lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6,
> total cpu time=11770.0000ms user time=11460.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'httpSSLWorkerThread-9001-4' Id=627, RUNNABLE on lock=, total cpu
> time=127170.0000ms user time=124390.0000ms
> at sun.management.ThreadImpl.getThreadInfo1(Native Method)
> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:154)
> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java
> from :265)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
>
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:723)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:558)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:490)
>
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:382)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
>
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
>
> at
> org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>
> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
> at
> com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>
> at
> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
> at
> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>
> at
> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>
> at
> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>
> at
> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>
> at
> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>
>
> 'New Relic Profiler Service' Id=521, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c389356,
> total cpu time=11000.0000ms user time=10620.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'FileCacheThread-8080WorkerThread-1-0' Id=192, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3723978c,
> total cpu time=0.0000ms user time=0.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at
> com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)
>
>
> 'pool-24-thread-1' Id=191, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7210a5d6,
> total cpu time=3200.0000ms user time=2650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'pool-19-thread-1' Id=190, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f66fb9b,
> total cpu time=153350.0000ms user time=153080.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Timer-6' Id=176, WAITING on lock=java.util.TaskQueue@22f7bf1b, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'httpSSLWorkerThread-9001-3' Id=170, WAITING on
> lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6,
> total cpu time=262580.0000ms user time=256970.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'New Relic Harvest Service' Id=162, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14c8375,
> total cpu time=8050.0000ms user time=7750.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'MultiThreadedHttpConnectionManager cleanup' Id=158, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@4ab38654, total cpu time=0.0000ms
> user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>
>
> 'httpSSLWorkerThread-9001-2' Id=157, WAITING on
> lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6,
> total cpu time=269650.0000ms user time=263680.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'JMX server connection timeout 156' Id=156, TIMED_WAITING on
> lock=[I@1712ab69, total cpu time=20.0000ms user time=20.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 155' Id=155, TIMED_WAITING on
> lock=[I@1373caf9, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 153' Id=153, TIMED_WAITING on
> lock=[I@776339fb, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 152' Id=152, TIMED_WAITING on
> lock=[I@44794bdb, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 151' Id=151, TIMED_WAITING on
> lock=[I@2eeb237d, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 150' Id=150, TIMED_WAITING on
> lock=[I@137bf469, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 146' Id=146, TIMED_WAITING on
> lock=[I@50381b1c, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 143' Id=143, TIMED_WAITING on
> lock=[I@51163f19, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Connection(7)-10.112.36.135' Id=138, TIMED_WAITING on
> lock=com.sun.jmx.remote.internal.ArrayNotificationBuffer@13bfde14, total
> cpu time=4750.0000ms user time=3520.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
>
> at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
>
> at
> com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
>
> at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 137' Id=137, TIMED_WAITING on
> lock=[I@7610ec7d, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
> 'Thread-35' Id=136, TIMED_WAITING on lock=, total cpu time=210.0000ms user
> time=180.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'DestroyJavaVM' Id=135, RUNNABLE on lock=, total cpu time=13840.0000ms
> user time=13160.0000ms
>
> 'Timer-5' Id=133, TIMED_WAITING on lock=java.util.TaskQueue@666452d1,
> total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'httpSSLWorkerThread-7001-1' Id=132, WAITING on
> lock=com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@39cd8ef4,
> total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'httpSSLWorkerThread-7001-0' Id=131, WAITING on
> lock=com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@39cd8ef4,
> total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'httpSSLWorkerThread-9001-1' Id=129, WAITING on
> lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6,
> total cpu time=274330.0000ms user time=268220.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'httpSSLWorkerThread-9001-0' Id=128, WAITING on
> lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6,
> total cpu time=272620.0000ms user time=266470.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
>
> at
> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
>
>
> 'SelectorThread-7001' Id=130, RUNNABLE on lock=, total cpu time=10.0000ms
> user time=10.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)
>
>
> 'SelectorThread-9001' Id=127, RUNNABLE on lock=, total cpu
> time=13970.0000ms user time=10020.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
>
> at
> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)
>
>
> 'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]' Id=126,
> TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]'
> Id=123, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]'
> Id=122, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]'
> Id=121, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/solr-admin]]'
> Id=120, TIMED_WAITING on lock=, total cpu time=10.0000ms user time=10.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'EEHADBHealthChecker' Id=119, TIMED_WAITING on lock=, total cpu
> time=0.0000ms user time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.threadSleep(EEHADBHealthChecker.java:467)
>
> at
> com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.run(EEHADBHealthChecker.java:479)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Thread-31' Id=118, TIMED_WAITING on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:151)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI RenewClean-[10.112.37.154:36622
> ,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory@174043f5]'
> Id=116, TIMED_WAITING on lock=java.lang.ref.ReferenceQueue$Lock@198cfd2d,
> total cpu time=60.0000ms user time=60.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'derby.rawStoreDaemon' Id=114, TIMED_WAITING on
> lock=org.apache.derby.impl.services.daemon.BasicDaemon@7741a275, total
> cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
> at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Timer-4' Id=111, WAITING on lock=java.util.TaskQueue@57c59197, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'derby.antiGC' Id=110, WAITING on
> lock=org.apache.derby.impl.services.monitor.AntiGC@383001ca, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:662)
>
> 'JMX server connection timeout 108' Id=108, TIMED_WAITING on
> lock=[I@6bd97d85, total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Connection(5)-10.112.36.135' Id=107, TIMED_WAITING on
> lock=com.sun.jmx.remote.internal.ArrayNotificationBuffer@13bfde14, total
> cpu time=4870.0000ms user time=3590.0000ms
> at java.lang.Object.wait(Native Method)
> at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
>
> at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
>
> at
> com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
>
> at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'iMQReadChannel-3' Id=103, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1811)
> at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1779)
> at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1706)
> at
> com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:109)
> at
> com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:64)
>
> at
> com.sun.messaging.jmq.jmsclient.protocol.SocketConnectionHandler.readPacket(SocketConnectionHandler.java:39)
>
> at
> com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1762)
>
> at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1217)
> at java.lang.Thread.run(Thread.java:662)
>
> 'imqConnectionFlowControl-3' Id=102, TIMED_WAITING on
> lock=com.sun.messaging.jmq.jmsclient.FlowControl@2a27bba2, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
> at java.lang.Thread.run(Thread.java:662)
>
> 'imqConsumerReader-2-5288420469363924736-0' Id=101, WAITING on
> lock=com.sun.messaging.jmq.jmsclient.SessionQueue@168f9e4d, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at
> com.sun.messaging.jmq.jmsclient.SessionQueue.dequeueWait(SessionQueue.java:236)
>
> at
> com.sun.messaging.jmq.jmsclient.ConsumerReader.run(ConsumerReader.java:162)
> at java.lang.Thread.run(Thread.java:662)
>
> 'iMQReadChannel-2' Id=100, RUNNABLE on lock=, total cpu time=20.0000ms
> user time=20.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1811)
> at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1779)
> at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1706)
> at
> com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:109)
> at
> com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:64)
>
> at
> com.sun.messaging.jmq.jmsclient.protocol.SocketConnectionHandler.readPacket(SocketConnectionHandler.java:39)
>
> at
> com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1762)
>
> at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1217)
> at java.lang.Thread.run(Thread.java:662)
>
> 'imqConnectionFlowControl-2' Id=99, TIMED_WAITING on
> lock=com.sun.messaging.jmq.jmsclient.FlowControl@2ee1dccd, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Thread-22' Id=98, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:220)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at java.io.BufferedReader.fill(BufferedReader.java:136)
> at java.io.BufferedReader.readLine(BufferedReader.java:299)
> at java.io.BufferedReader.readLine(BufferedReader.java:362)
> at
> com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$StreamGobbler.run(JMSAdminImpl.java:1456)
>
>
> 'Thread-21' Id=97, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:220)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at java.io.BufferedReader.fill(BufferedReader.java:136)
> at java.io.BufferedReader.readLine(BufferedReader.java:299)
> at java.io.BufferedReader.readLine(BufferedReader.java:362)
> at
> com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$StreamGobbler.run(JMSAdminImpl.java:1456)
>
>
> 'Thread-20' Id=96, WAITING on lock=java.lang.UNIXProcess@6e672ec6, total
> cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
> at
> com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$BrokerWatcher.run(JMSAdminImpl.java:1423)
>
>
> 'process reaper' Id=95, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.lang.UNIXProcess.waitForProcessExit(Native Method)
> at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
> at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
>
> 'RMI RenewClean-[10.112.37.154:45172]' Id=88, TIMED_WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@7d8ae692, total cpu time=80.0000ms
> user time=80.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Accept-0' Id=86, RUNNABLE on lock=, total cpu time=30.0000ms user
> time=20.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Accept-3012' Id=85, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Thread-16' Id=82, WAITING on
> lock=com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive@5fc6ef55, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:865)
>
> 'p: thread-pool-1; w: 2' Id=81, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at
> com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
>
> at
> com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:251)
>
> at
> com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
>
> at
> com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
>
>
> 'p: thread-pool-1; w: 1' Id=80, RUNNABLE on lock=, total cpu
> time=10.0000ms user time=10.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at
> com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
>
> at
> com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:251)
>
> at
> com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
>
> at
> com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
>
>
> 'SelectorThread' Id=79, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)
>
> 'Timer-3' Id=78, WAITING on lock=java.util.TaskQueue@1ce6efeb, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'Timer-2' Id=77, TIMED_WAITING on lock=java.util.TaskQueue@38c9223d,
> total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'Thread-12' Id=76, TIMED_WAITING on lock=, total cpu time=240.0000ms user
> time=210.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI RenewClean-[10.112.36.135:46272]' Id=75, TIMED_WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@4a0257a0, total cpu
> time=100.0000ms user time=100.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Timer-1' Id=73, WAITING on lock=java.util.TaskQueue@7d913c1f, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'BlockingMessenger self destruct timer' Id=70, TIMED_WAITING on
> lock=java.util.TaskQueue@183e0c4b, total cpu time=0.0000ms user
> time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'Listener Adaptor' Id=66, WAITING on lock=java.util.HashSet@5b14e50a,
> total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at net.jxta.util.SimpleSelector.select(SimpleSelector.java:142)
> at net.jxta.endpoint.ListenerAdaptor.run(ListenerAdaptor.java:412)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Executor - 5' Id=65, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
> total cpu time=6450.0000ms user time=6350.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'FailureVerifier Thread for Group:lucene-gog' Id=64, WAITING on
> lock=java.lang.Object@11991c5c, total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> com.sun.enterprise.jxtamgmt.HealthMonitor$FailureVerifier.run(HealthMonitor.java:1436)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'InDoubtPeerDetector Thread for Group:lucene-gog' Id=63, TIMED_WAITING on
> lock=java.lang.Object@642f83eb, total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at
> com.sun.enterprise.jxtamgmt.HealthMonitor$InDoubtPeerDetector.run(HealthMonitor.java:1299)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'HealthMonitor for Group:lucene-gog' Id=62, TIMED_WAITING on
> lock=java.lang.Object@6b7decaf, total cpu time=18270.0000ms user
> time=16210.0000ms
>
> at java.lang.Object.wait(Native Method)
> at com.sun.enterprise.jxtamgmt.HealthMonitor.run(HealthMonitor.java:829)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Executor - 4' Id=61, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
> total cpu time=6280.0000ms user time=6170.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Executor - 3' Id=59, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
> total cpu time=6280.0000ms user time=6200.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Executor - 2' Id=58, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
> total cpu time=6260.0000ms user time=6170.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Timer-0' Id=57, WAITING on lock=java.util.TaskQueue@51d4b6d5, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'Executor - 1' Id=56, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
> total cpu time=6100.0000ms user time=6010.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Relay Client Worker Thread for
> relay://uuid-59616261646162614A787461503250330A0958A7971B4310872AEDF7285B115C03'
> Id=55, TIMED_WAITING on lock=, total cpu time=1310.0000ms user
> time=1290.0000ms
> at java.lang.Thread.sleep(Native Method)
> at net.jxta.impl.endpoint.relay.RelayClient.run(RelayClient.java:327)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Endpoint Destinations GC' Id=54, TIMED_WAITING on
> lock=java.util.TaskQueue@205583e2, total cpu time=0.0000ms user
> time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'Pipe Resolver Srdi Thread' Id=53, TIMED_WAITING on
> lock=java.lang.String@4e457b03, total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
> at java.lang.Thread.run(Thread.java:662)
>
> 'SrdiIndex GC :pipeResolverSrdi every 60000ms' Id=52, TIMED_WAITING on
> lock=java.lang.Thread@338f2a1a, total cpu time=40.0000ms user
> time=40.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.SrdiIndex.run(SrdiIndex.java:807)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Discovery Srdi Thread' Id=51, TIMED_WAITING on
> lock=java.lang.String@50485a32, total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
> at java.lang.Thread.run(Thread.java:662)
>
> 'StdRendezVousService Timer for
> urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
> Id=50, TIMED_WAITING on lock=java.util.TaskQueue@3e19b1cb, total cpu
> time=2360.0000ms user time=2350.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'RendezVousServiceImpl Timer for
> urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
> Id=49, WAITING on lock=java.util.TaskQueue@8e7a647, total cpu
> time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'IP Multicast Listener for mcast://228.8.69.19:40213' Id=48, RUNNABLE on
> lock=, total cpu time=4360.0000ms user time=3130.0000ms
> at java.net.PlainDatagramSocketImpl.receive0(Native Method)
> at
> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
> at java.net.DatagramSocket.receive(DatagramSocket.java:725)
> at
> net.jxta.impl.endpoint.mcast.McastTransport.run(McastTransport.java:605)
> at java.lang.Thread.run(Thread.java:662)
>
> 'EndpointRouter Timer for
> urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
> Id=47, WAITING on lock=java.util.TaskQueue@7ba358b0, total cpu
> time=0.0000ms user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'CM GC Thread interval : 60000' Id=46, TIMED_WAITING on
> lock=net.jxta.impl.cm.Cm@6b4acc2f, total cpu time=0.0000ms user
> time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.Cm.run(Cm.java:1119)
> at java.lang.Thread.run(Thread.java:662)
>
> 'Discovery Srdi Thread' Id=45, TIMED_WAITING on
> lock=java.lang.String@1a41b6cd, total cpu time=0.0000ms user time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
> at java.lang.Thread.run(Thread.java:662)
>
> 'RendezVousServiceImpl Timer for urn:jxta:jxta-WorldGroup' Id=44, WAITING
> on lock=java.util.TaskQueue@2113ca3a, total cpu time=0.0000ms user
> time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> at java.util.TimerThread.run(Timer.java:462)
>
> 'ServerSocketChannel acceptor for /0:0:0:0:0:0:0:0:9807' Id=43, RUNNABLE
> on lock=, total cpu time=0.0000ms user time=0.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> at
> net.jxta.impl.endpoint.tcp.IncomingUnicastServer.run(IncomingUnicastServer.java:257)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'TCP Transport MessengerSelectorThread for
> net.jxta.impl.endpoint.tcp.TcpTransport@68ce16ba' Id=42, RUNNABLE on
> lock=, total cpu time=0.0000ms user time=0.0000ms
>
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> at
> net.jxta.impl.endpoint.tcp.TcpTransport$MessengerSelectorThread.run(TcpTransport.java:1009)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'CM GC Thread interval : 60000' Id=41, TIMED_WAITING on
> lock=net.jxta.impl.cm.Cm@3db3dc66, total cpu time=0.0000ms user
> time=0.0000ms
>
> at java.lang.Object.wait(Native Method)
> at net.jxta.impl.cm.Cm.run(Cm.java:1119)
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI Scheduler(0)' Id=40, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d379b49,
> total cpu time=220.0000ms user time=90.0000ms
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'ViewWindowThread:lucene-gog' Id=38, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1998df65,
> total cpu time=10.0000ms user time=10.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> at com.sun.enterprise.ee.cms.impl.jxta.ViewWindow.run(ViewWindow.java:113)
> at java.lang.Thread.run(Thread.java:662)
>
> 'MessageWindowThread:lucene-gog' Id=39, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3726d04,
> total cpu time=30.0000ms user time=30.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> at
> com.sun.enterprise.ee.cms.impl.jxta.MessageWindow.run(MessageWindow.java:88)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'com.sun.enterprise.ee.cms.impl.common.Router Thread' Id=36, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4d532360,
> total cpu time=10.0000ms user time=10.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> at
> com.sun.enterprise.ee.cms.impl.common.SignalHandler.run(SignalHandler.java:85)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'GC Daemon' Id=35, TIMED_WAITING on lock=sun.misc.GC$LatencyLock@2a643801,
> total cpu time=0.0000ms user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at sun.misc.GC$Daemon.run(GC.java:100)
>
> 'RMI Reaper' Id=34, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@4d0d54f, total cpu time=0.0000ms
> user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
> at java.lang.Thread.run(Thread.java:662)
>
> 'RMI TCP Accept-0' Id=33, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:662)
>
> 'New Relic RPM Connection Service' Id=9, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742a4bb6,
> total cpu time=190.0000ms user time=180.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'New Relic Sampler Service' Id=8, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20fbd054,
> total cpu time=800.0000ms user time=690.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'New Relic Deadlock Detector' Id=7, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3702ec9b,
> total cpu time=10.0000ms user time=10.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'New Relic Thread Service' Id=6, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b1bef69,
> total cpu time=0.0000ms user time=0.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
>
> 'Finalizer' Id=3, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@3f7927ad, total cpu time=90.0000ms
> user time=90.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@5b7487b3, total cpu time=2980.0000ms
> user time=2830.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> RETURN TO ADMIN PAGE
>
> Ravi Kiran Bhaskar
>
>
> On Wed, May 1, 2013 at 9:29 PM, Ravi Solr <ra...@gmail.com> wrote:
>
>> Shawn,
>> I don't believe its the container because we use the same container
>> in another setup that has 6 cores which is serving almost 1.8 Million
>> requests a day without a hitch.
>>
>> If you look at my email the container that is running SOLR got the
>> request params (http access logs provided in first email) but when it goes
>> through the SOLR app/code on the container (probably through request
>> filters or dispatchers..I don't know exactly) its getting lost, which is
>> what I am trying to understand. I want to understand under what situations
>> this mat happen.
>>
>> Having said that this application that uses this problematic SOLR
>> instance retrieves large number of facets results for each of 26 facets for
>> each query and every query is a group query, would that cause any issues
>> with SOLR caches that could lead to the issues like I am facing ???
>>
>> With regards to the port number, our paranoid security folks wanted me to
>> not reveal our ports so I put it as 80 without thinking :-), I assure use
>> that its not 80.
>>
>> Thanks,
>>
>> Ravi
>>
>>
>> On Wed, May 1, 2013 at 6:03 PM, Shawn Heisey <so...@elyograg.org> wrote:
>>
>>> On 5/1/2013 3:14 PM, Ravi Solr wrote:
>>>
>>>> We are using Solr 3.6.2 with a single core setup on a glassfish server,
>>>> every 4-5 hours the server gradually gets into a some kind of a
>>>> inconsistent state and stops accepting any queries giving back cached
>>>> results. Even the core reload fails giving the following. Has anybody
>>>> experienced such behavior ? Can anybody help me understand why this
>>>> might
>>>> happen ?
>>>>
>>>> http://searchserver:80/solr/**admin/cores?action=RELOAD&**core=core1<http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1>
>>>>
>>>> <response>
>>>> <lst name="responseHeader">
>>>> <int name="status">0</int>
>>>> <int name="QTime">9</int>
>>>> </lst>
>>>> <lst name="status">
>>>>
>>>
>>> It is dropping the parameters from the /admin/cores request too, so it
>>> returns status instead of acting on the RELOAD.
>>>
>>> This is acting like a servlet container issue more than a Solr issue.
>>> It's always possible that it actually is Solr.
>>>
>>> It's a little unusual to see Solr running on port 80. It's not
>>> impossible, just not the normal setup, because exposing Solr directly to
>>> the outside world is a very bad idea, so it's a lot safer to have it listen
>>> on another port.
>>>
>>> Is glassfish actually listening on port 80? If it's not, then you
>>> probably have something acting as a proxy in front of Solr. If your
>>> platform is a UNIX variant or Linux and has a fully functional 'lsof'
>>> command, the following will tell you which process is bound to port 80:
>>>
>>> lsof -nPi | grep ":80"
>>>
>>> Can you try running Solr under the jetty that's included with the Solr
>>> download? For Solr 3.6.2, this is a slightly modified Jetty 6. You can't
>>> use the Jetty 8 that's included with a newer version of Solr. If port 80
>>> is a requirement, that should be possible as long as it's running as root.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
>
Re: Server inconsistent state & Core Reload issue
Posted by Ravi Solr <ra...@gmail.com>.
Hello,
I managed to get the thread dump from solr admin console during the
time it was failing. I must admit I am not that great with reading thread
dumps, can somebody help me figure out anything weird in this thread dump
ava HotSpot(TM) 64-Bit Server VM20.4-b02
Thread Count: current=109, peak=130, daemon=85
'RMI TCP Connection(328)-10.112.36.135' Id=960, RUNNABLE on lock=, total
cpu time=0.0000ms user time=0.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 950' Id=950, TIMED_WAITING on
lock=[I@652381db, total cpu time=320.0000ms user time=230.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Connection(313)-10.8.7.69' Id=949, RUNNABLE on lock=, total cpu
time=8530.0000ms user time=7980.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Connection(327)-10.112.36.135' Id=948, RUNNABLE on lock=, total
cpu time=60.0000ms user time=60.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
'Thread-67' Id=942, TIMED_WAITING on
lock=com.sun.appserv.management.util.jmx.NotificationEmitterSupport$SenderThread@7e2284bc,
total cpu time=210.0000ms user time=150.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.appserv.management.util.jmx.NotificationEmitterSupport$SenderThread.run(NotificationEmitterSupport.java:414)
'httpSSLWorkerThread-9001-5' Id=930, WAITING on
lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6, total
cpu time=11770.0000ms user time=11460.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'httpSSLWorkerThread-9001-4' Id=627, RUNNABLE on lock=, total cpu
time=127170.0000ms user time=124390.0000ms
at sun.management.ThreadImpl.getThreadInfo1(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:154)
at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java from
:265)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
at
org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:723)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:558)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:490)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
'New Relic Profiler Service' Id=521, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c389356,
total cpu time=11000.0000ms user time=10620.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'FileCacheThread-8080WorkerThread-1-0' Id=192, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3723978c,
total cpu time=0.0000ms user time=0.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)
'pool-24-thread-1' Id=191, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7210a5d6,
total cpu time=3200.0000ms user time=2650.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'pool-19-thread-1' Id=190, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f66fb9b,
total cpu time=153350.0000ms user time=153080.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Timer-6' Id=176, WAITING on lock=java.util.TaskQueue@22f7bf1b, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'httpSSLWorkerThread-9001-3' Id=170, WAITING on
lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6, total
cpu time=262580.0000ms user time=256970.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'New Relic Harvest Service' Id=162, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14c8375,
total cpu time=8050.0000ms user time=7750.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'MultiThreadedHttpConnectionManager cleanup' Id=158, WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@4ab38654, total cpu time=0.0000ms
user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
'httpSSLWorkerThread-9001-2' Id=157, WAITING on
lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6, total
cpu time=269650.0000ms user time=263680.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'JMX server connection timeout 156' Id=156, TIMED_WAITING on
lock=[I@1712ab69, total cpu time=20.0000ms user time=20.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 155' Id=155, TIMED_WAITING on
lock=[I@1373caf9, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 153' Id=153, TIMED_WAITING on
lock=[I@776339fb, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 152' Id=152, TIMED_WAITING on
lock=[I@44794bdb, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 151' Id=151, TIMED_WAITING on
lock=[I@2eeb237d, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 150' Id=150, TIMED_WAITING on
lock=[I@137bf469, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 146' Id=146, TIMED_WAITING on
lock=[I@50381b1c, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 143' Id=143, TIMED_WAITING on
lock=[I@51163f19, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Connection(7)-10.112.36.135' Id=138, TIMED_WAITING on
lock=com.sun.jmx.remote.internal.ArrayNotificationBuffer@13bfde14, total
cpu time=4750.0000ms user time=3520.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
at
com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
at
com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
at
javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 137' Id=137, TIMED_WAITING on
lock=[I@7610ec7d, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'Thread-35' Id=136, TIMED_WAITING on lock=, total cpu time=210.0000ms user
time=180.0000ms
at java.lang.Thread.sleep(Native Method)
at
com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
at java.lang.Thread.run(Thread.java:662)
'DestroyJavaVM' Id=135, RUNNABLE on lock=, total cpu time=13840.0000ms user
time=13160.0000ms
'Timer-5' Id=133, TIMED_WAITING on lock=java.util.TaskQueue@666452d1, total
cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
'httpSSLWorkerThread-7001-1' Id=132, WAITING on
lock=com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@39cd8ef4,
total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'httpSSLWorkerThread-7001-0' Id=131, WAITING on
lock=com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@39cd8ef4,
total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'httpSSLWorkerThread-9001-1' Id=129, WAITING on
lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6, total
cpu time=274330.0000ms user time=268220.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'httpSSLWorkerThread-9001-0' Id=128, WAITING on
lock=com.sun.enterprise.web.portunif.PortUnificationPipeline@b88abe6, total
cpu time=272620.0000ms user time=266470.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)
'SelectorThread-7001' Id=130, RUNNABLE on lock=, total cpu time=10.0000ms
user time=10.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)
'SelectorThread-9001' Id=127, RUNNABLE on lock=, total cpu
time=13970.0000ms user time=10020.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)
'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]' Id=126,
TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
at java.lang.Thread.run(Thread.java:662)
'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]'
Id=123, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
at java.lang.Thread.run(Thread.java:662)
'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]'
Id=122, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
at java.lang.Thread.run(Thread.java:662)
'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]'
Id=121, TIMED_WAITING on lock=, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
at java.lang.Thread.run(Thread.java:662)
'ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/solr-admin]]'
Id=120, TIMED_WAITING on lock=, total cpu time=10.0000ms user time=10.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
at java.lang.Thread.run(Thread.java:662)
'EEHADBHealthChecker' Id=119, TIMED_WAITING on lock=, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.threadSleep(EEHADBHealthChecker.java:467)
at
com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.run(EEHADBHealthChecker.java:479)
at java.lang.Thread.run(Thread.java:662)
'Thread-31' Id=118, TIMED_WAITING on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Thread.sleep(Native Method)
at
com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:151)
at java.lang.Thread.run(Thread.java:662)
'RMI RenewClean-[10.112.37.154:36622
,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory@174043f5]'
Id=116, TIMED_WAITING on lock=java.lang.ref.ReferenceQueue$Lock@198cfd2d,
total cpu time=60.0000ms user time=60.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
'derby.rawStoreDaemon' Id=114, TIMED_WAITING on
lock=org.apache.derby.impl.services.daemon.BasicDaemon@7741a275, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
'Timer-4' Id=111, WAITING on lock=java.util.TaskQueue@57c59197, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'derby.antiGC' Id=110, WAITING on
lock=org.apache.derby.impl.services.monitor.AntiGC@383001ca, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
'JMX server connection timeout 108' Id=108, TIMED_WAITING on
lock=[I@6bd97d85, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Connection(5)-10.112.36.135' Id=107, TIMED_WAITING on
lock=com.sun.jmx.remote.internal.ArrayNotificationBuffer@13bfde14, total
cpu time=4870.0000ms user time=3590.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
at
com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
at
com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
at
javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
'iMQReadChannel-3' Id=103, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1811)
at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1779)
at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1706)
at
com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:109)
at
com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:64)
at
com.sun.messaging.jmq.jmsclient.protocol.SocketConnectionHandler.readPacket(SocketConnectionHandler.java:39)
at
com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1762)
at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1217)
at java.lang.Thread.run(Thread.java:662)
'imqConnectionFlowControl-3' Id=102, TIMED_WAITING on
lock=com.sun.messaging.jmq.jmsclient.FlowControl@2a27bba2, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
at java.lang.Thread.run(Thread.java:662)
'imqConsumerReader-2-5288420469363924736-0' Id=101, WAITING on
lock=com.sun.messaging.jmq.jmsclient.SessionQueue@168f9e4d, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.messaging.jmq.jmsclient.SessionQueue.dequeueWait(SessionQueue.java:236)
at
com.sun.messaging.jmq.jmsclient.ConsumerReader.run(ConsumerReader.java:162)
at java.lang.Thread.run(Thread.java:662)
'iMQReadChannel-2' Id=100, RUNNABLE on lock=, total cpu time=20.0000ms user
time=20.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1811)
at com.sun.messaging.jmq.io.Packet.readFully(Packet.java:1779)
at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1706)
at
com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:109)
at
com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:64)
at
com.sun.messaging.jmq.jmsclient.protocol.SocketConnectionHandler.readPacket(SocketConnectionHandler.java:39)
at
com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1762)
at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1217)
at java.lang.Thread.run(Thread.java:662)
'imqConnectionFlowControl-2' Id=99, TIMED_WAITING on
lock=com.sun.messaging.jmq.jmsclient.FlowControl@2ee1dccd, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
at java.lang.Thread.run(Thread.java:662)
'Thread-22' Id=98, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at
com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$StreamGobbler.run(JMSAdminImpl.java:1456)
'Thread-21' Id=97, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at
com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$StreamGobbler.run(JMSAdminImpl.java:1456)
'Thread-20' Id=96, WAITING on lock=java.lang.UNIXProcess@6e672ec6, total
cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
at
com.sun.messaging.jmq.admin.jmsspi.JMSAdminImpl$BrokerWatcher.run(JMSAdminImpl.java:1423)
'process reaper' Id=95, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
'RMI RenewClean-[10.112.37.154:45172]' Id=88, TIMED_WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@7d8ae692, total cpu time=80.0000ms
user time=80.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Accept-0' Id=86, RUNNABLE on lock=, total cpu time=30.0000ms user
time=20.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Accept-3012' Id=85, RUNNABLE on lock=, total cpu time=0.0000ms
user time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
'Thread-16' Id=82, WAITING on
lock=com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive@5fc6ef55, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:865)
'p: thread-pool-1; w: 2' Id=81, RUNNABLE on lock=, total cpu time=0.0000ms
user time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at
com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
at
com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:251)
at
com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
at
com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
'p: thread-pool-1; w: 1' Id=80, RUNNABLE on lock=, total cpu time=10.0000ms
user time=10.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at
com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
at
com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:251)
at
com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
at
com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
'SelectorThread' Id=79, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)
'Timer-3' Id=78, WAITING on lock=java.util.TaskQueue@1ce6efeb, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'Timer-2' Id=77, TIMED_WAITING on lock=java.util.TaskQueue@38c9223d, total
cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
'Thread-12' Id=76, TIMED_WAITING on lock=, total cpu time=240.0000ms user
time=210.0000ms
at java.lang.Thread.sleep(Native Method)
at
com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
at java.lang.Thread.run(Thread.java:662)
'RMI RenewClean-[10.112.36.135:46272]' Id=75, TIMED_WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@4a0257a0, total cpu time=100.0000ms
user time=100.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
'Timer-1' Id=73, WAITING on lock=java.util.TaskQueue@7d913c1f, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'BlockingMessenger self destruct timer' Id=70, TIMED_WAITING on
lock=java.util.TaskQueue@183e0c4b, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
'Listener Adaptor' Id=66, WAITING on lock=java.util.HashSet@5b14e50a, total
cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at net.jxta.util.SimpleSelector.select(SimpleSelector.java:142)
at net.jxta.endpoint.ListenerAdaptor.run(ListenerAdaptor.java:412)
at java.lang.Thread.run(Thread.java:662)
'Executor - 5' Id=65, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
total cpu time=6450.0000ms user time=6350.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'FailureVerifier Thread for Group:lucene-gog' Id=64, WAITING on
lock=java.lang.Object@11991c5c, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
com.sun.enterprise.jxtamgmt.HealthMonitor$FailureVerifier.run(HealthMonitor.java:1436)
at java.lang.Thread.run(Thread.java:662)
'InDoubtPeerDetector Thread for Group:lucene-gog' Id=63, TIMED_WAITING on
lock=java.lang.Object@642f83eb, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at
com.sun.enterprise.jxtamgmt.HealthMonitor$InDoubtPeerDetector.run(HealthMonitor.java:1299)
at java.lang.Thread.run(Thread.java:662)
'HealthMonitor for Group:lucene-gog' Id=62, TIMED_WAITING on
lock=java.lang.Object@6b7decaf, total cpu time=18270.0000ms user
time=16210.0000ms
at java.lang.Object.wait(Native Method)
at com.sun.enterprise.jxtamgmt.HealthMonitor.run(HealthMonitor.java:829)
at java.lang.Thread.run(Thread.java:662)
'Executor - 4' Id=61, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
total cpu time=6280.0000ms user time=6170.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Executor - 3' Id=59, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
total cpu time=6280.0000ms user time=6200.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Executor - 2' Id=58, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
total cpu time=6260.0000ms user time=6170.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Timer-0' Id=57, WAITING on lock=java.util.TaskQueue@51d4b6d5, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'Executor - 1' Id=56, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c152b67,
total cpu time=6100.0000ms user time=6010.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Relay Client Worker Thread for
relay://uuid-59616261646162614A787461503250330A0958A7971B4310872AEDF7285B115C03'
Id=55, TIMED_WAITING on lock=, total cpu time=1310.0000ms user
time=1290.0000ms
at java.lang.Thread.sleep(Native Method)
at net.jxta.impl.endpoint.relay.RelayClient.run(RelayClient.java:327)
at java.lang.Thread.run(Thread.java:662)
'Endpoint Destinations GC' Id=54, TIMED_WAITING on
lock=java.util.TaskQueue@205583e2, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
'Pipe Resolver Srdi Thread' Id=53, TIMED_WAITING on
lock=java.lang.String@4e457b03, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
at java.lang.Thread.run(Thread.java:662)
'SrdiIndex GC :pipeResolverSrdi every 60000ms' Id=52, TIMED_WAITING on
lock=java.lang.Thread@338f2a1a, total cpu time=40.0000ms user time=40.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.SrdiIndex.run(SrdiIndex.java:807)
at java.lang.Thread.run(Thread.java:662)
'Discovery Srdi Thread' Id=51, TIMED_WAITING on
lock=java.lang.String@50485a32, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
at java.lang.Thread.run(Thread.java:662)
'StdRendezVousService Timer for
urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
Id=50, TIMED_WAITING on lock=java.util.TaskQueue@3e19b1cb, total cpu
time=2360.0000ms user time=2350.0000ms
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
'RendezVousServiceImpl Timer for
urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
Id=49, WAITING on lock=java.util.TaskQueue@8e7a647, total cpu time=0.0000ms
user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'IP Multicast Listener for mcast://228.8.69.19:40213' Id=48, RUNNABLE on
lock=, total cpu time=4360.0000ms user time=3130.0000ms
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at net.jxta.impl.endpoint.mcast.McastTransport.run(McastTransport.java:605)
at java.lang.Thread.run(Thread.java:662)
'EndpointRouter Timer for
urn:jxta:uuid-8A7A26B97917415A922CB7504A883A4E59616261646162614E5047205032503302'
Id=47, WAITING on lock=java.util.TaskQueue@7ba358b0, total cpu
time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'CM GC Thread interval : 60000' Id=46, TIMED_WAITING on
lock=net.jxta.impl.cm.Cm@6b4acc2f, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.Cm.run(Cm.java:1119)
at java.lang.Thread.run(Thread.java:662)
'Discovery Srdi Thread' Id=45, TIMED_WAITING on
lock=java.lang.String@1a41b6cd, total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.Srdi.run(Srdi.java:607)
at java.lang.Thread.run(Thread.java:662)
'RendezVousServiceImpl Timer for urn:jxta:jxta-WorldGroup' Id=44, WAITING
on lock=java.util.TaskQueue@2113ca3a, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
'ServerSocketChannel acceptor for /0:0:0:0:0:0:0:0:9807' Id=43, RUNNABLE on
lock=, total cpu time=0.0000ms user time=0.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
net.jxta.impl.endpoint.tcp.IncomingUnicastServer.run(IncomingUnicastServer.java:257)
at java.lang.Thread.run(Thread.java:662)
'TCP Transport MessengerSelectorThread for
net.jxta.impl.endpoint.tcp.TcpTransport@68ce16ba' Id=42, RUNNABLE on lock=,
total cpu time=0.0000ms user time=0.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
net.jxta.impl.endpoint.tcp.TcpTransport$MessengerSelectorThread.run(TcpTransport.java:1009)
at java.lang.Thread.run(Thread.java:662)
'CM GC Thread interval : 60000' Id=41, TIMED_WAITING on
lock=net.jxta.impl.cm.Cm@3db3dc66, total cpu time=0.0000ms user
time=0.0000ms
at java.lang.Object.wait(Native Method)
at net.jxta.impl.cm.Cm.run(Cm.java:1119)
at java.lang.Thread.run(Thread.java:662)
'RMI Scheduler(0)' Id=40, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d379b49,
total cpu time=220.0000ms user time=90.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'ViewWindowThread:lucene-gog' Id=38, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1998df65,
total cpu time=10.0000ms user time=10.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at com.sun.enterprise.ee.cms.impl.jxta.ViewWindow.run(ViewWindow.java:113)
at java.lang.Thread.run(Thread.java:662)
'MessageWindowThread:lucene-gog' Id=39, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3726d04,
total cpu time=30.0000ms user time=30.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at
com.sun.enterprise.ee.cms.impl.jxta.MessageWindow.run(MessageWindow.java:88)
at java.lang.Thread.run(Thread.java:662)
'com.sun.enterprise.ee.cms.impl.common.Router Thread' Id=36, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4d532360,
total cpu time=10.0000ms user time=10.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at
com.sun.enterprise.ee.cms.impl.common.SignalHandler.run(SignalHandler.java:85)
at java.lang.Thread.run(Thread.java:662)
'GC Daemon' Id=35, TIMED_WAITING on lock=sun.misc.GC$LatencyLock@2a643801,
total cpu time=0.0000ms user time=0.0000ms
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(GC.java:100)
'RMI Reaper' Id=34, WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@4d0d54f, total cpu time=0.0000ms
user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at java.lang.Thread.run(Thread.java:662)
'RMI TCP Accept-0' Id=33, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
'New Relic RPM Connection Service' Id=9, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742a4bb6,
total cpu time=190.0000ms user time=180.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'New Relic Sampler Service' Id=8, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20fbd054,
total cpu time=800.0000ms user time=690.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'New Relic Deadlock Detector' Id=7, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3702ec9b,
total cpu time=10.0000ms user time=10.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'New Relic Thread Service' Id=6, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2b1bef69,
total cpu time=0.0000ms user time=0.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@3f7927ad,
total cpu time=90.0000ms user time=90.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
'Reference Handler' Id=2, WAITING on
lock=java.lang.ref.Reference$Lock@5b7487b3, total cpu time=2980.0000ms user
time=2830.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
RETURN TO ADMIN PAGE
Ravi Kiran Bhaskar
On Wed, May 1, 2013 at 9:29 PM, Ravi Solr <ra...@gmail.com> wrote:
> Shawn,
> I don't believe its the container because we use the same container
> in another setup that has 6 cores which is serving almost 1.8 Million
> requests a day without a hitch.
>
> If you look at my email the container that is running SOLR got the request
> params (http access logs provided in first email) but when it goes through
> the SOLR app/code on the container (probably through request filters or
> dispatchers..I don't know exactly) its getting lost, which is what I am
> trying to understand. I want to understand under what situations this mat
> happen.
>
> Having said that this application that uses this problematic SOLR instance
> retrieves large number of facets results for each of 26 facets for each
> query and every query is a group query, would that cause any issues with
> SOLR caches that could lead to the issues like I am facing ???
>
> With regards to the port number, our paranoid security folks wanted me to
> not reveal our ports so I put it as 80 without thinking :-), I assure use
> that its not 80.
>
> Thanks,
>
> Ravi
>
>
> On Wed, May 1, 2013 at 6:03 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
>> On 5/1/2013 3:14 PM, Ravi Solr wrote:
>>
>>> We are using Solr 3.6.2 with a single core setup on a glassfish server,
>>> every 4-5 hours the server gradually gets into a some kind of a
>>> inconsistent state and stops accepting any queries giving back cached
>>> results. Even the core reload fails giving the following. Has anybody
>>> experienced such behavior ? Can anybody help me understand why this might
>>> happen ?
>>>
>>> http://searchserver:80/solr/**admin/cores?action=RELOAD&**core=core1<http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1>
>>>
>>> <response>
>>> <lst name="responseHeader">
>>> <int name="status">0</int>
>>> <int name="QTime">9</int>
>>> </lst>
>>> <lst name="status">
>>>
>>
>> It is dropping the parameters from the /admin/cores request too, so it
>> returns status instead of acting on the RELOAD.
>>
>> This is acting like a servlet container issue more than a Solr issue.
>> It's always possible that it actually is Solr.
>>
>> It's a little unusual to see Solr running on port 80. It's not
>> impossible, just not the normal setup, because exposing Solr directly to
>> the outside world is a very bad idea, so it's a lot safer to have it listen
>> on another port.
>>
>> Is glassfish actually listening on port 80? If it's not, then you
>> probably have something acting as a proxy in front of Solr. If your
>> platform is a UNIX variant or Linux and has a fully functional 'lsof'
>> command, the following will tell you which process is bound to port 80:
>>
>> lsof -nPi | grep ":80"
>>
>> Can you try running Solr under the jetty that's included with the Solr
>> download? For Solr 3.6.2, this is a slightly modified Jetty 6. You can't
>> use the Jetty 8 that's included with a newer version of Solr. If port 80
>> is a requirement, that should be possible as long as it's running as root.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: Server inconsistent state & Core Reload issue
Posted by Ravi Solr <ra...@gmail.com>.
Shawn,
I don't believe its the container because we use the same container
in another setup that has 6 cores which is serving almost 1.8 Million
requests a day without a hitch.
If you look at my email the container that is running SOLR got the request
params (http access logs provided in first email) but when it goes through
the SOLR app/code on the container (probably through request filters or
dispatchers..I don't know exactly) its getting lost, which is what I am
trying to understand. I want to understand under what situations this mat
happen.
Having said that this application that uses this problematic SOLR instance
retrieves large number of facets results for each of 26 facets for each
query and every query is a group query, would that cause any issues with
SOLR caches that could lead to the issues like I am facing ???
With regards to the port number, our paranoid security folks wanted me to
not reveal our ports so I put it as 80 without thinking :-), I assure use
that its not 80.
Thanks,
Ravi
On Wed, May 1, 2013 at 6:03 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 5/1/2013 3:14 PM, Ravi Solr wrote:
>
>> We are using Solr 3.6.2 with a single core setup on a glassfish server,
>> every 4-5 hours the server gradually gets into a some kind of a
>> inconsistent state and stops accepting any queries giving back cached
>> results. Even the core reload fails giving the following. Has anybody
>> experienced such behavior ? Can anybody help me understand why this might
>> happen ?
>>
>> http://searchserver:80/solr/**admin/cores?action=RELOAD&**core=core1<http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1>
>>
>> <response>
>> <lst name="responseHeader">
>> <int name="status">0</int>
>> <int name="QTime">9</int>
>> </lst>
>> <lst name="status">
>>
>
> It is dropping the parameters from the /admin/cores request too, so it
> returns status instead of acting on the RELOAD.
>
> This is acting like a servlet container issue more than a Solr issue. It's
> always possible that it actually is Solr.
>
> It's a little unusual to see Solr running on port 80. It's not
> impossible, just not the normal setup, because exposing Solr directly to
> the outside world is a very bad idea, so it's a lot safer to have it listen
> on another port.
>
> Is glassfish actually listening on port 80? If it's not, then you
> probably have something acting as a proxy in front of Solr. If your
> platform is a UNIX variant or Linux and has a fully functional 'lsof'
> command, the following will tell you which process is bound to port 80:
>
> lsof -nPi | grep ":80"
>
> Can you try running Solr under the jetty that's included with the Solr
> download? For Solr 3.6.2, this is a slightly modified Jetty 6. You can't
> use the Jetty 8 that's included with a newer version of Solr. If port 80
> is a requirement, that should be possible as long as it's running as root.
>
> Thanks,
> Shawn
>
>
Re: Server inconsistent state & Core Reload issue
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/1/2013 3:14 PM, Ravi Solr wrote:
> We are using Solr 3.6.2 with a single core setup on a glassfish server,
> every 4-5 hours the server gradually gets into a some kind of a
> inconsistent state and stops accepting any queries giving back cached
> results. Even the core reload fails giving the following. Has anybody
> experienced such behavior ? Can anybody help me understand why this might
> happen ?
>
> http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1
>
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">9</int>
> </lst>
> <lst name="status">
It is dropping the parameters from the /admin/cores request too, so it
returns status instead of acting on the RELOAD.
This is acting like a servlet container issue more than a Solr issue.
It's always possible that it actually is Solr.
It's a little unusual to see Solr running on port 80. It's not
impossible, just not the normal setup, because exposing Solr directly to
the outside world is a very bad idea, so it's a lot safer to have it
listen on another port.
Is glassfish actually listening on port 80? If it's not, then you
probably have something acting as a proxy in front of Solr. If your
platform is a UNIX variant or Linux and has a fully functional 'lsof'
command, the following will tell you which process is bound to port 80:
lsof -nPi | grep ":80"
Can you try running Solr under the jetty that's included with the Solr
download? For Solr 3.6.2, this is a slightly modified Jetty 6. You
can't use the Jetty 8 that's included with a newer version of Solr. If
port 80 is a requirement, that should be possible as long as it's
running as root.
Thanks,
Shawn