You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yonik Seeley <yo...@lucidimagination.com> on 2009/04/02 21:00:28 UTC
Re: aka Replication Stall
Next time a hang happens, could you get a thread dump of the master also?
-Yonik
http://www.lucidimagination.com
On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
> I apologize for the delay. The replication stalling out doesn't happen
> daily. I will paste the thread dump below to try to help. This is on a
> server that is currently locked on replication for a few hours. Any more
> information please let me know. There are no errors in the logs either so
> very strange.
>
> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
> Thread Count: current=73, peak=125, daemon=70
> Full Thread Dump:
> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
> user time=23940.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
> a:182)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
> .java:108)
> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
> 22)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
> 808)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
> )
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
> at
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
> a:224)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
> 101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
> odic(ScheduledThreadPoolExecutor.java:181)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
> eduledThreadPoolExecutor.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-16-thread-1' Id=237, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-190' Id=232, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-189' Id=231, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-188' Id=230, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-187' Id=229, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-186' Id=228, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-182' Id=224, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-181' Id=223, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-180' Id=222, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-179' Id=221, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-178' Id=220, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-177' Id=219, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-175' Id=217, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-173' Id=215, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-171' Id=213, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-170' Id=212, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-169' Id=211, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-168' Id=210, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-167' Id=209, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-166' Id=208, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-165' Id=207, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-164' Id=206, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-161' Id=203, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-160' Id=202, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-159' Id=201, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-158' Id=200, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-157' Id=199, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-156' Id=198, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
> time=1800610.0000ms user time=1743610.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-153' Id=195, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-152' Id=194, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-150' Id=192, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-149' Id=191, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-147' Id=189, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-146' Id=188, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-145' Id=187, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-144' Id=186, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-143' Id=185, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-140' Id=182, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-137' Id=179, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-136' Id=178, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-133' Id=175, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-132' Id=174, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-131' Id=173, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-128' Id=170, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-127' Id=169, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-126' Id=165, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-125' Id=164, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-5' Id=152, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-4' Id=151, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-3' Id=150, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-2' Id=149, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-1' Id=148, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-107' Id=129, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-88' Id=110, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-65' Id=87, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
> user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
> time=31410.0000ms user time=24650.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
> user time=250.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> time=764410.0000ms user time=394670.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
> time=128740.0000ms user time=95550.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
> time=17960.0000ms user time=11850.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
> time=261720.0000ms user time=120120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
> SocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
> user time=15430.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
> time=3390.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
> time=1570.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject>
>>
>> We do not set an conn_timeoout,read_timeout for the httpclient in snappuller.
>>
>> I guess it should be set to some very high value say 1hr for
>> read-timeout and say 1 minute for conn_timeout and we can make it
>> configurable .
>>
>> --Noble
>>
>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>> <sh...@gmail.com> wrote:
>>> We should obviously get to the bottom of this. But I was thinking, should we
>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>
>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>> <yo...@lucidimagination.com>wrote:
>>>
>>>> So this is only one slave that hangs up and not the master?
>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>> wrote:
>>>>> We are having an intermittent problem with replication. We reindex
>>>> nightly
>>>>> which usually means there are 2 commits during replication then a final
>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>> occasionally with the following screenshot. This is frustrating as it
>>>> will
>>>>> completely stall out any further replications. Additionally, it seems to
>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>> always
>>>>> the same server.
>>>>>
>>>>>
>>>>> In case the screen shot doesn’t come through:
>>>>>
>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>> Replicatable Index Version:0, Generation: 0
>>>>> Poll Interval 00:05:00
>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>> Location: /opt/solr-data/zeta-main/index
>>>>> Size: 1.29 GB
>>>>> Times Replicated Since Startup: 3591
>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>> Config Files Replicated: [synonyms.txt]
>>>>> Times Config Files Replicated Since Startup: 4
>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>> Files Downloaded: 12 / 163
>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>> bytes/s
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
>>
>> --
>> --Noble Paul
>
>
Re: aka Replication Stall
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
On Sat, May 9, 2009 at 2:23 AM, Yonik Seeley <yo...@lucidimagination.com> wrote:
> 2009/5/7 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>:
>> BTW, if the timeout occurs it resumes from the point where the failure
>> happened. It retries 5 times before giving up.
>
> Sweet! I was just going to ask what happens on a timeout.
> Have you tested this out (say kill the master in the middle of
> replicating a big file)?
Actually yes,
not by killing the master (then the replication will abort). I
modified the master code to close the connection after transferring x
MB of data. The slave retried an completed the replication .
>
> -Yonik
> http://www.lucidimagination.com
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: aka Replication Stall
Posted by Yonik Seeley <yo...@lucidimagination.com>.
2009/5/7 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>:
> BTW, if the timeout occurs it resumes from the point where the failure
> happened. It retries 5 times before giving up.
Sweet! I was just going to ask what happens on a timeout.
Have you tested this out (say kill the master in the middle of
replicating a big file)?
-Yonik
http://www.lucidimagination.com
Re: aka Replication Stall
Posted by Jeff Newburn <jn...@zappos.com>.
Excellent! Thank you I am going to start testing that.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562
> From: Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Thu, 7 May 2009 20:26:02 +0530
> To: <so...@lucene.apache.org>
> Subject: Re: <no subject> aka Replication Stall
>
> the patches have gone into the trunk. The latest patch should be the
> one if you wish to run a patched Solr.
>
> 10 secs readTimeout means that if there is no data coming from the
> other end for 10 secs, then the waiting thread returns throwing an
> exception. It is not the total time taken to read the entire data. At
> least that is what I observed while testing.
>
> BTW, if the timeout occurs it resumes from the point where the failure
> happened. It retries 5 times before giving up.
>
> On Thu, May 7, 2009 at 7:32 PM, Jeff Newburn <jn...@zappos.com> wrote:
>> We have not pushed the fix into production yet. However, I am wondering two
>> things. 1. If the download takes more than 10 seconds (our replication can
>> take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
>> 2 line changes 1 has a large amount. Do we need the latest 2 or just the
>> latest 1?
>>
>> --
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> jnewburn@zappos.com - 702-943-7562
>>
>>
>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>> Reply-To: <so...@lucene.apache.org>
>>> Date: Wed, 6 May 2009 10:05:49 +0530
>>> To: <so...@lucene.apache.org>
>>> Subject: Re: <no subject> aka Replication Stall
>>>
>>> SOLR-1096
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
Re: aka Replication Stall
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
the patches have gone into the trunk. The latest patch should be the
one if you wish to run a patched Solr.
10 secs readTimeout means that if there is no data coming from the
other end for 10 secs, then the waiting thread returns throwing an
exception. It is not the total time taken to read the entire data. At
least that is what I observed while testing.
BTW, if the timeout occurs it resumes from the point where the failure
happened. It retries 5 times before giving up.
On Thu, May 7, 2009 at 7:32 PM, Jeff Newburn <jn...@zappos.com> wrote:
> We have not pushed the fix into production yet. However, I am wondering two
> things. 1. If the download takes more than 10 seconds (our replication can
> take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
> 2 line changes 1 has a large amount. Do we need the latest 2 or just the
> latest 1?
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Wed, 6 May 2009 10:05:49 +0530
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> SOLR-1096
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: aka Replication Stall
Posted by Jeff Newburn <jn...@zappos.com>.
We have not pushed the fix into production yet. However, I am wondering two
things. 1. If the download takes more than 10 seconds (our replication can
take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
2 line changes 1 has a large amount. Do we need the latest 2 or just the
latest 1?
--
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562
> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Wed, 6 May 2009 10:05:49 +0530
> To: <so...@lucene.apache.org>
> Subject: Re: <no subject> aka Replication Stall
>
> SOLR-1096
Re: aka Replication Stall
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
hi,
Jeff , did you use the fix for SOLR-1096 annd you still see that it hangs?
is the client stack trace same then?
We have set it to timeout after 10 secs
--Noble
On Tue, May 5, 2009 at 9:12 PM, Jeff Newburn <jn...@zappos.com> wrote:
> It does still stall. This time I have included the master server thread
> dump.
>
> SOLR
> Solr Amin ()
> Thread Dump
> 6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
> Thread Count: current=27, peak=29, daemon=23
> Full Thread Dump:
> 'http-8080-exec-13' Id=36, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
> a sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
> user time=1010960.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-exec-11' Id=34, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
> d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-10' Id=33, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
> d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
> at sun.mis.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbsractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.ang.Thread.run(Thread.java:619)
> 'http-8080-exec-9' Id=32, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
> at sun.misc.Unsaf.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-8' Id=29, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
> d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-7' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=26, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-5' Id=24, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-4' Id=23, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
> at sun.miscUnsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-3' Id=22, TIMED_WAITING on
> lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-2' Id=21, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-1' Id=19, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> tim=6480.0000ms user time=2900.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
> time=83050.0000ms user time=36360.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
> user time=11940.0000s
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
> TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-5-thread-1' Id=13, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
> 5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-3-thread-1' Id=12, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
> 5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-1-thread-1' Id=11, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
> 6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
> time=5580.0000ms user time=2120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
> time=333280.0000ms user time=107520.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
> total cpu time=4900.0000ms user time=940.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
> time=440.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
> time=11180.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Return to Admin Page
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yo...@lucidimagination.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Mon, 4 May 2009 16:07:50 -0400
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <jn...@zappos.com> wrote:
>>> Absolutely. I have been monitoring but it seems to only happen when a full
>>> reindex occurs and even then intermittently.
>>
>> Any update on this? Have you seen any additional hangs and been able
>> to get a thread dump of the master?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Yonik Seeley <yo...@lucidimagination.com>
>>>> Reply-To: <so...@lucene.apache.org>
>>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>>> To: <so...@lucene.apache.org>
>>>> Subject: Re: <no subject> aka Replication Stall
>>>>
>>>> Next time a hang happens, could you get a thread dump of the master also?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>>>>> I apologize for the delay. The replication stalling out doesn't happen
>>>>> daily. I will paste the thread dump below to try to help. This is on a
>>>>> server that is currently locked on replication for a few hours. Any more
>>>>> information please let me know. There are no errors in the logs either so
>>>>> very strange.
>>>>>
>>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>>> Thread Count: current=73, peak=125, daemon=70
>>>>> Full Thread Dump:
>>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>>> user time=23940.0000ms
>>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at
>>>>>
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
> v
>>>>> a:182)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>>> at
>>>>>
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
> m
>>>>> .java:108)
>>>>> at
>>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>>> at
>>>>>
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
> 1
>>>>> 22)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
> :
>>>>> 808)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
> 4
>>>>> )
>>>>> at
>>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>>> at
>>>>>
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
> v
>>>>> a:224)
>>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
> $
>>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
> i
>>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
> h
>>>>> eduledThreadPoolExecutor.java:205)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>>> time=1800610.0000ms user time=1743610.0000ms
>>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
> 3
>>>>> 74)
>>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:290)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
> .
>>>>> java:630)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
> s
>>>>> patcher.java:436)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
> h
>>>>> er.java:374)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
> r
>>>>> .java:302)
>>>>> at
>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
> :
>>>>> 273)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:235)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
> a
>>>>> va:233)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
> a
>>>>> va:175)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
> 8
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
> 2
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
> a
>>>>> :109)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
> :
>>>>> 879)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
> H
>>>>> ttp11NioProtocol.java:719)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
> :
>>>>> 2080)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>>> user time=0.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at
>>>>>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
> u
>>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>>> time=31410.0000ms user time=24650.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>>> user time=250.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>>> time=764410.0000ms user time=394670.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>>> time=128740.0000ms user time=95550.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
> t
>>>>> ainerBase.java:1579)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>>> time=17960.0000ms user time=11850.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>>> time=261720.0000ms user time=120120.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
> r
>>>>> SocketFactory.java:34)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>>> user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> 'Finalizer' Id=3, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>>> time=16460.0000ms
>>>>> user time=15430.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 'Reference Handler' Id=2, WAITING on
>>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>>> time=3390.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>>> time=1570.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
> 9
>>>>> )
>>>>> at
>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
> l
>>>>> .java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>
>>>>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>>>>> Reply-To: <so...@lucene.apache.org>
>>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>>> To: <so...@lucene.apache.org>
>>>>>> Subject: Re: <no subject>
>>>>>>
>>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>>> snappuller.
>>>>>>
>>>>>> I guess it should be set to some very high value say 1hr for
>>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>>> configurable .
>>>>>>
>>>>>> --Noble
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>>> <sh...@gmail.com> wrote:
>>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>>> we
>>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>>> <yo...@lucidimagination.com>wrote:
>>>>>>>
>>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Yonik
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>>>>> wrote:
>>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>>> nightly
>>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>>>>> will
>>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>>> to
>>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>>> always
>>>>>>>>> the same server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>>
>>>>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>>>>> Poll Interval 00:05:00
>>>>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>>>>> Size: 1.29 GB
>>>>>>>>> Times Replicated Since Startup: 3591
>>>>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>> Files Downloaded: 12 / 163
>>>>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>>> 163
>>>>>>>>> bytes/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>
>>>>>
>>>
>>>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: aka Replication Stall
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
hi Jeff,
Did you observe a hang even after applying SOLR-1096
(https://issues.apache.org/jira/browse/SOLR-1096) ? This change gets
applied only on the slave . So you would get the benefit only if the
slave is updated with a new build
On Tue, May 5, 2009 at 9:12 PM, Jeff Newburn <jn...@zappos.com> wrote:
> It does still stall. This time I have included the master server thread
> dump.
>
> SOLR
> Solr Amin ()
> Thread Dump
> 6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
> Thread Count: current=27, peak=29, daemon=23
> Full Thread Dump:
> 'http-8080-exec-13' Id=36, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
> a sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
> user time=1010960.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-exec-11' Id=34, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
> d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-10' Id=33, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
> d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
> at sun.mis.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbsractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.ang.Thread.run(Thread.java:619)
> 'http-8080-exec-9' Id=32, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
> at sun.misc.Unsaf.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-8' Id=29, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
> d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-7' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=26, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-5' Id=24, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-4' Id=23, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
> at sun.miscUnsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-3' Id=22, TIMED_WAITING on
> lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-2' Id=21, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-1' Id=19, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> tim=6480.0000ms user time=2900.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
> time=83050.0000ms user time=36360.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
> user time=11940.0000s
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
> TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-5-thread-1' Id=13, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
> 5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-3-thread-1' Id=12, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
> 5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-1-thread-1' Id=11, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
> 6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
> time=5580.0000ms user time=2120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
> time=333280.0000ms user time=107520.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
> total cpu time=4900.0000ms user time=940.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
> time=440.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
> time=11180.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Return to Admin Page
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yo...@lucidimagination.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Mon, 4 May 2009 16:07:50 -0400
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <jn...@zappos.com> wrote:
>>> Absolutely. I have been monitoring but it seems to only happen when a full
>>> reindex occurs and even then intermittently.
>>
>> Any update on this? Have you seen any additional hangs and been able
>> to get a thread dump of the master?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Yonik Seeley <yo...@lucidimagination.com>
>>>> Reply-To: <so...@lucene.apache.org>
>>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>>> To: <so...@lucene.apache.org>
>>>> Subject: Re: <no subject> aka Replication Stall
>>>>
>>>> Next time a hang happens, could you get a thread dump of the master also?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>>>>> I apologize for the delay. The replication stalling out doesn't happen
>>>>> daily. I will paste the thread dump below to try to help. This is on a
>>>>> server that is currently locked on replication for a few hours. Any more
>>>>> information please let me know. There are no errors in the logs either so
>>>>> very strange.
>>>>>
>>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>>> Thread Count: current=73, peak=125, daemon=70
>>>>> Full Thread Dump:
>>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>>> user time=23940.0000ms
>>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at
>>>>>
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
> v
>>>>> a:182)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>>> at
>>>>>
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
> m
>>>>> .java:108)
>>>>> at
>>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>>> at
>>>>>
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
> 1
>>>>> 22)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
> :
>>>>> 808)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
> 4
>>>>> )
>>>>> at
>>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>>> at
>>>>>
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
> v
>>>>> a:224)
>>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
> $
>>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
> i
>>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
> h
>>>>> eduledThreadPoolExecutor.java:205)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>>> time=1800610.0000ms user time=1743610.0000ms
>>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
> 3
>>>>> 74)
>>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:290)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
> .
>>>>> java:630)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
> s
>>>>> patcher.java:436)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
> h
>>>>> er.java:374)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
> r
>>>>> .java:302)
>>>>> at
>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
> :
>>>>> 273)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:235)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
> a
>>>>> va:233)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
> a
>>>>> va:175)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
> 8
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
> 2
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
> a
>>>>> :109)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
> :
>>>>> 879)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
> H
>>>>> ttp11NioProtocol.java:719)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
> :
>>>>> 2080)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>>> user time=0.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at
>>>>>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
> u
>>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>>> time=31410.0000ms user time=24650.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>>> user time=250.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>>> time=764410.0000ms user time=394670.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>>> time=128740.0000ms user time=95550.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
> t
>>>>> ainerBase.java:1579)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>>> time=17960.0000ms user time=11850.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>>> time=261720.0000ms user time=120120.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
> r
>>>>> SocketFactory.java:34)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>>> user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> 'Finalizer' Id=3, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>>> time=16460.0000ms
>>>>> user time=15430.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 'Reference Handler' Id=2, WAITING on
>>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>>> time=3390.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>>> time=1570.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
> 9
>>>>> )
>>>>> at
>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
> l
>>>>> .java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>
>>>>>
>>>>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>>>>> Reply-To: <so...@lucene.apache.org>
>>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>>> To: <so...@lucene.apache.org>
>>>>>> Subject: Re: <no subject>
>>>>>>
>>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>>> snappuller.
>>>>>>
>>>>>> I guess it should be set to some very high value say 1hr for
>>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>>> configurable .
>>>>>>
>>>>>> --Noble
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>>> <sh...@gmail.com> wrote:
>>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>>> we
>>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>>> <yo...@lucidimagination.com>wrote:
>>>>>>>
>>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Yonik
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>>>>> wrote:
>>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>>> nightly
>>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>>>>> will
>>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>>> to
>>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>>> always
>>>>>>>>> the same server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>>
>>>>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>>>>> Poll Interval 00:05:00
>>>>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>>>>> Size: 1.29 GB
>>>>>>>>> Times Replicated Since Startup: 3591
>>>>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>> Files Downloaded: 12 / 163
>>>>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>>> 163
>>>>>>>>> bytes/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>
>>>>>
>>>
>>>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: aka Replication Stall
Posted by Jeff Newburn <jn...@zappos.com>.
It does still stall. This time I have included the master server thread
dump.
SOLR
Solr Amin ()
Thread Dump
6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
Thread Count: current=27, peak=29, daemon=23
Full Thread Dump:
'http-8080-exec-13' Id=36, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
a sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
user time=1010960.0000ms
at sun.management.ThreadImpl.getThreadInfo0(Native Method)
at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
74)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
t
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
java:630)
at
org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
patcher.java:436)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:374)
at
org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
.java:302)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
273)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
at
org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
879)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
ttp11NioProtocol.java:719)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
2080)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
at java.lang.Thread.run(Thread.jav:619)
'http-8080-exec-11' Id=34, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-10' Id=33, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
at sun.mis.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbsractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.ang.Thread.run(Thread.java:619)
'http-8080-exec-9' Id=32, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
at sun.misc.Unsaf.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-8' Id=29, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-7' Id=28, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-6' Id=26, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-5' Id=24, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-4' Id=23, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
at sun.miscUnsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-3' Id=22, TIMED_WAITING on
lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-2' Id=21, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-1' Id=19, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
tim=6480.0000ms user time=2900.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
user time=0.0000ms
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
time=83050.0000ms user time=36360.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.jav:619)
'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
user time=11940.0000s
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
ainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
'pool-5-thread-1' Id=13, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'pool-3-thread-1' Id=12, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'pool-1-thread-1' Id=11, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
time=5580.0000ms user time=2120.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
time=333280.0000ms user time=107520.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
total cpu time=4900.0000ms user time=940.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
'Reference Handler' Id=2, WAITING on
lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
time=440.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
time=11180.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Return to Admin Page
--
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562
> From: Yonik Seeley <yo...@lucidimagination.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Mon, 4 May 2009 16:07:50 -0400
> To: <so...@lucene.apache.org>
> Subject: Re: <no subject> aka Replication Stall
>
> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <jn...@zappos.com> wrote:
>> Absolutely. I have been monitoring but it seems to only happen when a full
>> reindex occurs and even then intermittently.
>
> Any update on this? Have you seen any additional hangs and been able
> to get a thread dump of the master?
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> jnewburn@zappos.com - 702-943-7562
>>
>>
>>> From: Yonik Seeley <yo...@lucidimagination.com>
>>> Reply-To: <so...@lucene.apache.org>
>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>> To: <so...@lucene.apache.org>
>>> Subject: Re: <no subject> aka Replication Stall
>>>
>>> Next time a hang happens, could you get a thread dump of the master also?
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>>>> I apologize for the delay. The replication stalling out doesn't happen
>>>> daily. I will paste the thread dump below to try to help. This is on a
>>>> server that is currently locked on replication for a few hours. Any more
>>>> information please let me know. There are no errors in the logs either so
>>>> very strange.
>>>>
>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>> Thread Count: current=73, peak=125, daemon=70
>>>> Full Thread Dump:
>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>> user time=23940.0000ms
>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>> at
>>>>
org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
v
>>>> a:182)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>> at
>>>>
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
m
>>>> .java:108)
>>>> at
>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>> at
>>>>
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
1
>>>> 22)
>>>> at
>>>>
org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
:
>>>> 808)
>>>> at
>>>>
org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
4
>>>> )
>>>> at
>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>> at
>>>>
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
v
>>>> a:224)
>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
$
>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
i
>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
h
>>>> eduledThreadPoolExecutor.java:205)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
a
>>>> va:885)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 07)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>> time=1800610.0000ms user time=1743610.0000ms
>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>> at
>>>>
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
3
>>>> 74)
>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
n
>>>> FilterChain.java:290)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
h
>>>> ain.java:206)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
.
>>>> java:630)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
s
>>>> patcher.java:436)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
h
>>>> er.java:374)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
r
>>>> .java:302)
>>>> at
>>>>
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
:
>>>> 273)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
n
>>>> FilterChain.java:235)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
h
>>>> ain.java:206)
>>>> at
>>>>
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
a
>>>> va:233)
>>>> at
>>>>
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
a
>>>> va:175)
>>>> at
>>>>
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
8
>>>> )
>>>> at
>>>>
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
2
>>>> )
>>>> at
>>>>
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
a
>>>> :109)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>> at
>>>>
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
:
>>>> 879)
>>>> at
>>>>
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
H
>>>> ttp11NioProtocol.java:719)
>>>> at
>>>>
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
:
>>>> 2080)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
a
>>>> va:885)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 07)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>> user time=0.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>> at
>>>>
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
u
>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>> time=31410.0000ms user time=24650.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>> user time=250.0000ms
>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>> time=764410.0000ms user time=394670.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>> time=128740.0000ms user time=95550.0000ms
>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
t
>>>> ainerBase.java:1579)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>> time=17960.0000ms user time=11850.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at
>>>>
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
e
>>>> lector.java:305)
>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>> time=261720.0000ms user time=120120.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at
>>>>
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
e
>>>> lector.java:305)
>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
r
>>>> SocketFactory.java:34)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t
>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>> user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t
>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t
>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> 'Finalizer' Id=3, WAITING on
>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>> time=16460.0000ms
>>>> user time=15430.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>> 'Reference Handler' Id=2, WAITING on
>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>> time=3390.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>> time=1570.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
9
>>>> )
>>>> at
>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
l
>>>> .java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>
>>>> --
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> jnewburn@zappos.com - 702-943-7562
>>>>
>>>>
>>>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>>>> Reply-To: <so...@lucene.apache.org>
>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>> To: <so...@lucene.apache.org>
>>>>> Subject: Re: <no subject>
>>>>>
>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>> snappuller.
>>>>>
>>>>> I guess it should be set to some very high value say 1hr for
>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>> configurable .
>>>>>
>>>>> --Noble
>>>>>
>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>> <sh...@gmail.com> wrote:
>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>> we
>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>> <yo...@lucidimagination.com>wrote:
>>>>>>
>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>
>>>>>>>
>>>>>>> -Yonik
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>>>> wrote:
>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>> nightly
>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>>>> will
>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>> to
>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>> always
>>>>>>>> the same server.
>>>>>>>>
>>>>>>>>
>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>
>>>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>>>> Poll Interval 00:05:00
>>>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>>>> Size: 1.29 GB
>>>>>>>> Times Replicated Since Startup: 3591
>>>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>> Files Downloaded: 12 / 163
>>>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>> 163
>>>>>>>> bytes/s
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jeff Newburn
>>>>>>>> Software Engineer, Zappos.com
>>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>
>>>>
>>
>>
Re: aka Replication Stall
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <jn...@zappos.com> wrote:
> Absolutely. I have been monitoring but it seems to only happen when a full
> reindex occurs and even then intermittently.
Any update on this? Have you seen any additional hangs and been able
to get a thread dump of the master?
-Yonik
http://www.lucidimagination.com
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yo...@lucidimagination.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> Next time a hang happens, could you get a thread dump of the master also?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>>> I apologize for the delay. The replication stalling out doesn't happen
>>> daily. I will paste the thread dump below to try to help. This is on a
>>> server that is currently locked on replication for a few hours. Any more
>>> information please let me know. There are no errors in the logs either so
>>> very strange.
>>>
>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>> Thread Count: current=73, peak=125, daemon=70
>>> Full Thread Dump:
>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>> user time=23940.0000ms
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> at
>>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>>> a:182)
>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>> at
>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>>> .java:108)
>>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>> at
>>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>>> 22)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>>> 808)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>>> )
>>> at
>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>>> a:224)
>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>>> odic(ScheduledThreadPoolExecutor.java:181)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:205)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'pool-16-thread-1' Id=237, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>> time=1800610.0000ms user time=1743610.0000ms
>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>>> 74)
>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:290)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>>> java:630)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>>> patcher.java:436)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>>> er.java:374)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>>> .java:302)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>> 273)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:235)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>> va:233)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>> va:175)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>> )
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>> )
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>> :109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>> at
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>> 879)
>>> at
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>> ttp11NioProtocol.java:719)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>> 2080)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-5' Id=152, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-4' Id=151, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-3' Id=150, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-2' Id=149, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-1' Id=148, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>> user time=0.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>> time=31410.0000ms user time=24650.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>> user time=250.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>> time=764410.0000ms user time=394670.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>> time=128740.0000ms user time=95550.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>>> ainerBase.java:1579)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>> time=17960.0000ms user time=11850.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>> time=261720.0000ms user time=120120.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>>> SocketFactory.java:34)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> 'Finalizer' Id=3, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>>> user time=15430.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> 'Reference Handler' Id=2, WAITING on
>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>> time=3390.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.Object.wait(Object.java:485)
>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>> time=1570.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>> )
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>> .java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>
>>> --
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>>> Reply-To: <so...@lucene.apache.org>
>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>> To: <so...@lucene.apache.org>
>>>> Subject: Re: <no subject>
>>>>
>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>> snappuller.
>>>>
>>>> I guess it should be set to some very high value say 1hr for
>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>> configurable .
>>>>
>>>> --Noble
>>>>
>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>> <sh...@gmail.com> wrote:
>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>> we
>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>
>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>> <yo...@lucidimagination.com>wrote:
>>>>>
>>>>>> So this is only one slave that hangs up and not the master?
>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>>> wrote:
>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>> nightly
>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>>> will
>>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>> always
>>>>>>> the same server.
>>>>>>>
>>>>>>>
>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>
>>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>>> Poll Interval 00:05:00
>>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>>> Size: 1.29 GB
>>>>>>> Times Replicated Since Startup: 3591
>>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>> Files Downloaded: 12 / 163
>>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>>> bytes/s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Newburn
>>>>>>> Software Engineer, Zappos.com
>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shalin Shekhar Mangar.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>
>>>
>
>
Re: aka Replication Stall
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
I have raised an issue and have given a patch too
https://issues.apache.org/jira/browse/SOLR-1096
On Fri, Apr 3, 2009 at 2:00 AM, Jeff Newburn <jn...@zappos.com> wrote:
> Absolutely. I have been monitoring but it seems to only happen when a full
> reindex occurs and even then intermittently.
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
>> From: Yonik Seeley <yo...@lucidimagination.com>
>> Reply-To: <so...@lucene.apache.org>
>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>> To: <so...@lucene.apache.org>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> Next time a hang happens, could you get a thread dump of the master also?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>>> I apologize for the delay. The replication stalling out doesn't happen
>>> daily. I will paste the thread dump below to try to help. This is on a
>>> server that is currently locked on replication for a few hours. Any more
>>> information please let me know. There are no errors in the logs either so
>>> very strange.
>>>
>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>> Thread Count: current=73, peak=125, daemon=70
>>> Full Thread Dump:
>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>> user time=23940.0000ms
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> at
>>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>>> a:182)
>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>> at
>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>>> .java:108)
>>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>> at
>>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>>> 22)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>>> 808)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>>> )
>>> at
>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>>> a:224)
>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>>> odic(ScheduledThreadPoolExecutor.java:181)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:205)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'pool-16-thread-1' Id=237, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>> time=1800610.0000ms user time=1743610.0000ms
>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>>> 74)
>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:290)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>>> java:630)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>>> patcher.java:436)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>>> er.java:374)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>>> .java:302)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>> 273)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:235)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>> va:233)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>> va:175)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>> )
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>> )
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>> :109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>> at
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>> 879)
>>> at
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>> ttp11NioProtocol.java:719)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>> 2080)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-5' Id=152, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-4' Id=151, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-3' Id=150, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-2' Id=149, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-1' Id=148, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>> user time=0.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>> time=31410.0000ms user time=24650.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>> user time=250.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>> time=764410.0000ms user time=394670.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>> time=128740.0000ms user time=95550.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>>> ainerBase.java:1579)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>> time=17960.0000ms user time=11850.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>> time=261720.0000ms user time=120120.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>>> SocketFactory.java:34)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> 'Finalizer' Id=3, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>>> user time=15430.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> 'Reference Handler' Id=2, WAITING on
>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>> time=3390.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.Object.wait(Object.java:485)
>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>> time=1570.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>> )
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>> .java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>
>>> --
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> jnewburn@zappos.com - 702-943-7562
>>>
>>>
>>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>>> Reply-To: <so...@lucene.apache.org>
>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>> To: <so...@lucene.apache.org>
>>>> Subject: Re: <no subject>
>>>>
>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>> snappuller.
>>>>
>>>> I guess it should be set to some very high value say 1hr for
>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>> configurable .
>>>>
>>>> --Noble
>>>>
>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>> <sh...@gmail.com> wrote:
>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>> we
>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>
>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>> <yo...@lucidimagination.com>wrote:
>>>>>
>>>>>> So this is only one slave that hangs up and not the master?
>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>>> wrote:
>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>> nightly
>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>>> will
>>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>> always
>>>>>>> the same server.
>>>>>>>
>>>>>>>
>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>
>>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>>> Poll Interval 00:05:00
>>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>>> Size: 1.29 GB
>>>>>>> Times Replicated Since Startup: 3591
>>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>> Files Downloaded: 12 / 163
>>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>>> bytes/s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Newburn
>>>>>>> Software Engineer, Zappos.com
>>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shalin Shekhar Mangar.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>
>>>
>
>
--
--Noble Paul
Re: aka Replication Stall
Posted by Jeff Newburn <jn...@zappos.com>.
Absolutely. I have been monitoring but it seems to only happen when a full
reindex occurs and even then intermittently.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562
> From: Yonik Seeley <yo...@lucidimagination.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Thu, 2 Apr 2009 15:00:28 -0400
> To: <so...@lucene.apache.org>
> Subject: Re: <no subject> aka Replication Stall
>
> Next time a hang happens, could you get a thread dump of the master also?
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <jn...@zappos.com> wrote:
>> I apologize for the delay. The replication stalling out doesn't happen
>> daily. I will paste the thread dump below to try to help. This is on a
>> server that is currently locked on replication for a few hours. Any more
>> information please let me know. There are no errors in the logs either so
>> very strange.
>>
>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>> Thread Count: current=73, peak=125, daemon=70
>> Full Thread Dump:
>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>> user time=23940.0000ms
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> at
>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>> a:182)
>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>> at
>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>> .java:108)
>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>> at
>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>> 22)
>> at
>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>> 808)
>> at
>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>> )
>> at
>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>> at
>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>> a:224)
>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>> 101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>> odic(ScheduledThreadPoolExecutor.java:181)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>> eduledThreadPoolExecutor.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>> va:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 07)
>> at java.lang.Thread.run(Thread.java:619)
>> 'pool-16-thread-1' Id=237, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>> time=1800610.0000ms user time=1743610.0000ms
>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>> 74)
>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>> FilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>> ain.java:206)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>> java:630)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>> patcher.java:436)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>> er.java:374)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>> .java:302)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>> 273)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>> FilterChain.java:235)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>> ain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>> va:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>> va:175)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>> )
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>> )
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>> :109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>> at
>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>> 879)
>> at
>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>> ttp11NioProtocol.java:719)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>> 2080)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>> va:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 07)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-5' Id=152, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-4' Id=151, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-3' Id=150, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-2' Id=149, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-1' Id=148, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>> user time=0.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>> at
>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>> time=31410.0000ms user time=24650.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>> user time=250.0000ms
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>> time=764410.0000ms user time=394670.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>> time=128740.0000ms user time=95550.0000ms
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>> ainerBase.java:1579)
>> at java.lang.Thread.run(Thread.java:619)
>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>> time=17960.0000ms user time=11850.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>> lector.java:305)
>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>> time=261720.0000ms user time=120120.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>> lector.java:305)
>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>> SocketFactory.java:34)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> 'Finalizer' Id=3, WAITING on
>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>> user time=15430.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> 'Reference Handler' Id=2, WAITING on
>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>> time=3390.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:485)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>> time=1570.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>> )
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>> .java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>
>> --
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> jnewburn@zappos.com - 702-943-7562
>>
>>
>>> From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
>>> Reply-To: <so...@lucene.apache.org>
>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>> To: <so...@lucene.apache.org>
>>> Subject: Re: <no subject>
>>>
>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>> snappuller.
>>>
>>> I guess it should be set to some very high value say 1hr for
>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>> configurable .
>>>
>>> --Noble
>>>
>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>> <sh...@gmail.com> wrote:
>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>> we
>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>
>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>> <yo...@lucidimagination.com>wrote:
>>>>
>>>>> So this is only one slave that hangs up and not the master?
>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>
>>>>>
>>>>> -Yonik
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jn...@zappos.com>
>>>>> wrote:
>>>>>> We are having an intermittent problem with replication. We reindex
>>>>> nightly
>>>>>> which usually means there are 2 commits during replication then a final
>>>>>> commit/optimize at the end. For some reason the replication will hang
>>>>>> occasionally with the following screenshot. This is frustrating as it
>>>>> will
>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>> always
>>>>>> the same server.
>>>>>>
>>>>>>
>>>>>> In case the screen shot doesn’t come through:
>>>>>>
>>>>>> Master http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>> Latest Index Version:1233423827699, Generation: 6237
>>>>>> Replicatable Index Version:0, Generation: 0
>>>>>> Poll Interval 00:05:00
>>>>>> Local Index Index Version: 1233423827684, Generation: 6222
>>>>>> Location: /opt/solr-data/zeta-main/index
>>>>>> Size: 1.29 GB
>>>>>> Times Replicated Since Startup: 3591
>>>>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>> Config Files Replicated: [synonyms.txt]
>>>>>> Times Config Files Replicated Since Startup: 4
>>>>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>> Files Downloaded: 12 / 163
>>>>>> Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>> bytes/s
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jeff Newburn
>>>>>> Software Engineer, Zappos.com
>>>>>> jnewburn@zappos.com - 702-943-7562
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>
>>