You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/14 22:00:15 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8316 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8316/
Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3485, name=commitScheduler-2547-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3537, name=commitScheduler-2568-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3487, name=commitScheduler-2550-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3481, name=commitScheduler-2541-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3533, name=commitScheduler-2561-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3538, name=commitScheduler-2556-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3534, name=commitScheduler-2563-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3535, name=commitScheduler-2565-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3480, name=commitScheduler-2540-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3483, name=commitScheduler-2544-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3485, name=commitScheduler-2547-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3537, name=commitScheduler-2568-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3487, name=commitScheduler-2550-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3481, name=commitScheduler-2541-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3533, name=commitScheduler-2561-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3538, name=commitScheduler-2556-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3534, name=commitScheduler-2563-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3535, name=commitScheduler-2565-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3480, name=commitScheduler-2540-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3483, name=commitScheduler-2544-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([7FB3EC62B32046C7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3485, name=commitScheduler-2547-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3537, name=commitScheduler-2568-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3487, name=commitScheduler-2550-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3481, name=commitScheduler-2541-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3533, name=commitScheduler-2561-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3538, name=commitScheduler-2556-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3534, name=commitScheduler-2563-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3535, name=commitScheduler-2565-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3480, name=commitScheduler-2540-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3483, name=commitScheduler-2544-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3485, name=commitScheduler-2547-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3537, name=commitScheduler-2568-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3487, name=commitScheduler-2550-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3481, name=commitScheduler-2541-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3533, name=commitScheduler-2561-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3538, name=commitScheduler-2556-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3534, name=commitScheduler-2563-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3535, name=commitScheduler-2565-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3480, name=commitScheduler-2540-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3483, name=commitScheduler-2544-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([7FB3EC62B32046C7]:0)




Build Log:
[...truncated 10335 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.02s J0 | OpenCloseCoreStressTest.test1HourOld
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly)
   [junit4]   2> 925723 T3452 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsNew
   [junit4]   2> 925732 T3452 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 925735 T3452 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59148
   [junit4]   2> 925735 T3452 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 925736 T3452 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 925736 T3452 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 925736 T3452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 925748 T3452 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 925790 T3452 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 925791 T3452 oasc.CoreContainer.<init> New CoreContainer 29460342
   [junit4]   2> 925791 T3452 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 925792 T3452 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 925792 T3452 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 925793 T3452 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 925793 T3452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 925793 T3452 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 925794 T3452 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 925794 T3452 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 925794 T3452 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 925794 T3452 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 925796 T3452 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 925796 T3452 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 925796 T3452 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 925798 T3452 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 925799 T3452 oasc.CorePropertiesLocator.discoverUnder Found core 00001_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/
   [junit4]   2> 925800 T3452 oasc.CorePropertiesLocator.discoverUnder Found core 00000_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/
   [junit4]   2> 925800 T3452 oasc.CorePropertiesLocator.discoverUnder Found core 00004_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/
   [junit4]   2> 925801 T3452 oasc.CorePropertiesLocator.discoverUnder Found core 00002_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/
   [junit4]   2> 925801 T3452 oasc.CorePropertiesLocator.discoverUnder Found core 00003_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/
   [junit4]   2> 925801 T3452 oasc.CorePropertiesLocator.discover Found 5 core definitions
   [junit4]   2> 925802 T3452 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 925802 T3452 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 925806 T3452 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 925806 T3452 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 925808 T3465 oasc.OneIndexer.run Starting indexing thread: 3465
   [junit4]   2> 925808 T3467 oasc.OneIndexer.run Starting indexing thread: 3467
   [junit4]   2> 925808 T3466 oasc.OneIndexer.run Starting indexing thread: 3466
   [junit4]   2> 925808 T3464 oasc.OneIndexer.run Starting indexing thread: 3464
   [junit4]   2> 925811 T3463 oasc.OneIndexer.run Starting indexing thread: 3463
   [junit4]   2> 925814 T3469 oasc.OneQuery.run Starting query thread: 3469
   [junit4]   2> 925814 T3468 oasc.OneQuery.run Starting query thread: 3468
   [junit4]   2> 925814 T3470 oasc.OneQuery.run Starting query thread: 3470
   [junit4]   2> 925814 T3472 oasc.OneQuery.run Starting query thread: 3472
   [junit4]   2> 925814 T3471 oasc.OneQuery.run Starting query thread: 3471
   [junit4]   2> 925814 T3458 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 925815 T3461 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 925814 T3460 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 925814 T3459 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 925815 T3460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 925815 T3461 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 925815 T3458 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 925816 T3459 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 925891 T3459 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 925893 T3460 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 925912 T3458 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 925921 T3461 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 925985 T3459 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 925986 T3459 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 925990 T3460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 925991 T3460 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 926013 T3459 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 926015 T3460 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 926041 T3458 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 926042 T3460 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 926042 T3458 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 926043 T3460 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 926044 T3460 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 926044 T3460 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 926045 T3459 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 926045 T3461 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 926046 T3461 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 926046 T3459 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 926046 T3459 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 926047 T3459 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 926047 T3458 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 926046 T3460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 926058 T3459 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 926058 T3460 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 926059 T3459 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 926059 T3459 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 926060 T3459 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 926060 T3459 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8394773822925504]
   [junit4]   2> 926059 T3460 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 926063 T3460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 926063 T3460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8394773822925504]
   [junit4]   2> 926064 T3459 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@497e02 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c89adf),segFN=segments_1,generation=1}
   [junit4]   2> 926064 T3459 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926064 T3460 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1fb0061 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e3ea6),segFN=segments_1,generation=1}
   [junit4]   2> 926065 T3460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926065 T3461 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 926065 T3459 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 926066 T3460 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 926066 T3459 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 926066 T3460 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 926066 T3459 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 926067 T3460 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 926067 T3459 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 926068 T3460 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 926070 T3460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 926070 T3459 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 926074 T3459 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 926074 T3460 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 926075 T3459 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 926075 T3460 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 926076 T3459 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=70.478515625, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=3.2188323181542033, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.127773303657618
   [junit4]   2> 926076 T3458 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 926076 T3460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=70.478515625, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=3.2188323181542033, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.127773303657618
   [junit4]   2> 926077 T3459 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@497e02 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c89adf),segFN=segments_1,generation=1}
   [junit4]   2> 926077 T3459 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926077 T3460 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1fb0061 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e3ea6),segFN=segments_1,generation=1}
   [junit4]   2> 926078 T3460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926078 T3460 oass.SolrIndexSearcher.<init> Opening Searcher@12043b1 main
   [junit4]   2> 926078 T3459 oass.SolrIndexSearcher.<init> Opening Searcher@18c459a main
   [junit4]   2> 926089 T3479 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@12043b1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 926078 T3458 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 926090 T3478 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@18c459a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 926090 T3459 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 926089 T3460 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 926090 T3459 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 926090 T3458 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 926091 T3458 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 926091 T3460 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 926092 T3458 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 926092 T3458 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 926093 T3458 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 926093 T3458 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 926094 T3458 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8394773822925504]
   [junit4]   2> 926094 T3458 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4156fd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16ad918),segFN=segments_1,generation=1}
   [junit4]   2> 926095 T3458 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926096 T3458 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 926096 T3458 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 926096 T3458 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 926097 T3458 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> ASYNC  NEW_CORE C6820 name=00002_core org.apache.solr.core.SolrCore@8c56e1
   [junit4]   2> 926097 T3475 C6820 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 926098 T3458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> ASYNC  NEW_CORE C6821 name=00001_core org.apache.solr.core.SolrCore@1d050f7
   [junit4]   2> 926098 T3460 C6821 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 1
   [junit4]   2> 926098 T3475 C6820 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926098 T3461 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 926098 T3459 C6820 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 2
   [junit4]   2> 926100 T3458 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 926100 T3458 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 926100 T3461 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 926100 T3458 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=70.478515625, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=3.2188323181542033, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.127773303657618
   [junit4]   2> 926101 T3461 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 926102 T3458 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4156fd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16ad918),segFN=segments_1,generation=1}
   [junit4]   2> 926102 T3458 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926102 T3461 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 926102 T3475 C6820 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 2
   [junit4]   2> 926102 T3458 oass.SolrIndexSearcher.<init> Opening Searcher@8a0b70 main
   [junit4]   2> 926103 T3461 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 926104 T3461 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 926104 T3482 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@8a0b70 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 926104 T3458 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 926104 T3461 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 926104 T3458 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 926105 T3461 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 926106 T3461 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8394773822925504]
   [junit4]   2> 926114 T3461 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@27d9d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7aaca7),segFN=segments_1,generation=1}
   [junit4]   2> 926115 T3461 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926116 T3461 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 926117 T3461 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 926117 T3461 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 926118 T3461 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> ASYNC  NEW_CORE C6822 name=00003_core org.apache.solr.core.SolrCore@db17e1
   [junit4]   2> 926118 T3474 C6822 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926123 T3459 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 926121 T3473 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 1
   [junit4]   2> 926121 T3476 C6822 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926120 T3461 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 926120 T3475 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 3
   [junit4]   2> 926120 T3458 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 2
   [junit4]   2> 926126 T3473 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 926126 T3459 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 926127 T3461 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 926127 T3459 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 926127 T3461 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 926128 T3461 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=70.478515625, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=3.2188323181542033, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.127773303657618
   [junit4]   2> 926128 T3475 C6822 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 0
   [junit4]   2> 926129 T3461 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@27d9d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7aaca7),segFN=segments_1,generation=1}
   [junit4]   2> 926129 T3461 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926130 T3461 oass.SolrIndexSearcher.<init> Opening Searcher@155ac3 main
   [junit4]   2> 926130 T3461 oasc.SolrCores.putTransientCore Opening transient core 00000_core
   [junit4]   2> 926130 T3484 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@155ac3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C6823 name=00002_core org.apache.solr.core.SolrCore@8c56e1
   [junit4]   2> 926131 T3458 C6823 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 1
   [junit4]   2> 926131 T3461 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> ASYNC  NEW_CORE C6824 name=00000_core org.apache.solr.core.SolrCore@106a70
   [junit4]   2> 926151 T3474 C6824 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 2
   [junit4]   2> 926152 T3477 C6824 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 926153 T3461 C6824 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 2
   [junit4]   2> 926154 T3473 C6824 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6825 name=00001_core org.apache.solr.core.SolrCore@1d050f7
   [junit4]   2> 926156 T3475 C6825 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6826 name=00002_core org.apache.solr.core.SolrCore@8c56e1
   [junit4]   2> 926157 T3477 C6826 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6827 name=00003_core org.apache.solr.core.SolrCore@db17e1
   [junit4]   2> 926159 T3473 C6827 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 1
   [junit4]   2> 926160 T3458 C6826 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 926162 T3474 C6827 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 926163 T3475 C6827 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> 926166 T3477 C6825 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 1
   [junit4]   2> 926166 T3473 C6826 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 926182 T3459 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> ASYNC  NEW_CORE C6828 name=00000_core org.apache.solr.core.SolrCore@106a70
   [junit4]   2> 926208 T3477 C6828 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926217 T3459 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 926223 T3459 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> ASYNC  NEW_CORE C6829 name=00001_core org.apache.solr.core.SolrCore@1d050f7
   [junit4]   2> 926226 T3473 C6829 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C6830 name=00002_core org.apache.solr.core.SolrCore@8c56e1
   [junit4]   2> 926226 T3477 C6830 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926227 T3459 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 926250 T3459 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 926251 T3459 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 926252 T3459 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 926252 T3459 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 926252 T3459 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 926253 T3459 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 926253 T3459 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 926253 T3459 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 926254 T3459 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=48, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2469081874809529]
   [junit4]   2> 926254 T3459 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1f3701c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c9ca33),segFN=segments_1,generation=1}
   [junit4]   2> 926255 T3459 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> ASYNC  NEW_CORE C6831 name=00003_core org.apache.solr.core.SolrCore@db17e1
   [junit4]   2> 926255 T3473 C6831 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926256 T3459 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 926256 T3459 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 926256 T3459 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 926257 T3459 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 926258 T3459 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 926259 T3459 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 926259 T3459 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 926260 T3459 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=8.1748046875, floorSegmentMB=0.2431640625, forceMergeDeletesPctAllowed=15.218852107676422, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25922010131166107
   [junit4]   2> 926260 T3459 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1f3701c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c9ca33),segFN=segments_1,generation=1}
   [junit4]   2> 926260 T3459 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 926261 T3459 oass.SolrIndexSearcher.<init> Opening Searcher@32c846 main
   [junit4]   2> 926261 T3459 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 926261 T3486 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@32c846 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 926262 T3459 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> ASYNC  NEW_CORE C6832 name=00004_core org.apache.solr.core.SolrCore@f262b8
   [junit4]   2> 926266 T3459 C6832 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> 926267 T3475 C6832 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C6833 name=00004_core org.apache.solr.core.SolrCore@f262b8
   [junit4]   2> 926279 T3476 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 12
   [junit4]   2> 926279 T3458 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 12
   [junit4]   2> 926279 T3477 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 0
   [junit4]   2> 926279 T3461 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 12
   [junit4]   2> 926279 T3474 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C6834 name=00002_core org.apache.solr.core.SolrCore@8c56e1
   [junit4]   2> 926281 T3473 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C6835 name=00003_core org.apache.solr.core.SolrCore@db17e1
   [junit4]   2> 926281 T3459 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C6836 name=00000_core org.apache.solr.core.SolrCore@106a70
   [junit4]   2> 926282 T3475 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 926282 T3476 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6837 name=00001_core org.apache.solr.core.SolrCore@1d050f7
   [junit4]   2> 926282 T3458 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 0
   [junit4]   2> 926283 T3477 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 0
   [junit4]   2> 926284 T3461 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 926285 T3474 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> 926285 T3459 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 926285 T3473 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> 926286 T3475 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 926286 T3476 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 0
   [junit4]   2> 926287 T3458 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 926288 T3477 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 926288 T3461 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 926289 T3474 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 1
   [junit4]   2> 926289 T3477 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 926290 T3473 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 926290 T3475 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 0
   [junit4]   2> 926291 T3476 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> 926292 T3458 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 926292 T3461 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 926293 T3459 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 926293 T3474 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 0
   [junit4]   2> 926294 T3477 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 926294 T3473 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 0
   [junit4]   2> 926295 T3475 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 1
   [junit4]   2> 926296 T3476 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 926296 T3475 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 926297 T3461 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 926297 T3459 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 1
   [junit4]   2> 926297 T3474 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 0
   [junit4]   2> 926298 T3477 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 1
   [junit4]   2> 926299 T3474 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 926300 T3458 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 926300 T3476 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 926300 T3475 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 0
   [junit4]   2> 926301 T3461 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 926302 T3459 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 926302 T3477 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 0
   [junit4]   2> 926302 T3473 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 0
   [junit4]   2> 926303 T3458 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 0
   [junit4]   2> 926304 T3474 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 2
   [junit4]   2> 926305 T3476 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 926306 T3476 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 0
   [junit4]   2> 926307 T3474 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 2
   [junit4]   2> 926307 T3477 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 926307 T3475 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 926308 T3477 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 0
   [junit4]   2> 926309 T3458 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 926310 T3477 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 926311 T3461 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 926311 T3459 C6836 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926311 T3477 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 926311 T3458 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 926313 T3476 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 1
   [junit4]   2> 926313 T3475 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 2
   [junit4]   2> 926313 T3461 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 0
   [junit4]   2> 926318 T3461 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 3
   [junit4]   2> 926318 T3458 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 926318 T3474 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 926318 T3473 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 926318 T3459 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 926322 T3475 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 926322 T3476 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 926323 T3474 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 926322 T3477 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 926323 T3473 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 0
   [junit4]   2> 926324 T3458 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 0
   [junit4]   2> 926325 T3473 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 926326 T3458 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 926327 T3461 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 926327 T3474 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 926327 T3459 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 2
   [junit4]   2> 926329 T3477 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 926329 T3474 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 0
   [junit4]   2> 926330 T3473 C6836 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 926330 T3458 C6835 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926331 T3476 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 1
   [junit4]   2> 926331 T3458 C6837 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 926331 T3473 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 926332 T3476 C6833 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 926332 T3458 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 926333 T3473 C6837 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 926333 T3476 C6834 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 0
   [junit4]   2> 926334 T3477 C6835 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 0
   [junit4]   2> 926335 T3475 C6836 oas

[...truncated too long message...]

t]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=3481, name=commitScheduler-2541-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=3533, name=commitScheduler-2561-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=3538, name=commitScheduler-2556-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=3534, name=commitScheduler-2563-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=3535, name=commitScheduler-2565-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=3480, name=commitScheduler-2540-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=3483, name=commitScheduler-2544-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7FB3EC62B32046C7]:0)
   [junit4] Completed on J0 in 54.26s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 549 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 611 ignored (6 assumptions)

Total time: 38 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8319 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8319/
Java: 32bit/jdk1.7.0_45 -client -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)




Build Log:
[...truncated 10376 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 978831 T3301 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 978839 T3301 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 978862 T3301 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56396
   [junit4]   2> 978863 T3301 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 978863 T3301 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 978864 T3301 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 978864 T3301 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 978883 T3301 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 978943 T3301 oasc.CoreContainer.<init> New CoreContainer 11731529
   [junit4]   2> 978943 T3301 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 978947 T3301 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 978947 T3301 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 978948 T3301 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 978949 T3301 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 978949 T3301 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 978983 T3311 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 978983 T3311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 979016 T3311 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979060 T3311 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979061 T3311 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979065 T3311 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 979079 T3311 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979080 T3311 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979080 T3311 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 979081 T3311 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 979081 T3311 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 979081 T3311 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 979082 T3311 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979082 T3311 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 979082 T3311 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979083 T3311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68b475 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b3b2e2),segFN=segments_1,generation=1}
   [junit4]   2> 979083 T3311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979084 T3311 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979084 T3311 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979085 T3311 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979085 T3311 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979087 T3311 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979090 T3311 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979091 T3311 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979091 T3311 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979092 T3311 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68b475 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b3b2e2),segFN=segments_1,generation=1}
   [junit4]   2> 979092 T3311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979093 T3311 oass.SolrIndexSearcher.<init> Opening Searcher@11a8402 main
   [junit4]   2> 979093 T3312 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@11a8402 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979093 T3311 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 979094 T3301 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 979094 T3301 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 979098 T3301 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 979099 T3301 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 979100 T3314 oasc.OneIndexer.run Starting indexing thread: 3314
   [junit4]   2> 979100 T3317 oasc.OneIndexer.run Starting indexing thread: 3317
   [junit4]   2> 979100 T3318 oasc.OneIndexer.run Starting indexing thread: 3318
   [junit4]   2> 979100 T3316 oasc.OneIndexer.run Starting indexing thread: 3316
   [junit4]   2> 979100 T3315 oasc.OneIndexer.run Starting indexing thread: 3315
   [junit4]   2> 979102 T3320 oasc.OneQuery.run Starting query thread: 3320
   [junit4]   2> 979102 T3323 oasc.OneQuery.run Starting query thread: 3323
   [junit4]   2> 979102 T3322 oasc.OneQuery.run Starting query thread: 3322
   [junit4]   2> 979102 T3319 oasc.OneQuery.run Starting query thread: 3319
   [junit4]   2> 979102 T3321 oasc.OneQuery.run Starting query thread: 3321
   [junit4]   2> 979105 T3309 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 979105 T3309 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 979106 T3308 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 979106 T3308 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 979104 T3307 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 979107 T3307 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 979108 T3324 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 979108 T3324 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 979180 T3309 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979187 T3307 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979190 T3308 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979195 T3324 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979251 T3307 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979251 T3307 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979253 T3308 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979253 T3308 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979256 T3307 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 979257 T3308 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 979269 T3309 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979270 T3309 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979278 T3309 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 979280 T3307 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979282 T3308 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979282 T3307 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979282 T3307 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 979283 T3307 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 979283 T3308 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979283 T3308 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 979284 T3308 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 979283 T3307 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 979284 T3307 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 979284 T3308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 979284 T3307 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979285 T3308 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 979285 T3307 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 979285 T3308 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979286 T3307 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979286 T3308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 979287 T3307 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fcf4d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@32cd83),segFN=segments_1,generation=1}
   [junit4]   2> 979287 T3307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979287 T3308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979288 T3307 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979288 T3308 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@49f554 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a103d),segFN=segments_1,generation=1}
   [junit4]   2> 979289 T3308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979288 T3307 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979289 T3324 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979289 T3308 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979290 T3324 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979289 T3307 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979290 T3308 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979291 T3307 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979291 T3308 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979292 T3308 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979292 T3307 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979299 T3308 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979301 T3307 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979301 T3307 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979302 T3307 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979302 T3324 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 979303 T3307 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fcf4d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@32cd83),segFN=segments_1,generation=1}
   [junit4]   2> 979303 T3307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979303 T3308 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979304 T3307 oass.SolrIndexSearcher.<init> Opening Searcher@3817f6 main
   [junit4]   2> 979304 T3308 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979304 T3330 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@3817f6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979305 T3308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979305 T3307 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 979305 T3308 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@49f554 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a103d),segFN=segments_1,generation=1}
   [junit4]   2> 979306 T3308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979306 T3307 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 979306 T3308 oass.SolrIndexSearcher.<init> Opening Searcher@1688381 main
   [junit4]   2> 979307 T3308 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 979308 T3308 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 979310 T3309 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979312 T3309 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979313 T3309 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 979313 T3309 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> ASYNC  NEW_CORE C9051 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979314 T3327 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979314 T3309 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 979314 T3328 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979316 T3325 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 979315 T3331 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@1688381 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979315 T3309 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 979317 T3310 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C9052 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979317 T3308 C9052 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 2
   [junit4]   2> 979318 T3309 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979319 T3309 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 979320 T3309 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979321 T3325 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 1
   [junit4]   2> 979321 T3309 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@28c42e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f33d10),segFN=segments_1,generation=1}
   [junit4]   2> 979322 T3309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979322 T3329 C9052 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979323 T3307 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 1
   [junit4]   2> 979323 T3309 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979324 T3309 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979324 T3332 C9052 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 1
   [junit4]   2> 979324 T3309 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979325 T3309 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979326 T3309 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979333 T3309 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979334 T3309 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979334 T3309 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979335 T3309 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@28c42e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f33d10),segFN=segments_1,generation=1}
   [junit4]   2> 979335 T3309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> ASYNC  NEW_CORE C9053 name=00000_core org.apache.solr.core.SolrCore@1a99b03
   [junit4]   2> 979335 T3310 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 979335 T3309 oass.SolrIndexSearcher.<init> Opening Searcher@1cc8469 main
   [junit4]   2> 979336 T3309 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 979336 T3335 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@1cc8469 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979337 T3309 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 979339 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9054 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979342 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9055 name=00001_core org.apache.solr.core.SolrCore@1e04c58
   [junit4]   2> 979343 T3309 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 1
   [junit4]   2> 979343 T3324 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979344 T3325 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9056 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979345 T3307 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 979345 T3324 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979346 T3324 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 979346 T3310 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 0
   [junit4]   2> 979346 T3324 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 979347 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> 979348 T3324 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 979348 T3324 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 979349 T3324 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979349 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 979349 T3324 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 979349 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 979350 T3324 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979351 T3324 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@16a1bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1641fb2),segFN=segments_1,generation=1}
   [junit4]   2> 979352 T3324 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979351 T3325 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 0
   [junit4]   2> 979353 T3324 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979354 T3324 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979354 T3324 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979355 T3310 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 979355 T3324 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979357 T3324 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979357 T3308 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> 979360 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 979360 T3324 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979361 T3324 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979361 T3324 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979362 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 0
   [junit4]   2> 979362 T3324 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@16a1bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1641fb2),segFN=segments_1,generation=1}
   [junit4]   2> 979363 T3324 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979363 T3324 oass.SolrIndexSearcher.<init> Opening Searcher@1f172e0 main
   [junit4]   2> 979364 T3324 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 979364 T3325 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 0
   [junit4]   2> 979364 T3338 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@1f172e0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979365 T3324 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> ASYNC  NEW_CORE C9057 name=00003_core org.apache.solr.core.SolrCore@11dde9e
   [junit4]   2> 979373 T3326 C9057 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979376 T3328 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 4
   [junit4]   2> 979377 T3310 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 2
   [junit4]   2> 979376 T3324 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 1
   [junit4]   2> 979376 T3307 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 979377 T3332 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 0
   [junit4]   2> 979379 T3308 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 979380 T3309 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> 979380 T3325 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 979380 T3329 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 979381 T3326 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 979382 T3328 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 1
   [junit4]   2> 979382 T3310 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 0
   [junit4]   2> 979383 T3324 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 979383 T3307 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> 979384 T3332 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> 979384 T3308 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 0
   [junit4]   2> 979385 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 979385 T3325 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 0
   [junit4]   2> 979386 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 979386 T3326 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 0
   [junit4]   2> 979387 T3328 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 979387 T3310 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9058 name=00000_core org.apache.solr.core.SolrCore@1a99b03
   [junit4]   2> 979393 T3324 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 6
   [junit4]   2> ASYNC  NEW_CORE C9059 name=00001_core org.apache.solr.core.SolrCore@1e04c58
   [junit4]   2> 979394 T3329 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9060 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979393 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9061 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979394 T3310 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 0
   [junit4]   2> 979395 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 979396 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 979396 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 0
   [junit4]   2> 979397 T3326 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 979397 T3328 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 979397 T3332 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9062 name=00003_core org.apache.solr.core.SolrCore@11dde9e
   [junit4]   2> 979398 T3324 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 979399 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 979400 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 1
   [junit4]   2> 979399 T3307 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 0
   [junit4]   2> 979400 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 979401 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 979402 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 979403 T3326 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 979403 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 979404 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 979404 T3326 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 0
   [junit4]   2> 979406 T3329 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 979406 T3307 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 0
   [junit4]   2> 979407 T3310 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 979406 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 0
   [junit4]   2> 979407 T3329 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 0
   [junit4]   2> 979408 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 0
   [junit4]   2> 979409 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 979410 T3328 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 1
   [junit4]   2> 979410 T3325 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 0
   [junit4]   2> 979411 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 979412 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 979412 T3328 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 979413 T3307 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 979413 T3332 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 979413 T3329 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 0
   [junit4]   2> 979414 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 979414 T3326 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 0
   [junit4]   2> 979415 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 979415 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 979416 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 979416 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 0
   [junit4]   2> 979417 T3325 C9062 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979417 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 979418 T3328 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 979418 T3326 C9061 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979419 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 979419 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 979418 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 0
   [junit4]   2> 979420 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 979419 T3310 C9059 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979421 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 979422 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 979422 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 0
   [junit4]   2> 979424 T3308 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 979424 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 0
   [junit4]   2> 979423 T3325 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 2
   [junit4]   2> 979423 T3326 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 979424 T3332 C9062 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979426 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 979427 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 979427 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 979427 T3324 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 979428 T3308 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 979429 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 979430 T3308 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 979430 T3329 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 0
   [junit4]   2> 979430 T3328 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 0
   [junit4]   2> 979431 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 0
   [junit4]   2> 979431 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 0
   [junit4]   2> 979433 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 979433 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 1
   [junit4]   2> 979434 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 979434 T3326 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 0
   [junit4]   2> 979434 T3332 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 0
   [junit4]   2> 979436 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> 979436 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 979438 T3325 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 979437 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 0
   [junit4]   2> 979437 T3328 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 1
   [junit4]   2> 979439 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 979440 T3324 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 979441 T3326 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 979441 T3310 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 979443 T3329 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> 979442 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 0
   [junit4]   2> 979442 T3308 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 1
   [junit4]   2> 979441 T3332 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 0
   [junit4]   2> 979447 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 1
   [junit4]   2> 979447 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 979447 T3328 C9062 oasup.LogUpdateProcessor.fini

[...truncated too long message...]

]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)
   [junit4] Completed on J0 in 54.40s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 500 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 584 ignored (5 assumptions)

Total time: 45 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8318 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8318/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)




Build Log:
[...truncated 10288 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 775944 T3166 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsNew
   [junit4]   2> 775953 T3166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 775956 T3166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54976
   [junit4]   2> 775956 T3166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 775972 T3166 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 776004 T3166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 776005 T3166 oasc.CoreContainer.<init> New CoreContainer 312505862
   [junit4]   2> 776005 T3166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 776008 T3166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 776008 T3166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 776009 T3166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 776009 T3166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 776011 T3166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 776012 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00001_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/
   [junit4]   2> 776012 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00000_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/
   [junit4]   2> 776013 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00004_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/
   [junit4]   2> 776013 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00002_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/
   [junit4]   2> 776014 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00003_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/
   [junit4]   2> 776014 T3166 oasc.CorePropertiesLocator.discover Found 5 core definitions
   [junit4]   2> 776015 T3166 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 776015 T3166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 776020 T3166 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 776021 T3166 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 776023 T3179 oasc.OneIndexer.run Starting indexing thread: 3179
   [junit4]   2> 776024 T3181 oasc.OneIndexer.run Starting indexing thread: 3181
   [junit4]   2> 776024 T3177 oasc.OneIndexer.run Starting indexing thread: 3177
   [junit4]   2> 776023 T3180 oasc.OneIndexer.run Starting indexing thread: 3180
   [junit4]   2> 776023 T3178 oasc.OneIndexer.run Starting indexing thread: 3178
   [junit4]   2> 776028 T3182 oasc.OneQuery.run Starting query thread: 3182
   [junit4]   2> 776028 T3174 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 776028 T3173 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 776028 T3172 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 776028 T3186 oasc.OneQuery.run Starting query thread: 3186
   [junit4]   2> 776028 T3185 oasc.OneQuery.run Starting query thread: 3185
   [junit4]   2> 776028 T3183 oasc.OneQuery.run Starting query thread: 3183
   [junit4]   2> 776028 T3184 oasc.OneQuery.run Starting query thread: 3184
   [junit4]   2> 776029 T3172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 776029 T3173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 776029 T3174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 776068 T3172 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776071 T3174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776074 T3173 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776096 T3172 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776096 T3172 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776099 T3174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776099 T3174 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776100 T3172 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 776103 T3174 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 776126 T3173 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776126 T3173 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776132 T3189 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 776133 T3189 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 776133 T3173 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 776135 T3172 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776137 T3172 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776137 T3172 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 776138 T3172 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 776138 T3174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776138 T3172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 776139 T3172 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 776139 T3172 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776139 T3174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776140 T3172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 776140 T3174 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 776140 T3174 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 776140 T3172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5aa4e74a
   [junit4]   2> 776141 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 776141 T3174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 776141 T3172 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6b98e68a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@722ebd57),segFN=segments_1,generation=1}
   [junit4]   2> 776142 T3174 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776142 T3172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776142 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 776143 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@55b2afd6
   [junit4]   2> 776143 T3172 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776143 T3174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@685c64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55cabb67),segFN=segments_1,generation=1}
   [junit4]   2> 776144 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776144 T3172 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776144 T3172 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776145 T3174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776145 T3172 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776145 T3174 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776146 T3174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776146 T3174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776148 T3174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776148 T3172 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776151 T3174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776151 T3173 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776151 T3174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776151 T3172 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776152 T3172 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776152 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776153 T3173 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776153 T3174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@685c64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55cabb67),segFN=segments_1,generation=1}
   [junit4]   2> 776154 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776153 T3172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776154 T3174 oass.SolrIndexSearcher.<init> Opening Searcher@13665fd2 main
   [junit4]   2> 776154 T3172 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6b98e68a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@722ebd57),segFN=segments_1,generation=1}
   [junit4]   2> 776155 T3172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776155 T3174 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 776153 T3173 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 776156 T3173 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 776155 T3174 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 776155 T3172 oass.SolrIndexSearcher.<init> Opening Searcher@432a3217 main
   [junit4]   2> 776155 T3194 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@13665fd2 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776156 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 776157 T3195 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@432a3217 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776157 T3173 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 776157 T3172 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 776158 T3173 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776159 T3172 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 776159 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 776160 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@32d9aa1e
   [junit4]   2> 776161 T3173 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6e44cea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3daac8ba),segFN=segments_1,generation=1}
   [junit4]   2> 776161 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776163 T3173 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> ASYNC  NEW_CORE C624 name=00003_core org.apache.solr.core.SolrCore@5e531ef7
   [junit4]   2> 776164 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 2
   [junit4]   2> 776164 T3173 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776165 T3173 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776165 T3173 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> ASYNC  NEW_CORE C625 name=00004_core org.apache.solr.core.SolrCore@50557430
   [junit4]   2> 776167 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 2
   [junit4]   2> 776167 T3173 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776169 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 1
   [junit4]   2> 776169 T3188 C624 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776170 T3173 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776170 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 776170 T3173 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776170 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 1
   [junit4]   2> 776171 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 0
   [junit4]   2> 776171 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776172 T3173 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6e44cea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3daac8ba),segFN=segments_1,generation=1}
   [junit4]   2> 776173 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776172 T3188 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 776173 T3188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 776173 T3173 oass.SolrIndexSearcher.<init> Opening Searcher@11abeae main
   [junit4]   2> 776174 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 776174 T3173 oasc.SolrCores.putTransientCore Opening transient core 00000_core
   [junit4]   2> 776174 T3198 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@11abeae main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776174 T3173 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 776176 T3189 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776181 T3175 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C626 name=00000_core org.apache.solr.core.SolrCore@781fcd95
   [junit4]   2> 776181 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 3
   [junit4]   2> 776181 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 3
   [junit4]   2> 776184 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 776187 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> 776190 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 776192 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 776195 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 0
   [junit4]   2> 776198 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 776201 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 0
   [junit4]   2> 776207 T3189 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776208 T3188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776208 T3189 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776211 T3189 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 776223 T3189 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776224 T3189 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776224 T3189 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 776224 T3189 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 776225 T3189 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 776225 T3189 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 776225 T3189 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776225 T3189 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 776226 T3189 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3182e3df
   [junit4]   2> 776227 T3189 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10a84ae4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@468aca88),segFN=segments_1,generation=1}
   [junit4]   2> 776227 T3189 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776228 T3189 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776230 T3189 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776231 T3189 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776232 T3189 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776232 T3189 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776232 T3189 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10a84ae4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@468aca88),segFN=segments_1,generation=1}
   [junit4]   2> 776233 T3189 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776233 T3189 oass.SolrIndexSearcher.<init> Opening Searcher@20289fbc main
   [junit4]   2> 776234 T3189 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 776234 T3200 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@20289fbc main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776234 T3189 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 776236 T3188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> ASYNC  NEW_CORE C627 name=00001_core org.apache.solr.core.SolrCore@76ecef0a
   [junit4]   2> 776238 T3189 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776238 T3188 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776240 T3190 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776240 T3191 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776240 T3192 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776242 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 3
   [junit4]   2> 776242 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 2
   [junit4]   2> 776243 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> 776246 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 776246 T3188 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 776247 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 1
   [junit4]   2> 776248 T3191 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 0
   [junit4]   2> 776250 T3192 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 776251 T3174 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 776252 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> 776254 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 776255 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 776256 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 0
   [junit4]   2> 776260 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 776261 T3188 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776262 T3188 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776263 T3188 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 776263 T3188 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 776263 T3188 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 776264 T3188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 776264 T3188 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776264 T3188 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 776264 T3188 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2d681d4d
   [junit4]   2> 776265 T3188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@969c672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e98a567),segFN=segments_1,generation=1}
   [junit4]   2> 776265 T3188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776266 T3188 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776266 T3188 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776267 T3188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776267 T3188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776268 T3188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776269 T3188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776270 T3188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776270 T3188 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776271 T3188 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@969c672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e98a567),segFN=segments_1,generation=1}
   [junit4]   2> 776271 T3188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776271 T3188 oass.SolrIndexSearcher.<init> Opening Searcher@6428b28b main
   [junit4]   2> 776271 T3172 C625 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776272 T3188 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 776272 T3202 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@6428b28b main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776272 T3188 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> ASYNC  NEW_CORE C628 name=00002_core org.apache.solr.core.SolrCore@3ec4fbaf
   [junit4]   2> 776277 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 2
   [junit4]   2> 776281 T3175 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 3
   [junit4]   2> 776282 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 4
   [junit4]   2> 776282 T3174 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 4
   [junit4]   2> 776282 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 4
   [junit4]   2> 776283 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 2
   [junit4]   2> 776285 T3190 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 776285 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 776286 T3192 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 776286 T3172 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 776287 T3188 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 776288 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 776288 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 1
   [junit4]   2> 776289 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 776290 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> 776290 T3191 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 776291 T3190 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 1
   [junit4]   2> 776291 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 776292 T3192 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 776292 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 0
   [junit4]   2> 776293 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 776293 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 776294 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 776294 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 776295 T3173 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 776295 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 0
   [junit4]   2> 776296 T3190 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 776296 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 0
   [junit4]   2> 776297 T3192 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 776297 T3189 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 776298 T3188 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 776298 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 776299 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 776300 T3174 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 1
   [junit4]   2> 776300 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 776301 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 776301 T3190 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 776302 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 776303 T3192 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 776303 T3189 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 776304 T3188 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 776304 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 776304 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 0
   [junit4]   2> 776306 T3174 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 776306 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 0
   [junit4]   2> 776307 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 776307 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 776307 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 0
   [junit4]   2> 776309 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 776309 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 776309 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 0
   [junit4]   2> 776310 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 776310 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 776312 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 776312 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 776313 T3191 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 776313 T3190 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 776313 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 0
   [junit4]   2> 776315 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 776315 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 776315 T3188 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 0
   [junit4]   2> 776316 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 776316 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 0
   [junit4]   2> 776318 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 1
   [junit4]   2> 776318 T3174 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 776318 T3191 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 776320 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 776320 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 776321 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 776321 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 0
   [junit4]   2> 776321 T3192 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 0
   [junit4]   2> 776323 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 776324 T3189 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 2
   [junit4]   2> 776326 T3174 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 2
   [junit4]   2> 776326 T3191 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 776327 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 4
   [junit4]   2> 776327 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 776327 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 776330 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 2
   [junit4]   2> 776330 T3188 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 776331 T3175 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 776331 T3189 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 1
   [junit4]   2> 776331 T3192 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 2
   [junit4]   2> 776333 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 1
   [junit4]   2> 776335 T3191 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 2
   [junit4]   2> 776335 T3173 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> 776336 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 2
   [junit4]   2> 776336 T3190 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 1
   [junit4]   2> 776337 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 776339 T3188 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 2
   [junit4]   2> 776339 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 776340 T3192 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 776339 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 776340 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 776341 T3191 C625 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776343 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 2
   [junit4]   2> 776343 T3188 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776344 T3192 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776343 T3190 C626 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776344 T3172 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> 776345 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 776343 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 1
   [junit4]   2> 776346 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id130]} 0 1
   [junit4]   2> 776347 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 4
   [junit4]   2> 776349 T3191 C627 oas

[...truncated too long message...]

 [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)
   [junit4] Completed on J0 in 54.11s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 585 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 823 ignored (5 assumptions)

Total time: 39 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b114) - Build # 8317 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8317/
Java: 32bit/jdk1.8.0-ea-b114 -server -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=325, name=commitScheduler-259-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=326, name=commitScheduler-261-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=331, name=commitScheduler-266-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=294, name=commitScheduler-248-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=329, name=commitScheduler-254-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=280, name=commitScheduler-237-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=290, name=commitScheduler-241-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=327, name=commitScheduler-263-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=291, name=commitScheduler-245-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=292, name=commitScheduler-243-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=325, name=commitScheduler-259-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=326, name=commitScheduler-261-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=331, name=commitScheduler-266-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=294, name=commitScheduler-248-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=329, name=commitScheduler-254-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=280, name=commitScheduler-237-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=290, name=commitScheduler-241-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=327, name=commitScheduler-263-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=291, name=commitScheduler-245-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=292, name=commitScheduler-243-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([C9AB0ABE8B96413D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=325, name=commitScheduler-259-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=326, name=commitScheduler-261-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=331, name=commitScheduler-266-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=294, name=commitScheduler-248-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=329, name=commitScheduler-254-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=280, name=commitScheduler-237-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=290, name=commitScheduler-241-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=327, name=commitScheduler-263-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=291, name=commitScheduler-245-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=292, name=commitScheduler-243-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=325, name=commitScheduler-259-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=326, name=commitScheduler-261-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=331, name=commitScheduler-266-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=294, name=commitScheduler-248-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=329, name=commitScheduler-254-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=280, name=commitScheduler-237-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=290, name=commitScheduler-241-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=327, name=commitScheduler-263-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=291, name=commitScheduler-245-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=292, name=commitScheduler-243-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([C9AB0ABE8B96413D]:0)




Build Log:
[...truncated 10070 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.01s J1 | OpenCloseCoreStressTest.test1HourOld
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly)
   [junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test1HourNew
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly)
   [junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test10MinutesNew
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4]   2> 123053 T257 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsNew
   [junit4]   2> 123065 T257 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 123071 T257 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39762
   [junit4]   2> 123072 T257 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 123072 T257 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 123072 T257 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 123072 T257 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/'
   [junit4]   2> 123081 T257 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 123093 T257 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 123095 T257 oasc.CoreContainer.<init> New CoreContainer 5425943
   [junit4]   2> 123095 T257 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/]
   [junit4]   2> 123095 T257 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 123096 T257 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 123096 T257 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 123096 T257 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 123096 T257 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 123097 T257 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 123097 T257 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 123097 T257 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 123097 T257 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 123099 T257 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 123099 T257 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 123099 T257 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 123101 T257 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 123103 T257 oasc.CorePropertiesLocator.discoverUnder Found core 00001_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/
   [junit4]   2> 123104 T257 oasc.CorePropertiesLocator.discoverUnder Found core 00000_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/
   [junit4]   2> 123105 T257 oasc.CorePropertiesLocator.discoverUnder Found core 00004_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/
   [junit4]   2> 123105 T257 oasc.CorePropertiesLocator.discoverUnder Found core 00002_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/
   [junit4]   2> 123106 T257 oasc.CorePropertiesLocator.discoverUnder Found core 00003_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/
   [junit4]   2> 123106 T257 oasc.CorePropertiesLocator.discover Found 5 core definitions
   [junit4]   2> 123106 T257 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 123107 T257 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 123110 T257 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 123111 T257 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 123112 T268 oasc.OneIndexer.run Starting indexing thread: 268
   [junit4]   2> 123112 T271 oasc.OneIndexer.run Starting indexing thread: 271
   [junit4]   2> 123112 T270 oasc.OneIndexer.run Starting indexing thread: 270
   [junit4]   2> 123112 T269 oasc.OneIndexer.run Starting indexing thread: 269
   [junit4]   2> 123113 T272 oasc.OneIndexer.run Starting indexing thread: 272
   [junit4]   2> 123115 T273 oasc.OneQuery.run Starting query thread: 273
   [junit4]   2> 123116 T276 oasc.OneQuery.run Starting query thread: 276
   [junit4]   2> 123116 T275 oasc.OneQuery.run Starting query thread: 275
   [junit4]   2> 123116 T274 oasc.OneQuery.run Starting query thread: 274
   [junit4]   2> 123117 T263 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 123117 T264 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 123118 T264 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 123117 T263 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 123120 T277 oasc.OneQuery.run Starting query thread: 277
   [junit4]   2> 123120 T265 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 123121 T265 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 123127 T278 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 123127 T278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 123143 T264 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 123150 T264 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 123151 T264 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 123168 T265 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 123172 T264 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 123175 T265 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 123176 T265 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 123177 T264 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 123177 T264 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 123183 T264 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 123184 T264 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 123185 T265 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 123185 T264 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 123185 T264 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 123185 T264 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 123186 T264 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 123186 T264 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=99.892578125, floorSegmentMB=1.435546875, forceMergeDeletesPctAllowed=1.2122909516147706, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6921858646775758
   [junit4]   2> 123187 T263 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 123187 T264 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1f8436b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@155cd69),segFN=segments_1,generation=1}
   [junit4]   2> 123188 T264 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123189 T265 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 123189 T264 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 123189 T264 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 123190 T264 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 123190 T264 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 123189 T265 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 123197 T278 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 123198 T265 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 123198 T265 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 123198 T264 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 123199 T265 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 123200 T265 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 123200 T265 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 123199 T264 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 123201 T264 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 123201 T263 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 123202 T265 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 123202 T263 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 123202 T264 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=85.65625, floorSegmentMB=1.9619140625, forceMergeDeletesPctAllowed=8.938063013590337, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 123202 T265 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=99.892578125, floorSegmentMB=1.435546875, forceMergeDeletesPctAllowed=1.2122909516147706, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6921858646775758
   [junit4]   2> 123204 T264 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1f8436b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@155cd69),segFN=segments_1,generation=1}
   [junit4]   2> 123204 T264 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123204 T264 oass.SolrIndexSearcher.<init> Opening Searcher@11054ff main
   [junit4]   2> 123205 T263 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 123205 T264 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 123205 T278 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 123206 T264 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 123206 T278 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 123208 T265 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1d08903 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cc3bc6),segFN=segments_1,generation=1}
   [junit4]   2> 123213 T265 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123215 T263 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 123215 T265 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 123216 T278 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 123205 T279 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@11054ff main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 123216 T265 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 123215 T263 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 123218 T265 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 123218 T263 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> ASYNC  NEW_CORE C92 name=00004_core org.apache.solr.core.SolrCore@1c98a67
   [junit4]   2> 123219 T264 C92 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 12
   [junit4]   2> 123219 T266 C92 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 12
   [junit4]   2> 123219 T265 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 123219 T263 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 123222 T278 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 123222 T278 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 123222 T278 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 123223 T278 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 123223 T265 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 123224 T265 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 123224 T265 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 123232 T265 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=85.65625, floorSegmentMB=1.9619140625, forceMergeDeletesPctAllowed=8.938063013590337, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 123232 T263 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 123233 T263 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 123234 T263 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 123234 T265 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1d08903 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cc3bc6),segFN=segments_1,generation=1}
   [junit4]   2> 123235 T265 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123235 T265 oass.SolrIndexSearcher.<init> Opening Searcher@105ecf3 main
   [junit4]   2> 123236 T286 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 123236 T278 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 123236 T286 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 123236 T263 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 123237 T263 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=99.892578125, floorSegmentMB=1.435546875, forceMergeDeletesPctAllowed=1.2122909516147706, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6921858646775758
   [junit4]   2> 123238 T263 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4cf11a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a24808),segFN=segments_1,generation=1}
   [junit4]   2> 123238 T263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123238 T278 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 123239 T278 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 123239 T263 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 123239 T278 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 123239 T263 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 123240 T278 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=99.892578125, floorSegmentMB=1.435546875, forceMergeDeletesPctAllowed=1.2122909516147706, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6921858646775758
   [junit4]   2> 123240 T263 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 123240 T263 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 123241 T278 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@164703b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a4ea0),segFN=segments_1,generation=1}
   [junit4]   2> 123241 T278 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123242 T263 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 123242 T278 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 123242 T278 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 123242 T278 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 123243 T278 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 123244 T278 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 123245 T278 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 123245 T265 oasc.SolrCores.putTransientCore Opening transient core 00000_core
   [junit4]   2> 123245 T281 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@105ecf3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 123245 T263 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 123245 T278 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 123246 T263 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 123247 T263 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=85.65625, floorSegmentMB=1.9619140625, forceMergeDeletesPctAllowed=8.938063013590337, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 123247 T263 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4cf11a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a24808),segFN=segments_1,generation=1}
   [junit4]   2> 123247 T263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123248 T263 oass.SolrIndexSearcher.<init> Opening Searcher@acc6cf main
   [junit4]   2> 123246 T265 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 123248 T288 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@acc6cf main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 123248 T263 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 123246 T278 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=85.65625, floorSegmentMB=1.9619140625, forceMergeDeletesPctAllowed=8.938063013590337, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> ASYNC  NEW_CORE C93 name=00000_core org.apache.solr.core.SolrCore@8e5847
   [junit4]   2> 123252 T264 C93 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 2
   [junit4]   2> 123254 T278 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@164703b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a4ea0),segFN=segments_1,generation=1}
   [junit4]   2> ASYNC  NEW_CORE C94 name=00000_core org.apache.solr.core.SolrCore@8e5847
   [junit4]   2> 123253 T265 C94 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 2
   [junit4]   2> 123253 T263 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> ASYNC  NEW_CORE C95 name=00002_core org.apache.solr.core.SolrCore@5fa69a
   [junit4]   2> 123266 T285 C95 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123264 T286 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 123267 T264 C94 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 1
   [junit4]   2> 123263 T278 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123267 T278 oass.SolrIndexSearcher.<init> Opening Searcher@1c5f46f main
   [junit4]   2> 123266 T282 C94 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=15 
   [junit4]   2> 123269 T289 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@1c5f46f main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 123269 T278 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 123270 T278 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 123268 T265 C95 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 2
   [junit4]   2> 123271 T285 C95 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C96 name=00003_core org.apache.solr.core.SolrCore@142672d
   [junit4]   2> 123272 T287 C96 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123268 T263 C95 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 3
   [junit4]   2> 123273 T278 C96 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C97 name=00004_core org.apache.solr.core.SolrCore@1c98a67
   [junit4]   2> 123276 T264 C97 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> 123278 T286 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 123279 T283 C96 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 5
   [junit4]   2> 123281 T286 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 123281 T282 C97 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 3
   [junit4]   2> 123282 T265 C96 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> 123283 T263 C96 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C98 name=00002_core org.apache.solr.core.SolrCore@5fa69a
   [junit4]   2> 123295 T264 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C99 name=00004_core org.apache.solr.core.SolrCore@1c98a67
   [junit4]   2> 123296 T283 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C100 name=00000_core org.apache.solr.core.SolrCore@8e5847
   [junit4]   2> 123295 T278 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 1
   [junit4]   2> 123299 T282 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C101 name=00003_core org.apache.solr.core.SolrCore@142672d
   [junit4]   2> 123299 T265 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 123301 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 3
   [junit4]   2> 123303 T283 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 1
   [junit4]   2> 123305 T278 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 1
   [junit4]   2> 123305 T264 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 3
   [junit4]   2> 123307 T286 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 123307 T282 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 1
   [junit4]   2> 123309 T265 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 2
   [junit4]   2> 123309 T263 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 0
   [junit4]   2> 123311 T286 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 123311 T286 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 123311 T286 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 123312 T286 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 123314 T278 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 2
   [junit4]   2> 123314 T264 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 2
   [junit4]   2> 123317 T282 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 1
   [junit4]   2> 123318 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 1
   [junit4]   2> 123321 T278 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> 123321 T265 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 123326 T282 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 123326 T286 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 123326 T264 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 2
   [junit4]   2> 123326 T286 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 123327 T286 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 123328 T286 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 123328 T286 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=99.892578125, floorSegmentMB=1.435546875, forceMergeDeletesPctAllowed=1.2122909516147706, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6921858646775758
   [junit4]   2> 123329 T286 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@177429e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18fc471),segFN=segments_1,generation=1}
   [junit4]   2> 123329 T286 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123330 T286 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 123330 T286 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 123331 T286 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 123331 T286 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 123332 T286 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 123333 T286 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 123333 T286 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 123334 T286 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=85.65625, floorSegmentMB=1.9619140625, forceMergeDeletesPctAllowed=8.938063013590337, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 123334 T286 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@177429e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18fc471),segFN=segments_1,generation=1}
   [junit4]   2> 123335 T286 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 123335 T278 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 6
   [junit4]   2> 123335 T286 oass.SolrIndexSearcher.<init> Opening Searcher@1e18cc9 main
   [junit4]   2> 123336 T286 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 123335 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 123336 T286 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 123336 T293 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@1e18cc9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C102 name=00001_core org.apache.solr.core.SolrCore@d56d7
   [junit4]   2> 123339 T284 C102 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123341 T286 C102 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123348 T283 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 10
   [junit4]   2> 123349 T287 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 10
   [junit4]   2> 123349 T285 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 11
   [junit4]   2> 123351 T265 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 2
   [junit4]   2> 123352 T264 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 0
   [junit4]   2> 123354 T278 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 2
   [junit4]   2> 123355 T282 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 4
   [junit4]   2> 123356 T286 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 123357 T263 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> 123357 T284 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 123359 T287 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 123361 T283 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 1
   [junit4]   2> 123362 T264 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 123361 T285 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 123363 T278 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 0
   [junit4]   2> 123362 T283 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 0
   [junit4]   2> 123366 T283 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 123367 T283 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 123369 T285 C101 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123369 T265 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 123369 T264 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 0
   [junit4]   2> 123369 T283 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 123371 T282 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 123371 T278 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 123372 T286 C98 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 123374 T263 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 1
   [junit4]   2> 123376 T284 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 123376 T285 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 0
   [junit4]   2> 123378 T264 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 123376 T287 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 123379 T283 C98 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 123380 T278 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 123380 T287 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 123382 T286 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 123383 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 123383 T285 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 123382 T284 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 0
   [junit4]   2> 123387 T264 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 1
   [junit4]   2> 123382 T265 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 123389 T282 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 123389 T283 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 123390 T278 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 123391 T287 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 123392 T263 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 123392 T286 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 123393 T284 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 123394 T285 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 123395 T264 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 123396 T265 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 2
   [junit4]   2> 123396 T285 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 1
   [junit4]   2> 123396 T282 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 0
   [junit4]   2> 123397 T287 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 0
   [junit4]   2> 123399 T286 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 123399 T263 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 123400 T284 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 123400 T278 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 123401 T283 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 123402 T285 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 123403 T282 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 123404 T264 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 123404 T265 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 0
   [junit4]   2> 123405 T287 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 1
   [junit4]   2> 123406 T286 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 0
   [junit4]   2> 123408 T286 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 123409 T286 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 123410 T286 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 123411 T286 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 0
   [junit4]   2> 123412 T284 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 123413 T284 C100 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 0
   [junit4]   2> 123413 T286 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 123414 T284 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 123415 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 123416 T284 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 123417 T263 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 123418 T284 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 123418 T263 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 0
   [junit4]   2> 123419 T284 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 123421 T283 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 123421 T278 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 1
   [junit4]   2> 123421 T282 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 1
   [junit4]   2> 123428 T287 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 1
   [junit4]   2> 123428 T286 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 2
   [junit4]   2> 123429 T264 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 3
   [junit4]   2> 123429 T265 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 2
   [junit4]   2> 123429 T285 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 1
   [junit4]   2> 123431 T283 C98 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 123431 T282 C99 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 0
   [junit4]   2> 123432 T278 C102 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 123433 T263 C101 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 123434 T282 C101 oasup.LogUpdateProcessor.finis

[...truncated too long message...]

seCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=294, name=commitScheduler-248-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=329, name=commitScheduler-254-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=280, name=commitScheduler-237-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=290, name=commitScheduler-241-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=327, name=commitScheduler-263-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=291, name=commitScheduler-245-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=292, name=commitScheduler-243-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C9AB0ABE8B96413D]:0)
   [junit4] Completed on J1 in 54.46s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 865 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 995 ignored (5 assumptions)

Total time: 35 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b114 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure